From kbusch at kernel.org  Tue Jun  1 06:53:10 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 1 Jun 2021 06:53:10 -0700
Subject: NVME_IOCTL_SUBMIT_IO access control
In-Reply-To: <YLTD7gcSIiNRJvGX@x1-carbon>
References: <YLTD7gcSIiNRJvGX@x1-carbon>
Message-ID: <20210601135310.GA4197@dhcp-10-100-145-180.wdc.com>

On Mon, May 31, 2021 at 11:09:35AM +0000, Niklas Cassel wrote:
> Hello there,
> 
> How is the NVME_IOCTL_SUBMIT_IO access control supposed to work?
> 
> $ echo "hello" | nvme write /dev/nvme0n1 -s 0 -c 1 -z 512
> /dev/nvme0n1: Permission denied
> 
> $ sudo chmod o+r /dev/nvme0n1
> 
> $ echo "hello" | nvme write /dev/nvme0n1 -s 0 -c 1 -z 512
> Rounding data size to fit block count (8192 bytes)
> write: Success
> 
> 
> Am I supposed to be able to do a write if I only have read permission?
> 
> $ ls -al /dev/nvme0n1
> brw-rw-r-- 1 root disk 259, 0 May 31 10:59 /dev/nvme0n1

I'm not sure, we've always allowed any user passthrough command if
CAP_SYS_ADMIN capable. It should be pretty easy to check permissions for
any data-out opcode against FMODE_WRITE. We'll probably break something
if we do, though.


From mario.limonciello at amd.com  Tue Jun  1 08:15:09 2021
From: mario.limonciello at amd.com (Limonciello, Mario)
Date: Tue, 1 Jun 2021 10:15:09 -0500
Subject: [PATCH v3 1/2] nvme: Look for StorageD3Enable on companion ACPI
 device instead
In-Reply-To: <PH0PR11MB4853EBD21B8D89AC8138308F9E3E9@PH0PR11MB4853.namprd11.prod.outlook.com>
References: <20210528160234.9402-1-mario.limonciello@amd.com>
 <20210531064655.GA1417@lst.de>
 <PH0PR11MB4853EBD21B8D89AC8138308F9E3E9@PH0PR11MB4853.namprd11.prod.outlook.com>
Message-ID: <474a58bf-dcc9-48a8-c5ce-fb79f2b76a79@amd.com>

> 
> I think we need to wait for the confirmation from David.  This looks good, but I'd like to see testing.  I also wonder how many of the simple suspend quirks we can drop with this.

David, do you think you can get this soonish or are you similarly 
blocked like Shyjumon by remote work restrictions?  As this is causing 
problems for a variety of already shipped OEM systems I would like to 
see this series land for 5.14.  If you can't get to it in time, I would 
prefer that we switch this patch back to how it was conservatively done 
in v1 for 5.14 (and redo 2/2 on top of that):
https://lore.kernel.org/linux-nvme/20210527113751.GB17266 at lst.de/T/#t

and let you get back to the list when you can to optimize it "later".

> 
> Shyjumon and Jon, can you retests the platforms quirked in
> 1fae37accfc5 ("nvme/pci: Add sleep quirk for Samsung and Toshiba drives") with this fix?
> 
> Shyjumon>> Yes, I do agree we need to test this also. However the boards where which I had these issues are not in my remote access now (as the work frequency on this boards are less and also due to Covid situation),
>                          It might  take some time for me to test. I will update as soon as I can.
> 


From Niklas.Cassel at wdc.com  Tue Jun  1 08:15:14 2021
From: Niklas.Cassel at wdc.com (Niklas Cassel)
Date: Tue, 1 Jun 2021 15:15:14 +0000
Subject: NVME_IOCTL_SUBMIT_IO access control
In-Reply-To: <20210601135310.GA4197@dhcp-10-100-145-180.wdc.com>
References: <YLTD7gcSIiNRJvGX@x1-carbon>
 <20210601135310.GA4197@dhcp-10-100-145-180.wdc.com>
Message-ID: <YLZPAV4Cu8ZuYe/Y@x1-carbon>

On Tue, Jun 01, 2021 at 06:53:10AM -0700, Keith Busch wrote:
> On Mon, May 31, 2021 at 11:09:35AM +0000, Niklas Cassel wrote:
> > Hello there,
> > 
> > How is the NVME_IOCTL_SUBMIT_IO access control supposed to work?
> > 
> > $ echo "hello" | nvme write /dev/nvme0n1 -s 0 -c 1 -z 512
> > /dev/nvme0n1: Permission denied
> > 
> > $ sudo chmod o+r /dev/nvme0n1
> > 
> > $ echo "hello" | nvme write /dev/nvme0n1 -s 0 -c 1 -z 512
> > Rounding data size to fit block count (8192 bytes)
> > write: Success
> > 
> > 
> > Am I supposed to be able to do a write if I only have read permission?
> > 
> > $ ls -al /dev/nvme0n1
> > brw-rw-r-- 1 root disk 259, 0 May 31 10:59 /dev/nvme0n1
> 
> I'm not sure, we've always allowed any user passthrough command if
> CAP_SYS_ADMIN capable. It should be pretty easy to check permissions for
> any data-out opcode against FMODE_WRITE. We'll probably break something
> if we do, though.

Hello Keith,

Indeed,
NVME_IOCTL_IO_CMD and NVME_IOCTL_IO64_CMD does require CAP_SYS_ADMIN,
however, the same is not true for NVME_IOCTL_SUBMIT_IO.

Which is why as a regular user, with only o+r, I could do a write using
NVME_IOCTL_SUBMIT_IO.

In one way it makes sense that NVME_IOCTL_IO_CMD requires CAP_SYS_ADMIN,
since you can send any command to the drive using that ioctl.

With NVME_IOCTL_SUBMIT_IO, you can only submit write/read/compare.
So it makes sense to not need CAP_SYS_ADMIN.

But I probably would expect data-out to be checked against FMODE_WRITE.

I just wanted to raise the issue, in case this was accidentally overlooked.


Kind regards,
Niklas


From elad.grupi at dell.com  Tue Jun  1 09:02:34 2021
From: elad.grupi at dell.com (elad.grupi at dell.com)
Date: Tue,  1 Jun 2021 19:02:34 +0300
Subject: [PATCH] nvmet-tcp: fix memory leak when having inflight commands on
 disconnect
Message-ID: <20210601160234.5200-1-elad.grupi@dell.com>

From: Elad Grupi <elad.grupi at dell.com>

Some nvme commands might get completed during the execution of
nvmet_tcp_release_queue_work. Those commands are being added to
resp_list on the queue and never processed after cancel_work_sync
is done, causing memory leak of the resources allocated for those commands.

Signed-off-by: Elad Grupi <elad.grupi at dell.com>
---
 drivers/nvme/target/tcp.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index f9f34f6caf5e..04d53b532ae2 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -1426,6 +1426,31 @@ static void nvmet_tcp_uninit_data_in_cmds(struct nvmet_tcp_queue *queue)
 	}
 }
 
+static void nvmet_tcp_free_resp_list(struct nvmet_tcp_queue *queue)
+{
+	struct nvmet_tcp_cmd *cmd;
+	int c = 0;
+
+	nvmet_tcp_process_resp_list(queue);
+
+	list_for_each_entry(cmd, &queue->resp_send_list, entry) {
+		kfree(cmd->iov);
+		sgl_free(cmd->req.sg);
+		c++;
+	}
+
+	WARN_ON(c != queue->send_list_len);
+
+	if (queue->snd_cmd) {
+		kfree(queue->snd_cmd->iov);
+		sgl_free(queue->snd_cmd->req.sg);
+		c++;
+	}
+
+	pr_debug("qid %u send_list_len %d, free %d unprocessed commands\n",
+			queue->nvme_sq.qid, queue->send_list_len, c);
+}
+
 static void nvmet_tcp_release_queue_work(struct work_struct *w)
 {
 	struct nvmet_tcp_queue *queue =
@@ -1441,6 +1466,7 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)
 	nvmet_tcp_uninit_data_in_cmds(queue);
 	nvmet_sq_destroy(&queue->nvme_sq);
 	cancel_work_sync(&queue->io_work);
+	nvmet_tcp_free_resp_list(queue);
 	sock_release(queue->sock);
 	nvmet_tcp_free_cmds(queue);
 	if (queue->hdr_digest || queue->data_digest)
-- 
2.18.2



From mgurtovoy at nvidia.com  Tue Jun  1 09:22:05 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Tue, 1 Jun 2021 19:22:05 +0300
Subject: [PATCH 1/1] nvmet: fix freeing unallocated p2pmem
Message-ID: <20210601162205.38326-1-mgurtovoy@nvidia.com>

In case p2p device was found but the p2p pool is empty, the nvme target
is still trying to free the sgl from the p2p pool instead of the
regular sgl pool and causing a crash (BUG() is called). Instead, assign
the p2p_dev for the request only if it was allocated from p2p pool.

This is the crash that was caused:

[Sun May 30 19:13:53 2021] ------------[ cut here ]------------
[Sun May 30 19:13:53 2021] kernel BUG at lib/genalloc.c:518!
[Sun May 30 19:13:53 2021] invalid opcode: 0000 [#1] SMP PTI
...
[Sun May 30 19:13:53 2021] kernel BUG at lib/genalloc.c:518!
...
[Sun May 30 19:13:53 2021] RIP: 0010:gen_pool_free_owner+0xa8/0xb0
...
[Sun May 30 19:13:53 2021] Call Trace:
[Sun May 30 19:13:53 2021] ------------[ cut here ]------------
[Sun May 30 19:13:53 2021]  pci_free_p2pmem+0x2b/0x70
[Sun May 30 19:13:53 2021]  pci_p2pmem_free_sgl+0x4f/0x80
[Sun May 30 19:13:53 2021]  nvmet_req_free_sgls+0x1e/0x80 [nvmet]
[Sun May 30 19:13:53 2021] kernel BUG at lib/genalloc.c:518!
[Sun May 30 19:13:53 2021]  nvmet_rdma_release_rsp+0x4e/0x1f0 [nvmet_rdma]
[Sun May 30 19:13:53 2021]  nvmet_rdma_send_done+0x1c/0x60 [nvmet_rdma]

Fixes: c6e3f1339812 ("nvmet: add metadata support for block devices")
Reviewed-by: Israel Rukshin <israelr at nvidia.com>
Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
---
 drivers/nvme/target/core.c | 35 ++++++++++++++++++++---------------
 1 file changed, 20 insertions(+), 15 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 4b29a5bac896..8afbb989b828 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1005,19 +1005,23 @@ static unsigned int nvmet_data_transfer_len(struct nvmet_req *req)
 	return req->transfer_len - req->metadata_len;
 }
 
-static int nvmet_req_alloc_p2pmem_sgls(struct nvmet_req *req)
+static int nvmet_req_alloc_p2pmem_sgls(struct pci_dev *p2p_dev,
+		struct nvmet_req *req)
 {
-	req->sg = pci_p2pmem_alloc_sgl(req->p2p_dev, &req->sg_cnt,
+	req->sg = pci_p2pmem_alloc_sgl(p2p_dev, &req->sg_cnt,
 			nvmet_data_transfer_len(req));
 	if (!req->sg)
 		goto out_err;
 
 	if (req->metadata_len) {
-		req->metadata_sg = pci_p2pmem_alloc_sgl(req->p2p_dev,
+		req->metadata_sg = pci_p2pmem_alloc_sgl(p2p_dev,
 				&req->metadata_sg_cnt, req->metadata_len);
 		if (!req->metadata_sg)
 			goto out_free_sg;
 	}
+
+	req->p2p_dev = p2p_dev;
+
 	return 0;
 out_free_sg:
 	pci_p2pmem_free_sgl(req->p2p_dev, req->sg);
@@ -1025,25 +1029,25 @@ static int nvmet_req_alloc_p2pmem_sgls(struct nvmet_req *req)
 	return -ENOMEM;
 }
 
-static bool nvmet_req_find_p2p_dev(struct nvmet_req *req)
+static struct pci_dev *nvmet_req_find_p2p_dev(struct nvmet_req *req)
 {
-	if (!IS_ENABLED(CONFIG_PCI_P2PDMA))
-		return false;
+	struct pci_dev *p2p_dev = NULL;
 
-	if (req->sq->ctrl && req->sq->qid && req->ns) {
-		req->p2p_dev = radix_tree_lookup(&req->sq->ctrl->p2p_ns_map,
-						 req->ns->nsid);
-		if (req->p2p_dev)
-			return true;
-	}
+	if (!IS_ENABLED(CONFIG_PCI_P2PDMA))
+		goto out;
 
-	req->p2p_dev = NULL;
-	return false;
+	if (req->sq->ctrl && req->sq->qid && req->ns)
+		p2p_dev = radix_tree_lookup(&req->sq->ctrl->p2p_ns_map,
+					    req->ns->nsid);
+out:
+	return p2p_dev;
 }
 
 int nvmet_req_alloc_sgls(struct nvmet_req *req)
 {
-	if (nvmet_req_find_p2p_dev(req) && !nvmet_req_alloc_p2pmem_sgls(req))
+	struct pci_dev *p2p_dev = nvmet_req_find_p2p_dev(req);
+
+	if (p2p_dev && !nvmet_req_alloc_p2pmem_sgls(p2p_dev, req))
 		return 0;
 
 	req->sg = sgl_alloc(nvmet_data_transfer_len(req), GFP_KERNEL,
@@ -1072,6 +1076,7 @@ void nvmet_req_free_sgls(struct nvmet_req *req)
 		pci_p2pmem_free_sgl(req->p2p_dev, req->sg);
 		if (req->metadata_sg)
 			pci_p2pmem_free_sgl(req->p2p_dev, req->metadata_sg);
+		req->p2p_dev = NULL;
 	} else {
 		sgl_free(req->sg);
 		if (req->metadata_sg)
-- 
2.18.1



From logang at deltatee.com  Tue Jun  1 10:09:21 2021
From: logang at deltatee.com (Logan Gunthorpe)
Date: Tue, 1 Jun 2021 11:09:21 -0600
Subject: [PATCH 1/1] nvmet: fix freeing unallocated p2pmem
In-Reply-To: <20210601162205.38326-1-mgurtovoy@nvidia.com>
References: <20210601162205.38326-1-mgurtovoy@nvidia.com>
Message-ID: <d487d20e-4c9b-227a-62b8-92d7c6c46aae@deltatee.com>



On 2021-06-01 10:22 a.m., Max Gurtovoy wrote:
> In case p2p device was found but the p2p pool is empty, the nvme target
> is still trying to free the sgl from the p2p pool instead of the
> regular sgl pool and causing a crash (BUG() is called). Instead, assign
> the p2p_dev for the request only if it was allocated from p2p pool.
> 
> This is the crash that was caused:
> 
> [Sun May 30 19:13:53 2021] ------------[ cut here ]------------
> [Sun May 30 19:13:53 2021] kernel BUG at lib/genalloc.c:518!
> [Sun May 30 19:13:53 2021] invalid opcode: 0000 [#1] SMP PTI
> ...
> [Sun May 30 19:13:53 2021] kernel BUG at lib/genalloc.c:518!
> ...
> [Sun May 30 19:13:53 2021] RIP: 0010:gen_pool_free_owner+0xa8/0xb0
> ...
> [Sun May 30 19:13:53 2021] Call Trace:
> [Sun May 30 19:13:53 2021] ------------[ cut here ]------------
> [Sun May 30 19:13:53 2021]  pci_free_p2pmem+0x2b/0x70
> [Sun May 30 19:13:53 2021]  pci_p2pmem_free_sgl+0x4f/0x80
> [Sun May 30 19:13:53 2021]  nvmet_req_free_sgls+0x1e/0x80 [nvmet]
> [Sun May 30 19:13:53 2021] kernel BUG at lib/genalloc.c:518!
> [Sun May 30 19:13:53 2021]  nvmet_rdma_release_rsp+0x4e/0x1f0 [nvmet_rdma]
> [Sun May 30 19:13:53 2021]  nvmet_rdma_send_done+0x1c/0x60 [nvmet_rdma]
> 
> Fixes: c6e3f1339812 ("nvmet: add metadata support for block devices")
> Reviewed-by: Israel Rukshin <israelr at nvidia.com>
> Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>

This makes sense to me save one small nit below. Thanks!

Reviewed-by: Logan Gunthorpe <logang at deltatee.com>



> -static bool nvmet_req_find_p2p_dev(struct nvmet_req *req)
> +static struct pci_dev *nvmet_req_find_p2p_dev(struct nvmet_req *req)
>  {
> -	if (!IS_ENABLED(CONFIG_PCI_P2PDMA))
> -		return false;
> +	struct pci_dev *p2p_dev = NULL;
>  
> -	if (req->sq->ctrl && req->sq->qid && req->ns) {
> -		req->p2p_dev = radix_tree_lookup(&req->sq->ctrl->p2p_ns_map,
> -						 req->ns->nsid);
> -		if (req->p2p_dev)
> -			return true;
> -	}
> +	if (!IS_ENABLED(CONFIG_PCI_P2PDMA))
> +		goto out;

Seems like we could just return NULL here and save the label and goto.

> -	req->p2p_dev = NULL;
> -	return false;
> +	if (req->sq->ctrl && req->sq->qid && req->ns)
> +		p2p_dev = radix_tree_lookup(&req->sq->ctrl->p2p_ns_map,
> +					    req->ns->nsid);
> +out:
> +	return p2p_dev;
>  }
>  


From kbusch at kernel.org  Tue Jun  1 10:15:13 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 1 Jun 2021 10:15:13 -0700
Subject: NVME_IOCTL_SUBMIT_IO access control
In-Reply-To: <YLZPAV4Cu8ZuYe/Y@x1-carbon>
References: <YLTD7gcSIiNRJvGX@x1-carbon>
 <20210601135310.GA4197@dhcp-10-100-145-180.wdc.com>
 <YLZPAV4Cu8ZuYe/Y@x1-carbon>
Message-ID: <20210601171513.GA4506@dhcp-10-100-145-180.wdc.com>

On Tue, Jun 01, 2021 at 03:15:14PM +0000, Niklas Cassel wrote:
> On Tue, Jun 01, 2021 at 06:53:10AM -0700, Keith Busch wrote:
> > On Mon, May 31, 2021 at 11:09:35AM +0000, Niklas Cassel wrote:
> > > Hello there,
> > > 
> > > How is the NVME_IOCTL_SUBMIT_IO access control supposed to work?
> > > 
> > > $ echo "hello" | nvme write /dev/nvme0n1 -s 0 -c 1 -z 512
> > > /dev/nvme0n1: Permission denied
> > > 
> > > $ sudo chmod o+r /dev/nvme0n1
> > > 
> > > $ echo "hello" | nvme write /dev/nvme0n1 -s 0 -c 1 -z 512
> > > Rounding data size to fit block count (8192 bytes)
> > > write: Success
> > > 
> > > 
> > > Am I supposed to be able to do a write if I only have read permission?
> > > 
> > > $ ls -al /dev/nvme0n1
> > > brw-rw-r-- 1 root disk 259, 0 May 31 10:59 /dev/nvme0n1
> > 
> > I'm not sure, we've always allowed any user passthrough command if
> > CAP_SYS_ADMIN capable. It should be pretty easy to check permissions for
> > any data-out opcode against FMODE_WRITE. We'll probably break something
> > if we do, though.
> 
> Hello Keith,
> 
> Indeed,
> NVME_IOCTL_IO_CMD and NVME_IOCTL_IO64_CMD does require CAP_SYS_ADMIN,
> however, the same is not true for NVME_IOCTL_SUBMIT_IO.
> 
> Which is why as a regular user, with only o+r, I could do a write using
> NVME_IOCTL_SUBMIT_IO.
> 
> In one way it makes sense that NVME_IOCTL_IO_CMD requires CAP_SYS_ADMIN,
> since you can send any command to the drive using that ioctl.
> 
> With NVME_IOCTL_SUBMIT_IO, you can only submit write/read/compare.
> So it makes sense to not need CAP_SYS_ADMIN.
> 
> But I probably would expect data-out to be checked against FMODE_WRITE.
> 
> I just wanted to raise the issue, in case this was accidentally overlooked.

You're right, and SUBMIT_IO has always behaved that way. I believe this
was an oversight and should have been checking fmode_t against the
opcode.


From Chaitanya.Kulkarni at wdc.com  Tue Jun  1 10:28:05 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue, 1 Jun 2021 17:28:05 +0000
Subject: [PATCH] nvme: verify MNAN value if ANA is enabled
References: <20210531103651.109426-1-dwagner@suse.de>
Message-ID: <DM6PR04MB4972F39AB64A70710D8CD9CB863E9@DM6PR04MB4972.namprd04.prod.outlook.com>

On 5/31/21 03:43, Daniel Wagner wrote:
> The controller is required to have a non-zero MNAN value if it supports
> ANA:
>
>    If the controller supports Asymmetric Namespace Access Reporting, then
>    this field shall be set to a non-zero value that is less than or equal
>    to the NN value.
>
> Signed-off-by: Daniel Wagner <dwagner at suse.de>


Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>




From Chaitanya.Kulkarni at wdc.com  Tue Jun  1 10:39:15 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue, 1 Jun 2021 17:39:15 +0000
Subject: [PATCH 1/4] nvme/loop: reset queue count to 1 in
 nvme_loop_destroy_io_queues()
References: <20210526152318.107202-1-hare@suse.de>
 <20210526152318.107202-2-hare@suse.de>
Message-ID: <DM6PR04MB4972429B02A7CC58C3807F13863E9@DM6PR04MB4972.namprd04.prod.outlook.com>

On 5/26/21 10:10, Hannes Reinecke wrote:
> The queue count is increased in nvme_loop_init_io_queues(), so we
> need to reset it to 1 at the end of nvme_loop_destroy_io_queues().
> Otherwise the function is not re-entrant safe, and crash will happen
> during concurrent reset and remove calls.
>
> Signed-off-by: Hannes Reinecke <hare at suse.de>

Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>




From Chaitanya.Kulkarni at wdc.com  Tue Jun  1 10:41:07 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue, 1 Jun 2021 17:41:07 +0000
Subject: [PATCH 2/4] nvme/loop: clear NVME_LOOP_Q_LIVE when
 nvme_loop_configure_admin_queue() fails
References: <20210526152318.107202-1-hare@suse.de>
 <20210526152318.107202-3-hare@suse.de>
Message-ID: <DM6PR04MB4972386AF5FCA1CA51B734B1863E9@DM6PR04MB4972.namprd04.prod.outlook.com>

On 5/26/21 10:07, Hannes Reinecke wrote:
> When the call to nvme_enable_ctrl() in nvme_loop_configure_admin_queue()
> fails the NVME_LOOP_Q_LIVE flag is not cleared.
>
> Signed-off-by: Hannes Reinecke <hare at suse.de>

Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>




From Chaitanya.Kulkarni at wdc.com  Tue Jun  1 10:43:18 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue, 1 Jun 2021 17:43:18 +0000
Subject: [PATCH 3/4] nvme/loop: check for NVME_LOOP_Q_LIVE in
 nvme_loop_destroy_admin_queue()
References: <20210526152318.107202-1-hare@suse.de>
 <20210526152318.107202-4-hare@suse.de>
Message-ID: <DM6PR04MB4972D4B3421E2C87D15E4297863E9@DM6PR04MB4972.namprd04.prod.outlook.com>

On 5/26/21 10:11, Hannes Reinecke wrote:
> We need to check the NVME_LOOP_Q_LIVE flag in
> nvme_loop_destroy_admin_queue() to protect against duplicate
> invocations eg during concurrent reset and remove calls.
>
> Signed-off-by: Hannes Reinecke <hare at suse.de>

It will be great if we can document here what is the concurrent
reset scenario is make understanding of the patch much easy.

Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>




From Amit.Engel at Dell.com  Tue Jun  1 10:51:48 2021
From: Amit.Engel at Dell.com (Engel, Amit)
Date: Tue, 1 Jun 2021 17:51:48 +0000
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at
 0000000000000230
Message-ID: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>

Hello,

We hit the below kernel panic ?BUG: unable to handle kernel NULL pointer dereference at 0000000000000230?
when running with RHEL8.3 host. This happens after we reboot the target side application (when multiple ctrls/connections exist)
Based on vmcore analysis it seems that when nvme_tcp_restore_sock_calls is called (from __nvme_tcp_stop_queue)
queue->sock is NULL

Are you familiar with such an issue ?

crash> bt
PID: 193053  TASK: ffff9491bdad17c0  CPU: 7   COMMAND: "kworker/u193:9"
 #0 [ffffb2e9cfdbbb70] machine_kexec at ffffffffb245bf3e
 #1 [ffffb2e9cfdbbbc8] __crash_kexec at ffffffffb256072d
 #2 [ffffb2e9cfdbbc90] crash_kexec at ffffffffb256160d
 #3 [ffffb2e9cfdbbca8] oops_end at ffffffffb2422d4d
 #4 [ffffb2e9cfdbbcc8] no_context at ffffffffb246ba9e
 #5 [ffffb2e9cfdbbd20] do_page_fault at ffffffffb246c5c2
 #6 [ffffb2e9cfdbbd50] page_fault at ffffffffb2e0122e
    [exception RIP: _raw_write_lock_bh+23]
    RIP: ffffffffb2cd6cc7  RSP: ffffb2e9cfdbbe00  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff94b2aefb4000  RCX: 0000000000000003
    RDX: 00000000000000ff  RSI: 00000000fffffe01  RDI: 0000000000000230
    RBP: ffff94923f793f40   R8: ffff9492ff1ea7f8   R9: 0000000000000000
    R10: 0000000000000000  R11: ffff9492ff1e8c64  R12: ffff94b2b7210338
    R13: 0000000000000000  R14: ffff94b27f7a4100  R15: ffff94b2b72110a0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffffb2e9cfdbbe00] __nvme_tcp_stop_queue at ffffffffc02dc0aa [nvme_tcp]
 #8 [ffffb2e9cfdbbe18] nvme_tcp_start_queue at ffffffffc02dcd18 [nvme_tcp]
 #9 [ffffb2e9cfdbbe38] nvme_tcp_setup_ctrl at ffffffffc02df258 [nvme_tcp]
#10 [ffffb2e9cfdbbe80] nvme_tcp_reconnect_ctrl_work at ffffffffc02df4bf [nvme_tcp]
#11 [ffffb2e9cfdbbe98] process_one_work at ffffffffb24d3477
#12 [ffffb2e9cfdbbed8] worker_thread at ffffffffb24d3b40
#13 [ffffb2e9cfdbbf10] kthread at ffffffffb24d9502
#14 [ffffb2e9cfdbbf50] ret_from_fork at ffffffffb2e00255


crash> bt -l
PID: 193053  TASK: ffff9491bdad17c0  CPU: 7   COMMAND: "kworker/u193:9"
 #0 [ffffb2e9cfdbbb70] machine_kexec at ffffffffb245bf3e
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/kernel/machine_kexec_64.c: 389
 #1 [ffffb2e9cfdbbbc8] __crash_kexec at ffffffffb256072d
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/kernel/kexec_core.c: 956
 #2 [ffffb2e9cfdbbc90] crash_kexec at ffffffffb256160d
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./include/linux/compiler.h: 219
 #3 [ffffb2e9cfdbbca8] oops_end at ffffffffb2422d4d
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/kernel/dumpstack.c: 334
 #4 [ffffb2e9cfdbbcc8] no_context at ffffffffb246ba9e
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/mm/fault.c: 773
 #5 [ffffb2e9cfdbbd20] do_page_fault at ffffffffb246c5c2
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./arch/x86/include/asm/jump_label.h: 38
 #6 [ffffb2e9cfdbbd50] page_fault at ffffffffb2e0122e
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/entry/entry_64.S: 1183
    [exception RIP: _raw_write_lock_bh+23]
    RIP: ffffffffb2cd6cc7  RSP: ffffb2e9cfdbbe00  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff94b2aefb4000  RCX: 0000000000000003
    RDX: 00000000000000ff  RSI: 00000000fffffe01  RDI: 0000000000000230
    RBP: ffff94923f793f40   R8: ffff9492ff1ea7f8   R9: 0000000000000000
    R10: 0000000000000000  R11: ffff9492ff1e8c64  R12: ffff94b2b7210338
    R13: 0000000000000000  R14: ffff94b27f7a4100  R15: ffff94b2b72110a0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./arch/x86/include/asm/atomic.h: 194
 #7 [ffffb2e9cfdbbe00] __nvme_tcp_stop_queue at ffffffffc02dc0aa [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1486
 #8 [ffffb2e9cfdbbe18] nvme_tcp_start_queue at ffffffffc02dcd18 [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1525
 #9 [ffffb2e9cfdbbe38] nvme_tcp_setup_ctrl at ffffffffc02df258 [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1814
#10 [ffffb2e9cfdbbe80] nvme_tcp_reconnect_ctrl_work at ffffffffc02df4bf [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1962
#11 [ffffb2e9cfdbbe98] process_one_work at ffffffffb24d3477
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./arch/x86/include/asm/jump_label.h: 38
#12 [ffffb2e9cfdbbed8] worker_thread at ffffffffb24d3b40
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./include/linux/compiler.h: 193
#13 [ffffb2e9cfdbbf10] kthread at ffffffffb24d9502
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/kernel/kthread.c: 280
#14 [ffffb2e9cfdbbf50] ret_from_fork at ffffffffb2e00255
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/entry/entry_64.S: 360

nvme/host/tcp.c
..snip
1481 static void nvme_tcp_restore_sock_calls(struct nvme_tcp_queue *queue)
1482 {
1483 >-------struct socket *sock = queue->sock;
1484 
1485 >-------write_lock_bh(&sock->sk->sk_callback_lock);
1486 >-------sock->sk->sk_user_data  = NULL;
1487 >-------sock->sk->sk_data_ready = queue->data_ready;
1488 >-------sock->sk->sk_state_change = queue->state_change;
1489 >-------sock->sk->sk_write_space  = queue->write_space;
1490 >-------write_unlock_bh(&sock->sk->sk_callback_lock);
1491 }
..snip


NULL pointer dereference at 0x230 ? 560 decimal
crash> struct sock -o
struct sock {
   [0] struct sock_common __sk_common;
   ?
   ...
   ?
   [560] rwlock_t sk_callback_lock;


From Chaitanya.Kulkarni at wdc.com  Tue Jun  1 11:25:55 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue, 1 Jun 2021 18:25:55 +0000
Subject: [PATCH 1/1] nvmet: fix freeing unallocated p2pmem
References: <20210601162205.38326-1-mgurtovoy@nvidia.com>
Message-ID: <DM6PR04MB4972308EB90B231859C1C83B863E9@DM6PR04MB4972.namprd04.prod.outlook.com>

On 6/1/21 09:22, Max Gurtovoy wrote:
> In case p2p device was found but the p2p pool is empty, the nvme target
> is still trying to free the sgl from the p2p pool instead of the
> regular sgl pool and causing a crash (BUG() is called). Instead, assign
> the p2p_dev for the request only if it was allocated from p2p pool.
>
> This is the crash that was caused:
>
> [Sun May 30 19:13:53 2021] ------------[ cut here ]------------
> [Sun May 30 19:13:53 2021] kernel BUG at lib/genalloc.c:518!
> [Sun May 30 19:13:53 2021] invalid opcode: 0000 [#1] SMP PTI
> ...
> [Sun May 30 19:13:53 2021] kernel BUG at lib/genalloc.c:518!
> ...
> [Sun May 30 19:13:53 2021] RIP: 0010:gen_pool_free_owner+0xa8/0xb0
> ...
> [Sun May 30 19:13:53 2021] Call Trace:
> [Sun May 30 19:13:53 2021] ------------[ cut here ]------------
> [Sun May 30 19:13:53 2021]  pci_free_p2pmem+0x2b/0x70
> [Sun May 30 19:13:53 2021]  pci_p2pmem_free_sgl+0x4f/0x80
> [Sun May 30 19:13:53 2021]  nvmet_req_free_sgls+0x1e/0x80 [nvmet]
> [Sun May 30 19:13:53 2021] kernel BUG at lib/genalloc.c:518!
> [Sun May 30 19:13:53 2021]  nvmet_rdma_release_rsp+0x4e/0x1f0 [nvmet_rdma]
> [Sun May 30 19:13:53 2021]  nvmet_rdma_send_done+0x1c/0x60 [nvmet_rdma]
>
> Fixes: c6e3f1339812 ("nvmet: add metadata support for block devices")
> Reviewed-by: Israel Rukshin <israelr at nvidia.com>
> Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>

Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>





From david.e.box at linux.intel.com  Tue Jun  1 12:12:26 2021
From: david.e.box at linux.intel.com (David E. Box)
Date: Tue, 01 Jun 2021 12:12:26 -0700
Subject: [PATCH v3 1/2] nvme: Look for StorageD3Enable on companion ACPI
 device instead
In-Reply-To: <20210531064655.GA1417@lst.de>
References: <20210528160234.9402-1-mario.limonciello@amd.com>
 <20210531064655.GA1417@lst.de>
Message-ID: <108f45a7c73d3ae571ee518c33dc793f36c1ffd2.camel@linux.intel.com>

On Mon, 2021-05-31 at 08:46 +0200, Christoph Hellwig wrote:
> On Fri, May 28, 2021 at 11:02:34AM -0500, Mario Limonciello wrote:
> > The documentation around the StorageD3Enable property hints that it
> > should be made on the PCI device.? This is where newer AMD systems
> > set
> > the property and it's required for S0i3 support.
> > 
> > So rather than look for nodes of the root port only present on
> > Intel
> > systems, switch to the companion ACPI device for all systems.
> > David Box from Intel indicated this should work on Intel as well.
> 
> I think we need to wait for the confirmation from David.

I've tested one configuration remotely already and it works. I've got
one more to test. I'll know by tomorrow.

David




From belegdol at gmail.com  Tue Jun  1 22:55:07 2021
From: belegdol at gmail.com (Julian Sikorski)
Date: Wed, 2 Jun 2021 07:55:07 +0200
Subject: [PATCH v3 0/2] Improvements to StorageD3Enable
In-Reply-To: <20210528160120.9299-1-mario.limonciello@amd.com>
References: <20210528160120.9299-1-mario.limonciello@amd.com>
Message-ID: <46e8a6a1-4dd3-9aa6-4131-6b7eaff44b64@gmail.com>

Am 28.05.21 um 18:01 schrieb Mario Limonciello:
> A number of AMD based OEM systems have problems coming out of s2idle,
> which is rooted in that the NVME device power is cut off during s2idle.
> 
> That alone is not a bug - the architecture used on Cezanne, Renoir and
> Picasso expects this.
> 
> Many of these systems do include the StorageD3Enable property, but it is
> located in the PCI device itself not in a root port sibling like on Intel.
> 
> Intel confirmed that this during pre-production it was placed there, and
> actually for production using the PCI device itself is sufficient.
> 
> During the course of discussions on the merits of different approaches it
> was mentioned that although originally introduced for NVME devices, the
> Microsoft specification makes allusions to non-PCI based ACPI storage
> devices as well, so a proposal was created to move this into the ACPI
> subsystem.
> 
> If at a later time different firmware solutions decide to advertise this
> functionality, it may make sense to move out of acpi into a more generic
> location.  However both AMD's and Intel's solutions for s2idle also rely
> upon calling other ACPI drivers and adopting another solution will require
> coming up with alternatives for those as well.
> 
> Mario Limonciello (2):
>    nvme: Look for StorageD3Enable on companion ACPI device instead
>    acpi: Move check for _DSD StorageD3Enable property to acpi
> 
>   drivers/acpi/device_pm.c | 24 +++++++++++++++++++
>   drivers/nvme/host/pci.c  | 50 +---------------------------------------
>   include/linux/acpi.h     |  5 ++++
>   3 files changed, 30 insertions(+), 49 deletions(-)
> 
Are these patches meant to replace the blanket Renoir simple suspend patch?
https://gitlab.com/belegdol/kernel-ark/-/commit/a83315923e87a169ceb6c839a84ee0455868218f
I can test it on an Asus Zenbook 14 UM425IA if needed, you should be 
also be able to get a considerable amount of feedback from 
https://gitlab.freedesktop.org/drm/amd/-/issues/1230

Best regards,
Julian


From hch at lst.de  Wed Jun  2 00:07:08 2021
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 2 Jun 2021 09:07:08 +0200
Subject: [PATCH 0/4] nvme-loop: fixes for concurrent reset and delete
In-Reply-To: <20210526152318.107202-1-hare@suse.de>
References: <20210526152318.107202-1-hare@suse.de>
Message-ID: <20210602070708.GA10115@lst.de>

Thanks,

applied to nvme-5.13.


From hch at lst.de  Wed Jun  2 00:11:09 2021
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 2 Jun 2021 09:11:09 +0200
Subject: [PATCH 1/1] nvmet: fix freeing unallocated p2pmem
In-Reply-To: <20210601162205.38326-1-mgurtovoy@nvidia.com>
References: <20210601162205.38326-1-mgurtovoy@nvidia.com>
Message-ID: <20210602071109.GA10133@lst.de>

Applied including the edit to streamline nvmet_req_find_p2p_dev.


From mgurtovoy at nvidia.com  Wed Jun  2 04:10:52 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Wed, 2 Jun 2021 14:10:52 +0300
Subject: [RFC PATCH 0/3] PCI/P2PMEM: introduce pci_p2pdma_align_size API
Message-ID: <20210602111055.10480-1-mgurtovoy@nvidia.com>

Hi all,
In hotplugged memory (from check_pfn_span function):
"
Disallow all operations smaller than a sub-section and only
allow operations smaller than a section for
SPARSEMEM_VMEMMAP. Note that check_hotplug_memory_range()
enforces a larger memory_block_size_bytes() granularity for
memory that will be marked online, so this check should only
fire for direct arch_{add,remove}_memory() users outside of
add_memory_resource()
"

This restriction will disqualify, for example, large NVMe CMBs that might have
non power of 2 number of pages (e.g. 32767 pages of 4KB). For these
devices, the CMB size will be rounded down from 0x7fff000 to 0x7e00000
but it's better than having un-mapped CMB.

If we all agree on the approach, this RFC can assist as-is to these NVMe
devices and other P2PMEM devices in the future and can considered for
the next merge window.

Max Gurtovoy (3):
  mm,memory_hotplug: export mhp min alignment
  PCI/P2PMEM: introduce pci_p2pdma_align_size API
  nvme-pci: align CMB size according to P2PMEM alignment

 drivers/nvme/host/pci.c        |  7 +++++++
 drivers/pci/p2pdma.c           | 23 +++++++++++++++++++++++
 include/linux/memory_hotplug.h |  5 +++++
 include/linux/pci-p2pdma.h     |  5 +++++
 mm/memory_hotplug.c            | 33 +++++++++++++++++++--------------
 5 files changed, 59 insertions(+), 14 deletions(-)

-- 
2.18.1



From mgurtovoy at nvidia.com  Wed Jun  2 04:10:53 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Wed, 2 Jun 2021 14:10:53 +0300
Subject: [PATCH 1/3] mm,memory_hotplug: export mhp min alignment
In-Reply-To: <20210602111055.10480-1-mgurtovoy@nvidia.com>
References: <20210602111055.10480-1-mgurtovoy@nvidia.com>
Message-ID: <20210602111055.10480-2-mgurtovoy@nvidia.com>

Hotplugged memory has alignmet restrictions. E.g, it disallows all
operations smaller than a sub-section and only allow operations smaller
than a section for SPARSEMEM_VMEMMAP. Export the alignment restrictions
for mhp users.

Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
---
 include/linux/memory_hotplug.h |  5 +++++
 mm/memory_hotplug.c            | 33 +++++++++++++++++++--------------
 2 files changed, 24 insertions(+), 14 deletions(-)

diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 28f32fd00fe9..c55a9049b11e 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -76,6 +76,7 @@ struct mhp_params {
 
 bool mhp_range_allowed(u64 start, u64 size, bool need_mapping);
 struct range mhp_get_pluggable_range(bool need_mapping);
+unsigned long mhp_get_min_align(void);
 
 /*
  * Zone resizing functions
@@ -248,6 +249,10 @@ void mem_hotplug_done(void);
 	___page;				\
  })
 
+static inline unsigned long mhp_get_min_align(void)
+{
+	return 0;
+}
 static inline unsigned zone_span_seqbegin(struct zone *zone)
 {
 	return 0;
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 9e86e9ee0a10..161bb6704a9b 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -270,24 +270,29 @@ void __init register_page_bootmem_info_node(struct pglist_data *pgdat)
 }
 #endif /* CONFIG_HAVE_BOOTMEM_INFO_NODE */
 
+/*
+ * Disallow all operations smaller than a sub-section and only
+ * allow operations smaller than a section for
+ * SPARSEMEM_VMEMMAP. Note that check_hotplug_memory_range()
+ * enforces a larger memory_block_size_bytes() granularity for
+ * memory that will be marked online, so this check should only
+ * fire for direct arch_{add,remove}_memory() users outside of
+ * add_memory_resource().
+ */
+unsigned long mhp_get_min_align(void)
+{
+	if (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP))
+		return PAGES_PER_SUBSECTION;
+	return PAGES_PER_SECTION;
+}
+EXPORT_SYMBOL_GPL(mhp_get_min_align);
+
+
 static int check_pfn_span(unsigned long pfn, unsigned long nr_pages,
 		const char *reason)
 {
-	/*
-	 * Disallow all operations smaller than a sub-section and only
-	 * allow operations smaller than a section for
-	 * SPARSEMEM_VMEMMAP. Note that check_hotplug_memory_range()
-	 * enforces a larger memory_block_size_bytes() granularity for
-	 * memory that will be marked online, so this check should only
-	 * fire for direct arch_{add,remove}_memory() users outside of
-	 * add_memory_resource().
-	 */
-	unsigned long min_align;
+	unsigned long min_align = mhp_get_min_align();
 
-	if (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP))
-		min_align = PAGES_PER_SUBSECTION;
-	else
-		min_align = PAGES_PER_SECTION;
 	if (!IS_ALIGNED(pfn, min_align) || !IS_ALIGNED(nr_pages, min_align)) {
 		WARN(1, "Misaligned __%s_pages min_align: %#lx start: %#lx end: %#lx\n",
 		     reason, min_align, pfn, pfn + nr_pages - 1);
-- 
2.18.1



From mgurtovoy at nvidia.com  Wed Jun  2 04:10:54 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Wed, 2 Jun 2021 14:10:54 +0300
Subject: [PATCH 2/3] PCI/P2PMEM: introduce pci_p2pdma_align_size API
In-Reply-To: <20210602111055.10480-1-mgurtovoy@nvidia.com>
References: <20210602111055.10480-1-mgurtovoy@nvidia.com>
Message-ID: <20210602111055.10480-3-mgurtovoy@nvidia.com>

The mhp layer has alignment restrictions that may cause a failure in
adding a large P2PMEM that has non-aligned number of pages. The new API
pci_p2pdma_align_size will align the P2PMEM size to be inline with the
mhp restrictions.

Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
---
 drivers/pci/p2pdma.c       | 23 +++++++++++++++++++++++
 include/linux/pci-p2pdma.h |  5 +++++
 2 files changed, 28 insertions(+)

diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
index 196382630363..d232bc4ada2b 100644
--- a/drivers/pci/p2pdma.c
+++ b/drivers/pci/p2pdma.c
@@ -220,6 +220,29 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size,
 }
 EXPORT_SYMBOL_GPL(pci_p2pdma_add_resource);
 
+/**
+ * pci_p2pdma_align_size - align p2p memory size
+ * @size: size of the initial memory to align before adding.
+ *
+ * The size will be aligned according to mph alignment rules. This function
+ * should be called before adding new p2pdma resource since non-aligned memory
+ * will not be added.
+ */
+size_t pci_p2pdma_align_size(size_t size)
+{
+	unsigned int min_align, nr_pages;
+
+	min_align = mhp_get_min_align();
+	nr_pages = size >> PAGE_SHIFT;
+	if (!IS_ALIGNED(nr_pages, min_align)) {
+		nr_pages = ALIGN_DOWN(nr_pages, min_align);
+		size = nr_pages << PAGE_SHIFT;
+	}
+
+	return size;
+}
+EXPORT_SYMBOL_GPL(pci_p2pdma_align_size);
+
 /*
  * Note this function returns the parent PCI device with a
  * reference taken. It is the caller's responsibility to drop
diff --git a/include/linux/pci-p2pdma.h b/include/linux/pci-p2pdma.h
index 8318a97c9c61..33a8fce52bec 100644
--- a/include/linux/pci-p2pdma.h
+++ b/include/linux/pci-p2pdma.h
@@ -38,6 +38,7 @@ int pci_p2pdma_enable_store(const char *page, struct pci_dev **p2p_dev,
 			    bool *use_p2pdma);
 ssize_t pci_p2pdma_enable_show(char *page, struct pci_dev *p2p_dev,
 			       bool use_p2pdma);
+size_t pci_p2pdma_align_size(size_t size);
 #else /* CONFIG_PCI_P2PDMA */
 static inline int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar,
 		size_t size, u64 offset)
@@ -105,6 +106,10 @@ static inline ssize_t pci_p2pdma_enable_show(char *page,
 {
 	return sprintf(page, "none\n");
 }
+static inline size_t pci_p2pdma_align_size(size_t size)
+{
+	return 0;
+}
 #endif /* CONFIG_PCI_P2PDMA */
 
 
-- 
2.18.1



From mgurtovoy at nvidia.com  Wed Jun  2 04:10:55 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Wed, 2 Jun 2021 14:10:55 +0300
Subject: [PATCH 3/3] nvme-pci: align CMB size according to P2PMEM alignment
In-Reply-To: <20210602111055.10480-1-mgurtovoy@nvidia.com>
References: <20210602111055.10480-1-mgurtovoy@nvidia.com>
Message-ID: <20210602111055.10480-4-mgurtovoy@nvidia.com>

P2PMEM is using mhp framework to connect to the memory subsystem. In
case the CMB size is not compatible to mhp alignment, the CMB mapping
will fail. Use pci_p2pdma_align_size to align CMB size in order to
successfully map non-aligned original CMB.

Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
---
 drivers/nvme/host/pci.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index a29b170701fc..1197263b4cd0 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1817,6 +1817,7 @@ static void nvme_map_cmb(struct nvme_dev *dev)
 	u64 size, offset;
 	resource_size_t bar_size;
 	struct pci_dev *pdev = to_pci_dev(dev->dev);
+	unsigned long nr_pages, min_align;
 	int bar;
 
 	if (dev->cmb_size)
@@ -1856,6 +1857,12 @@ static void nvme_map_cmb(struct nvme_dev *dev)
 	if (size > bar_size - offset)
 		size = bar_size - offset;
 
+	size = pci_p2pdma_align_size(size);
+	if (!size) {
+		dev_warn(dev->ctrl.device, "CMB size is 0 after alignment\n");
+		return;
+	}
+
 	if (pci_p2pdma_add_resource(pdev, bar, size, offset)) {
 		dev_warn(dev->ctrl.device,
 			 "failed to register the CMB\n");
-- 
2.18.1



From david at redhat.com  Wed Jun  2 05:08:58 2021
From: david at redhat.com (David Hildenbrand)
Date: Wed, 2 Jun 2021 14:08:58 +0200
Subject: [RFC PATCH 0/3] PCI/P2PMEM: introduce pci_p2pdma_align_size API
In-Reply-To: <20210602111055.10480-1-mgurtovoy@nvidia.com>
References: <20210602111055.10480-1-mgurtovoy@nvidia.com>
Message-ID: <4bbb0934-384a-533f-b964-6b3695719b25@redhat.com>

On 02.06.21 13:10, Max Gurtovoy wrote:
> Hi all,
> In hotplugged memory (from check_pfn_span function):
> "
> Disallow all operations smaller than a sub-section and only
> allow operations smaller than a section for
> SPARSEMEM_VMEMMAP. Note that check_hotplug_memory_range()
> enforces a larger memory_block_size_bytes() granularity for
> memory that will be marked online, so this check should only
> fire for direct arch_{add,remove}_memory() users outside of
> add_memory_resource()
> "
> 
> This restriction will disqualify, for example, large NVMe CMBs that might have
> non power of 2 number of pages (e.g. 32767 pages of 4KB). For these
> devices, the CMB size will be rounded down from 0x7fff000 to 0x7e00000
> but it's better than having un-mapped CMB.

Just some high-level questions:

A CMB is just a PCI BAR used for communicating with the device, to be 
mapped into physical address space, right? I assume the relevant hotplug 
code is:

drivers/pci/p2pdma.c:   addr = devm_memremap_pages(&pdev->dev, pgmap);

correct?


Having a BAR span such weird sizes will most probably never be fully 
supported. But if sub-sections work for you, great.



-- 
Thanks,

David / dhildenb



From david at redhat.com  Wed Jun  2 05:14:48 2021
From: david at redhat.com (David Hildenbrand)
Date: Wed, 2 Jun 2021 14:14:48 +0200
Subject: [PATCH 1/3] mm,memory_hotplug: export mhp min alignment
In-Reply-To: <20210602111055.10480-2-mgurtovoy@nvidia.com>
References: <20210602111055.10480-1-mgurtovoy@nvidia.com>
 <20210602111055.10480-2-mgurtovoy@nvidia.com>
Message-ID: <283740c3-db3f-3c9a-2954-f1c037a13e86@redhat.com>

On 02.06.21 13:10, Max Gurtovoy wrote:
> Hotplugged memory has alignmet restrictions. E.g, it disallows all
> operations smaller than a sub-section and only allow operations smaller
> than a section for SPARSEMEM_VMEMMAP. Export the alignment restrictions
> for mhp users.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
> ---
>   include/linux/memory_hotplug.h |  5 +++++
>   mm/memory_hotplug.c            | 33 +++++++++++++++++++--------------
>   2 files changed, 24 insertions(+), 14 deletions(-)
> 
> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
> index 28f32fd00fe9..c55a9049b11e 100644
> --- a/include/linux/memory_hotplug.h
> +++ b/include/linux/memory_hotplug.h
> @@ -76,6 +76,7 @@ struct mhp_params {
>   
>   bool mhp_range_allowed(u64 start, u64 size, bool need_mapping);
>   struct range mhp_get_pluggable_range(bool need_mapping);
> +unsigned long mhp_get_min_align(void);
>   
>   /*
>    * Zone resizing functions
> @@ -248,6 +249,10 @@ void mem_hotplug_done(void);
>   	___page;				\
>    })
>   
> +static inline unsigned long mhp_get_min_align(void)
> +{
> +	return 0;
> +}
>   static inline unsigned zone_span_seqbegin(struct zone *zone)
>   {
>   	return 0;
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 9e86e9ee0a10..161bb6704a9b 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -270,24 +270,29 @@ void __init register_page_bootmem_info_node(struct pglist_data *pgdat)
>   }
>   #endif /* CONFIG_HAVE_BOOTMEM_INFO_NODE */
>   
> +/*
> + * Disallow all operations smaller than a sub-section and only
> + * allow operations smaller than a section for
> + * SPARSEMEM_VMEMMAP. Note that check_hotplug_memory_range()
> + * enforces a larger memory_block_size_bytes() granularity for
> + * memory that will be marked online, so this check should only
> + * fire for direct arch_{add,remove}_memory() users outside of
> + * add_memory_resource().
> + */
> +unsigned long mhp_get_min_align(void)
> +{
> +	if (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP))
> +		return PAGES_PER_SUBSECTION;
> +	return PAGES_PER_SECTION;
> +}
> +EXPORT_SYMBOL_GPL(mhp_get_min_align);

We have to main interfaces to "hotplug" memory:

a) add_memory() and friends for System RAM, which have memory block 
alignment requirements.

b) memremap_pages(), which has the alignemnt requirements you mention here.

I feel like what you need would better be exposed in mm/memremap.c, for 
example, via "memremap_min_alignment" so it matches the "memremap_pages" 
semantics.

And then, memremap_pages() is only available with CONFIG_ZONE_DEVICE, 
which depends on SPARSEMEM_VMEMMAP. So you'll always have 
PAGES_PER_SUBSECTION.

I can already spot "memremap_compat_align", maybe you can reuse that or 
handle it accordingly in there?


-- 
Thanks,

David / dhildenb



From Amit.Engel at Dell.com  Wed Jun  2 05:28:46 2021
From: Amit.Engel at Dell.com (Engel, Amit)
Date: Wed, 2 Jun 2021 12:28:46 +0000
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at
 0000000000000230
In-Reply-To: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>
References: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>
Message-ID: <CO1PR19MB4885314DCBFEB376C7305C98EE3D9@CO1PR19MB4885.namprd19.prod.outlook.com>

Hi Sagi,

A correction to the below analysis:
It seems like sock->sk is NULL and not queue->sock

As part of _nvme_tcp_stop_queue
kernel_sock_shutdown and nvme_tcp_restore_sock_calls are being called:
kernel_sock_shutdown leads to nvme_tcp_state_change which will trigger err_work (nvme_tcp_error_recovery_work)

As part of nvme_tcp_error_recovery_work, nvme_tcp_free_queue is being called which releases the socket (sock_release)

In our case, based on the below bt:
nvme_tcp_error_recovery_work is being triggered (and so sock_release) before nvme_tcp_restore_sock_calls , which end up with NULL dereference pointer at 'rwlock_t sk_callback_lock' ?

Can you please review and provide your inputs for this potential race ?

Thanks
Amit Engel

-----Original Message-----
From: Engel, Amit 
Sent: Tuesday, June 1, 2021 8:52 PM
To: linux-nvme at lists.infradead.org; sagi at grimberg.me
Cc: Engel, Amit
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at 0000000000000230

Hello,

We hit the below kernel panic ?BUG: unable to handle kernel NULL pointer dereference at 0000000000000230?
when running with RHEL8.3 host. This happens after we reboot the target side application (when multiple ctrls/connections exist) Based on vmcore analysis it seems that when nvme_tcp_restore_sock_calls is called (from __nvme_tcp_stop_queue)
queue->sock is NULL

Are you familiar with such an issue ?

crash> bt
PID: 193053  TASK: ffff9491bdad17c0  CPU: 7   COMMAND: "kworker/u193:9"
 #0 [ffffb2e9cfdbbb70] machine_kexec at ffffffffb245bf3e
 #1 [ffffb2e9cfdbbbc8] __crash_kexec at ffffffffb256072d
 #2 [ffffb2e9cfdbbc90] crash_kexec at ffffffffb256160d
 #3 [ffffb2e9cfdbbca8] oops_end at ffffffffb2422d4d
 #4 [ffffb2e9cfdbbcc8] no_context at ffffffffb246ba9e
 #5 [ffffb2e9cfdbbd20] do_page_fault at ffffffffb246c5c2
 #6 [ffffb2e9cfdbbd50] page_fault at ffffffffb2e0122e
    [exception RIP: _raw_write_lock_bh+23]
    RIP: ffffffffb2cd6cc7  RSP: ffffb2e9cfdbbe00  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff94b2aefb4000  RCX: 0000000000000003
    RDX: 00000000000000ff  RSI: 00000000fffffe01  RDI: 0000000000000230
    RBP: ffff94923f793f40   R8: ffff9492ff1ea7f8   R9: 0000000000000000
    R10: 0000000000000000  R11: ffff9492ff1e8c64  R12: ffff94b2b7210338
    R13: 0000000000000000  R14: ffff94b27f7a4100  R15: ffff94b2b72110a0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffffb2e9cfdbbe00] __nvme_tcp_stop_queue at ffffffffc02dc0aa [nvme_tcp]
 #8 [ffffb2e9cfdbbe18] nvme_tcp_start_queue at ffffffffc02dcd18 [nvme_tcp]
 #9 [ffffb2e9cfdbbe38] nvme_tcp_setup_ctrl at ffffffffc02df258 [nvme_tcp]
#10 [ffffb2e9cfdbbe80] nvme_tcp_reconnect_ctrl_work at ffffffffc02df4bf [nvme_tcp]
#11 [ffffb2e9cfdbbe98] process_one_work at ffffffffb24d3477
#12 [ffffb2e9cfdbbed8] worker_thread at ffffffffb24d3b40
#13 [ffffb2e9cfdbbf10] kthread at ffffffffb24d9502
#14 [ffffb2e9cfdbbf50] ret_from_fork at ffffffffb2e00255


crash> bt -l
PID: 193053  TASK: ffff9491bdad17c0  CPU: 7   COMMAND: "kworker/u193:9"
 #0 [ffffb2e9cfdbbb70] machine_kexec at ffffffffb245bf3e
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/kernel/machine_kexec_64.c: 389
 #1 [ffffb2e9cfdbbbc8] __crash_kexec at ffffffffb256072d
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/kernel/kexec_core.c: 956
 #2 [ffffb2e9cfdbbc90] crash_kexec at ffffffffb256160d
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./include/linux/compiler.h: 219
 #3 [ffffb2e9cfdbbca8] oops_end at ffffffffb2422d4d
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/kernel/dumpstack.c: 334
 #4 [ffffb2e9cfdbbcc8] no_context at ffffffffb246ba9e
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/mm/fault.c: 773
 #5 [ffffb2e9cfdbbd20] do_page_fault at ffffffffb246c5c2
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./arch/x86/include/asm/jump_label.h: 38
 #6 [ffffb2e9cfdbbd50] page_fault at ffffffffb2e0122e
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/entry/entry_64.S: 1183
    [exception RIP: _raw_write_lock_bh+23]
    RIP: ffffffffb2cd6cc7  RSP: ffffb2e9cfdbbe00  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff94b2aefb4000  RCX: 0000000000000003
    RDX: 00000000000000ff  RSI: 00000000fffffe01  RDI: 0000000000000230
    RBP: ffff94923f793f40   R8: ffff9492ff1ea7f8   R9: 0000000000000000
    R10: 0000000000000000  R11: ffff9492ff1e8c64  R12: ffff94b2b7210338
    R13: 0000000000000000  R14: ffff94b27f7a4100  R15: ffff94b2b72110a0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./arch/x86/include/asm/atomic.h: 194
 #7 [ffffb2e9cfdbbe00] __nvme_tcp_stop_queue at ffffffffc02dc0aa [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1486
 #8 [ffffb2e9cfdbbe18] nvme_tcp_start_queue at ffffffffc02dcd18 [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1525
 #9 [ffffb2e9cfdbbe38] nvme_tcp_setup_ctrl at ffffffffc02df258 [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1814
#10 [ffffb2e9cfdbbe80] nvme_tcp_reconnect_ctrl_work at ffffffffc02df4bf [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1962
#11 [ffffb2e9cfdbbe98] process_one_work at ffffffffb24d3477
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./arch/x86/include/asm/jump_label.h: 38
#12 [ffffb2e9cfdbbed8] worker_thread at ffffffffb24d3b40
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./include/linux/compiler.h: 193
#13 [ffffb2e9cfdbbf10] kthread at ffffffffb24d9502
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/kernel/kthread.c: 280
#14 [ffffb2e9cfdbbf50] ret_from_fork at ffffffffb2e00255
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/entry/entry_64.S: 360

nvme/host/tcp.c
..snip
1481 static void nvme_tcp_restore_sock_calls(struct nvme_tcp_queue *queue)
1482 {
1483 >-------struct socket *sock = queue->sock;
1484
1485 >-------write_lock_bh(&sock->sk->sk_callback_lock);
1486 >-------sock->sk->sk_user_data  = NULL;
1487 >-------sock->sk->sk_data_ready = queue->data_ready;
1488 >-------sock->sk->sk_state_change = queue->state_change;
1489 >-------sock->sk->sk_write_space  = queue->write_space;
1490 >-------write_unlock_bh(&sock->sk->sk_callback_lock);
1491 }
..snip


NULL pointer dereference at 0x230 ? 560 decimal
crash> struct sock -o
struct sock {
   [0] struct sock_common __sk_common;
   ?
   ...
   ?
   [560] rwlock_t sk_callback_lock;


From kbusch at kernel.org  Wed Jun  2 07:39:40 2021
From: kbusch at kernel.org (Keith Busch)
Date: Wed, 2 Jun 2021 07:39:40 -0700
Subject: [PATCH 3/3] nvme-pci: align CMB size according to P2PMEM alignment
In-Reply-To: <20210602111055.10480-4-mgurtovoy@nvidia.com>
References: <20210602111055.10480-1-mgurtovoy@nvidia.com>
 <20210602111055.10480-4-mgurtovoy@nvidia.com>
Message-ID: <20210602143940.GA5751@dhcp-10-100-145-180.wdc.com>

On Wed, Jun 02, 2021 at 02:10:55PM +0300, Max Gurtovoy wrote:
> P2PMEM is using mhp framework to connect to the memory subsystem. In
> case the CMB size is not compatible to mhp alignment, the CMB mapping
> will fail. Use pci_p2pdma_align_size to align CMB size in order to
> successfully map non-aligned original CMB.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
> ---
>  drivers/nvme/host/pci.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index a29b170701fc..1197263b4cd0 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -1817,6 +1817,7 @@ static void nvme_map_cmb(struct nvme_dev *dev)
>  	u64 size, offset;
>  	resource_size_t bar_size;
>  	struct pci_dev *pdev = to_pci_dev(dev->dev);
> +	unsigned long nr_pages, min_align;

These new variables don't appear to be used anywhere.

>  	int bar;
>  
>  	if (dev->cmb_size)
> @@ -1856,6 +1857,12 @@ static void nvme_map_cmb(struct nvme_dev *dev)
>  	if (size > bar_size - offset)
>  		size = bar_size - offset;
>  
> +	size = pci_p2pdma_align_size(size);
> +	if (!size) {
> +		dev_warn(dev->ctrl.device, "CMB size is 0 after alignment\n");
> +		return;
> +	}
> +
>  	if (pci_p2pdma_add_resource(pdev, bar, size, offset)) {
>  		dev_warn(dev->ctrl.device,
>  			 "failed to register the CMB\n");
> -- 


From mgurtovoy at nvidia.com  Wed Jun  2 07:51:05 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Wed, 2 Jun 2021 17:51:05 +0300
Subject: [PATCH 3/3] nvme-pci: align CMB size according to P2PMEM alignment
In-Reply-To: <20210602143940.GA5751@dhcp-10-100-145-180.wdc.com>
References: <20210602111055.10480-1-mgurtovoy@nvidia.com>
 <20210602111055.10480-4-mgurtovoy@nvidia.com>
 <20210602143940.GA5751@dhcp-10-100-145-180.wdc.com>
Message-ID: <9ff88c55-2e9c-3cec-81a5-b47b344815a4@nvidia.com>


On 6/2/2021 5:39 PM, Keith Busch wrote:
> On Wed, Jun 02, 2021 at 02:10:55PM +0300, Max Gurtovoy wrote:
>> P2PMEM is using mhp framework to connect to the memory subsystem. In
>> case the CMB size is not compatible to mhp alignment, the CMB mapping
>> will fail. Use pci_p2pdma_align_size to align CMB size in order to
>> successfully map non-aligned original CMB.
>>
>> Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
>> ---
>>   drivers/nvme/host/pci.c | 7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
>> index a29b170701fc..1197263b4cd0 100644
>> --- a/drivers/nvme/host/pci.c
>> +++ b/drivers/nvme/host/pci.c
>> @@ -1817,6 +1817,7 @@ static void nvme_map_cmb(struct nvme_dev *dev)
>>   	u64 size, offset;
>>   	resource_size_t bar_size;
>>   	struct pci_dev *pdev = to_pci_dev(dev->dev);
>> +	unsigned long nr_pages, min_align;
> These new variables don't appear to be used anywhere.

oh, I have some leftovers..

Sorry.


>
>>   	int bar;
>>   
>>   	if (dev->cmb_size)
>> @@ -1856,6 +1857,12 @@ static void nvme_map_cmb(struct nvme_dev *dev)
>>   	if (size > bar_size - offset)
>>   		size = bar_size - offset;
>>   
>> +	size = pci_p2pdma_align_size(size);
>> +	if (!size) {
>> +		dev_warn(dev->ctrl.device, "CMB size is 0 after alignment\n");
>> +		return;
>> +	}
>> +
>>   	if (pci_p2pdma_add_resource(pdev, bar, size, offset)) {
>>   		dev_warn(dev->ctrl.device,
>>   			 "failed to register the CMB\n");
>> -- 


From dwagner at suse.de  Wed Jun  2 05:49:19 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Wed, 2 Jun 2021 14:49:19 +0200
Subject: [PATCH] nvme: reset disk to the mpath node also when requeuing
In-Reply-To: <20210602123429.103935-1-dwagner@suse.de>
References: <20210602123429.103935-1-dwagner@suse.de>
Message-ID: <20210602124919.gdzorzhbjitjr4uj@beryllium.lan>

On Wed, Jun 02, 2021 at 02:34:29PM +0200, Daniel Wagner wrote:
>  		spin_lock_irq(&head->requeue_lock);
> +		bio_set_dev(bio, head->disk->part0);

Obviously, this needs to be done for all bios. Let me send an updated
version.x


From dwagner at suse.de  Wed Jun  2 05:34:29 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Wed,  2 Jun 2021 14:34:29 +0200
Subject: [PATCH] nvme: reset disk to the mpath node also when requeuing
Message-ID: <20210602123429.103935-1-dwagner@suse.de>

Commit ce86dad222e9 ("nvme-multipath: reset bdev to ns head when
failover") moved the reset code where the bio is added to the
requeue_list for the failover path. But it left the original
bio_set_dev() in nvme_requeue_work().

Let's move the bio_set_dev() where we add the bio to the requeue_list
and avoid the double setting in case of the failover path. And this
makes the code more consistent.

Signed-off-by: Daniel Wagner <dwagner at suse.de>
---
 drivers/nvme/host/multipath.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 127a17b4c13d..df07fc2ad7bc 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -324,6 +324,7 @@ static blk_qc_t nvme_ns_head_submit_bio(struct bio *bio)
 		dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");
 
 		spin_lock_irq(&head->requeue_lock);
+		bio_set_dev(bio, head->disk->part0);
 		bio_list_add(&head->requeue_list, bio);
 		spin_unlock_irq(&head->requeue_lock);
 	} else {
@@ -435,11 +436,6 @@ static void nvme_requeue_work(struct work_struct *work)
 		next = bio->bi_next;
 		bio->bi_next = NULL;
 
-		/*
-		 * Reset disk to the mpath node and resubmit to select a new
-		 * path.
-		 */
-		bio_set_dev(bio, head->disk->part0);
 		submit_bio_noacct(bio);
 	}
 }
-- 
2.29.2



From dwagner at suse.de  Wed Jun  2 06:00:39 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Wed,  2 Jun 2021 15:00:39 +0200
Subject: [PATCH v2] nvme: reset disk to the mpath node also when requeuing
Message-ID: <20210602130039.122879-1-dwagner@suse.de>

Commit ce86dad222e9 ("nvme-multipath: reset bdev to ns head when
failover") moved the reset code where the bio is added to the
requeue_list for the failover path. But it left the original
bio_set_dev() in nvme_requeue_work().

Let's move the bio_set_dev() where we add the bio to the requeue_list
and avoid the double setting in case of the failover path. And this
makes the code more consistent.

Signed-off-by: Daniel Wagner <dwagner at suse.de>
---

v2: loop over all bios and reset them

 drivers/nvme/host/multipath.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 127a17b4c13d..3334c6f23476 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -302,6 +302,7 @@ static blk_qc_t nvme_ns_head_submit_bio(struct bio *bio)
 	struct nvme_ns_head *head = bio->bi_bdev->bd_disk->private_data;
 	struct device *dev = disk_to_dev(head->disk);
 	struct nvme_ns *ns;
+	struct bio *b;
 	blk_qc_t ret = BLK_QC_T_NONE;
 	int srcu_idx;
 
@@ -324,6 +325,8 @@ static blk_qc_t nvme_ns_head_submit_bio(struct bio *bio)
 		dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");
 
 		spin_lock_irq(&head->requeue_lock);
+		for (b = bio; b; b = b->bi_next)
+			bio_set_dev(b, head->disk->part0);
 		bio_list_add(&head->requeue_list, bio);
 		spin_unlock_irq(&head->requeue_lock);
 	} else {
@@ -435,11 +438,6 @@ static void nvme_requeue_work(struct work_struct *work)
 		next = bio->bi_next;
 		bio->bi_next = NULL;
 
-		/*
-		 * Reset disk to the mpath node and resubmit to select a new
-		 * path.
-		 */
-		bio_set_dev(bio, head->disk->part0);
 		submit_bio_noacct(bio);
 	}
 }
-- 
2.29.2



From malin1024 at gmail.com  Wed Jun  2 09:58:36 2021
From: malin1024 at gmail.com (Shai Malin)
Date: Wed, 2 Jun 2021 19:58:36 +0300
Subject: [RFC PATCH v7 00/27] NVMeTCP Offload ULP and QEDN Device Driver
In-Reply-To: <20210531225222.16992-1-smalin@marvell.com>
References: <20210531225222.16992-1-smalin@marvell.com>
Message-ID: <CAKKgK4z7o1f-PJ7Ta_fGAhxMie_ftB2UV9Phy2KjrEh5WFx9mA@mail.gmail.com>

On 6/1/21 1:53 AM, Shai Malin wrote:
> With the goal of enabling a generic infrastructure that allows NVMe/TCP
> offload devices like NICs to seamlessly plug into the NVMe-oF stack, this
> patch series introduces the nvme-tcp-offload ULP host layer, which will
> be a new transport type called "tcp-offload" and will serve as an
> abstraction layer to work with vendor specific nvme-tcp offload drivers.

If there are no additional comments on RFC v7, I will split the RFC series
between the following non-RFC sub-series:

- Part 1 (RFC patch 1-8): NVMeTCP Offload ULP
  The nvme-tcp-offload patches will be sent to
  'linux-nvme at lists.infradead.org'.

- Part 2 (RFC patches 9-15): QED NVMeTCP Offload
  The qed infrastructure will be sent to 'netdev at vger.kernel.org'.

Once part 1 and 2 are accepted:

- Part 3 (RFC patches 16-27): QEDN NVMeTCP Offload
  The qedn patches will be sent to 'linux-nvme at lists.infradead.org'.


From smalin at marvell.com  Wed Jun  2 10:16:48 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 20:16:48 +0300
Subject: [PATCH 0/7] QED NVMeTCP Offload
Message-ID: <20210602171655.23581-1-smalin@marvell.com>

Hi Dave,

Intro:
======
This is the qed part of Marvell???s NVMeTCP offload series, shared as 
RFC series "NVMeTCP Offload ULP and QEDN Device Drive".
This part is a standalone series, and is not dependent on other parts 
of the RFC.
The overall goal is to add qedn as the offload driver for NVMeTCP, 
alongside the existing offload drivers (qedr, qedi and qedf for rdma, 
iscsi and fcoe respectively).

In this series we are making the necessary changes to qed to enable this 
by exposing APIs for FW/HW initializations.

The qedn series (and required changes to NVMe stack) will be sent to the 
linux-nvme mailing list.
I have included more details on the upstream plan under section with the 
same name below.


The Series Patches:
===================
1. qed: Add TCP_ULP FW resource layout ??? replacing iSCSI when common 
   with NVMeTCP.
2. qed: Add NVMeTCP Offload PF Level FW and HW HSI.
3. qed: Add NVMeTCP Offload Connection Level FW and HW HSI.
4. qed: Add support of HW filter block ??? enables redirecting NVMeTCP 
   traffic to the dedicated PF.
5. qed: Add NVMeTCP Offload IO Level FW and HW HSI.
6. qed: Add NVMeTCP Offload IO Level FW Initializations.
7. qed: Add IP services APIs support ???VLAN, IP routing and reserving 
   TCP ports for the offload device.


The NVMeTCP Offload:
====================
With the goal of enabling a generic infrastructure that allows NVMe/TCP
offload devices like NICs to seamlessly plug into the NVMe-oF stack, this
patch series introduces the nvme-tcp-offload ULP host layer, which will
be a new transport type called "tcp-offload" and will serve as an 
abstraction layer to work with vendor specific nvme-tcp offload drivers.

NVMeTCP offload is a full offload of the NVMeTCP protocol, this includes 
both the TCP level and the NVMeTCP level.

The nvme-tcp-offload transport can co-exist with the existing tcp and 
other transports. The tcp offload was designed so that stack changes are 
kept to a bare minimum: only registering new transports.
All other APIs, ops etc. are identical to the regular tcp transport.
Representing the TCP offload as a new transport allows clear and manageable
differentiation between the connections which should use the offload path 
and those that are not offloaded (even on the same device).

The nvme-tcp-offload layers and API compared to nvme-tcp and nvme-rdma:

* NVMe layer: *

       [ nvme/nvme-fabrics/blk-mq ]
             |
        (nvme API and blk-mq API)
             |
             |			 
* Vendor agnostic transport layer: *

      [ nvme-rdma ] [ nvme-tcp ] [ nvme-tcp-offload ]
             |        |             |
           (Verbs) 
             |        |             |
             |     (Socket)
             |        |             |
             |        |        (nvme-tcp-offload API)
             |        |             |
             |        |             |
* Vendor Specific Driver: *

             |        |             |
           [ qedr ]       
                      |             |
                   [ qede ]
                                    |
                                  [ qedn ]


Performance:
============
With this implementation on top of the Marvell qedn driver (using the
Marvell FastLinQ NIC), we were able to demonstrate the following CPU
utilization improvement:

On AMD EPYC 7402, 2.80GHz, 28 cores:
- For 16K queued read IOs, 16jobs, 4qd (50Gbps line rate): 
  Improved the CPU utilization from 15.1% with NVMeTCP SW to 4.7% with 
  NVMeTCP offload.

On Intel(R) Xeon(R) Gold 5122 CPU, 3.60GHz, 16 cores: 
- For 512K queued read IOs, 16jobs, 4qd (25Gbps line rate): 
  Improved the CPU utilization from 16.3% with NVMeTCP SW to 1.1% with 
  NVMeTCP offload.

In addition, we were able to demonstrate the following latency improvement:
- For 200K read IOPS (16 jobs, 16 qd, with fio rate limiter):
  Improved the average latency from 105 usec with NVMeTCP SW to 39 usec 
  with NVMeTCP offload.
  
  Improved the 99.99 tail latency from 570 usec with NVMeTCP SW to 91 usec 
  with NVMeTCP offload.

The end-to-end offload latency was measured from fio while running against 
back end of null device.


The Marvell FastLinQ NIC HW engine:
====================================
The Marvell NIC HW engine is capable of offloading the entire TCP/IP
stack and managing up to 64K connections per PF, already implemented and 
upstream use cases for this include iWARP (by the Marvell qedr driver) 
and iSCSI (by the Marvell qedi driver).
In addition, the Marvell NIC HW engine offloads the NVMeTCP queue layer
and is able to manage the IO level also in case of TCP re-transmissions
and OOO events.
The HW engine enables direct data placement (including the data digest CRC
calculation and validation) and direct data transmission (including data
digest CRC calculation).


The Marvell qedn driver:
========================
The new driver will be added under "drivers/nvme/hw" and will be enabled
by the Kconfig "Marvell NVM Express over Fabrics TCP offload".
As part of the qedn init, the driver will register as a pci device driver 
and will work with the Marvell fastlinQ NIC.
As part of the probe, the driver will register to the nvme_tcp_offload
(ULP) and to the qed module (qed_nvmetcp_ops) - similar to other
"qed_*_ops" which are used by the qede, qedr, qedf and qedi device
drivers.


Upstream Plan:
=============
The RFC series "NVMeTCP Offload ULP and QEDN Device Driver" 
https://lore.kernel.org/netdev/20210531225222.16992-1-smalin at marvell.com/
was designed in a modular way so that part 1 (nvme-tcp-offload) and 
part 2 (qed) are independent and part 3 (qedn) depends on both parts 1+2.

- Part 1 (RFC patch 1-8): NVMeTCP Offload ULP
  The nvme-tcp-offload patches, will be sent to 
  'linux-nvme at lists.infradead.org'.
  
- Part 2 (RFC patches 9-15): QED NVMeTCP Offload
  The qed infrastructure, will be sent to 'netdev at vger.kernel.org'.

Once part 1 and 2 are accepted:

- Part 3 (RFC patches 16-27): QEDN NVMeTCP Offload
  The qedn patches, will be sent to 'linux-nvme at lists.infradead.org'.


Changes since RFC v1,2:
=====================
- No changes in qed (only in nvme-tcp-offload and qedn).

Changes since RFC v3:
=====================
- qed: Add support for the new AHP HW. 

Changes since RFC v4:
=====================
(Many thanks to Hannes Reinecke for his feedback)
- qed: Add TCP_ULP FW resource layout.
- qed: Fix ipv4/ipv6 address initialization.
- qed: Replace structures with nvme-tcp.h structures.

Changes since RFC v5,6:
=====================
- No changes in qed (only in nvme-tcp-offload and qedn).


Nikolay Assa (1):
  qed: Add IP services APIs support

Omkar Kulkarni (1):
  qed: Add TCP_ULP FW resource layout

Prabhakar Kushwaha (1):
  qed: Add support of HW filter block

Shai Malin (4):
  qed: Add NVMeTCP Offload PF Level FW and HW HSI
  qed: Add NVMeTCP Offload Connection Level FW and HW HSI
  qed: Add NVMeTCP Offload IO Level FW and HW HSI
  qed: Add NVMeTCP Offload IO Level FW Initializations

 drivers/net/ethernet/qlogic/Kconfig           |   3 +
 drivers/net/ethernet/qlogic/qed/Makefile      |   5 +
 drivers/net/ethernet/qlogic/qed/qed.h         |  14 +
 drivers/net/ethernet/qlogic/qed/qed_cxt.c     |  45 +-
 drivers/net/ethernet/qlogic/qed/qed_cxt.h     |   2 +-
 drivers/net/ethernet/qlogic/qed/qed_dev.c     | 140 ++-
 drivers/net/ethernet/qlogic/qed/qed_hsi.h     |   6 +-
 drivers/net/ethernet/qlogic/qed/qed_iscsi.c   |  20 +-
 drivers/net/ethernet/qlogic/qed/qed_ll2.c     |  40 +-
 drivers/net/ethernet/qlogic/qed/qed_mcp.c     |   3 +
 drivers/net/ethernet/qlogic/qed/qed_mng_tlv.c |   3 +-
 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c | 829 ++++++++++++++++++
 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h | 103 +++
 .../qlogic/qed/qed_nvmetcp_fw_funcs.c         | 376 ++++++++
 .../qlogic/qed/qed_nvmetcp_fw_funcs.h         |  40 +
 .../qlogic/qed/qed_nvmetcp_ip_services.c      | 238 +++++
 drivers/net/ethernet/qlogic/qed/qed_ooo.c     |   5 +-
 drivers/net/ethernet/qlogic/qed/qed_sp.h      |   5 +
 .../net/ethernet/qlogic/qed/qed_sp_commands.c |   3 +-
 include/linux/qed/common_hsi.h                |   2 +-
 include/linux/qed/nvmetcp_common.h            | 531 +++++++++++
 include/linux/qed/qed_if.h                    |  18 +
 include/linux/qed/qed_ll2_if.h                |   2 +-
 include/linux/qed/qed_nvmetcp_if.h            | 240 +++++
 .../linux/qed/qed_nvmetcp_ip_services_if.h    |  29 +
 25 files changed, 2650 insertions(+), 52 deletions(-)
 create mode 100644 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
 create mode 100644 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h
 create mode 100644 drivers/net/ethernet/qlogic/qed/qed_nvmetcp_fw_funcs.c
 create mode 100644 drivers/net/ethernet/qlogic/qed/qed_nvmetcp_fw_funcs.h
 create mode 100644 drivers/net/ethernet/qlogic/qed/qed_nvmetcp_ip_services.c
 create mode 100644 include/linux/qed/nvmetcp_common.h
 create mode 100644 include/linux/qed/qed_nvmetcp_if.h
 create mode 100644 include/linux/qed/qed_nvmetcp_ip_services_if.h

-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 10:16:50 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 20:16:50 +0300
Subject: [PATCH 2/7] qed: Add NVMeTCP Offload PF Level FW and HW HSI
In-Reply-To: <20210602171655.23581-1-smalin@marvell.com>
References: <20210602171655.23581-1-smalin@marvell.com>
Message-ID: <20210602171655.23581-3-smalin@marvell.com>

This patch introduces the NVMeTCP device and PF level HSI and HSI
functionality in order to initialize and interact with the HW device.
The patch also adds qed NVMeTCP personality.

This patch is based on the qede, qedr, qedi, qedf drivers HSI.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/net/ethernet/qlogic/Kconfig           |   3 +
 drivers/net/ethernet/qlogic/qed/Makefile      |   2 +
 drivers/net/ethernet/qlogic/qed/qed.h         |   6 +
 drivers/net/ethernet/qlogic/qed/qed_cxt.c     |  27 +-
 drivers/net/ethernet/qlogic/qed/qed_dev.c     |  48 +++-
 drivers/net/ethernet/qlogic/qed/qed_hsi.h     |   4 +-
 drivers/net/ethernet/qlogic/qed/qed_ll2.c     |  32 ++-
 drivers/net/ethernet/qlogic/qed/qed_mcp.c     |   3 +
 drivers/net/ethernet/qlogic/qed/qed_mng_tlv.c |   3 +-
 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c | 266 ++++++++++++++++++
 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h |  51 ++++
 drivers/net/ethernet/qlogic/qed/qed_ooo.c     |   3 +-
 drivers/net/ethernet/qlogic/qed/qed_sp.h      |   2 +
 .../net/ethernet/qlogic/qed/qed_sp_commands.c |   1 +
 include/linux/qed/nvmetcp_common.h            |  54 ++++
 include/linux/qed/qed_if.h                    |  18 ++
 include/linux/qed/qed_nvmetcp_if.h            |  71 +++++
 17 files changed, 572 insertions(+), 22 deletions(-)
 create mode 100644 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
 create mode 100644 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h
 create mode 100644 include/linux/qed/nvmetcp_common.h
 create mode 100644 include/linux/qed/qed_nvmetcp_if.h

diff --git a/drivers/net/ethernet/qlogic/Kconfig b/drivers/net/ethernet/qlogic/Kconfig
index 6b5ddb07ee83..98f430905ffa 100644
--- a/drivers/net/ethernet/qlogic/Kconfig
+++ b/drivers/net/ethernet/qlogic/Kconfig
@@ -110,6 +110,9 @@ config QED_RDMA
 config QED_ISCSI
 	bool
 
+config QED_NVMETCP
+	bool
+
 config QED_FCOE
 	bool
 
diff --git a/drivers/net/ethernet/qlogic/qed/Makefile b/drivers/net/ethernet/qlogic/qed/Makefile
index 8251755ec18c..7cb0db67ba5b 100644
--- a/drivers/net/ethernet/qlogic/qed/Makefile
+++ b/drivers/net/ethernet/qlogic/qed/Makefile
@@ -28,6 +28,8 @@ qed-$(CONFIG_QED_ISCSI) += qed_iscsi.o
 qed-$(CONFIG_QED_LL2) += qed_ll2.o
 qed-$(CONFIG_QED_OOO) += qed_ooo.o
 
+qed-$(CONFIG_QED_NVMETCP) += qed_nvmetcp.o
+
 qed-$(CONFIG_QED_RDMA) +=	\
 	qed_iwarp.o		\
 	qed_rdma.o		\
diff --git a/drivers/net/ethernet/qlogic/qed/qed.h b/drivers/net/ethernet/qlogic/qed/qed.h
index a20cb8a0c377..bc9bdb9d1bb9 100644
--- a/drivers/net/ethernet/qlogic/qed/qed.h
+++ b/drivers/net/ethernet/qlogic/qed/qed.h
@@ -200,6 +200,7 @@ enum qed_pci_personality {
 	QED_PCI_ETH,
 	QED_PCI_FCOE,
 	QED_PCI_ISCSI,
+	QED_PCI_NVMETCP,
 	QED_PCI_ETH_ROCE,
 	QED_PCI_ETH_IWARP,
 	QED_PCI_ETH_RDMA,
@@ -239,6 +240,7 @@ enum QED_FEATURE {
 	QED_PF_L2_QUE,
 	QED_VF,
 	QED_RDMA_CNQ,
+	QED_NVMETCP_CQ,
 	QED_ISCSI_CQ,
 	QED_FCOE_CQ,
 	QED_VF_L2_QUE,
@@ -284,6 +286,8 @@ struct qed_hw_info {
 	((dev)->hw_info.personality == QED_PCI_FCOE)
 #define QED_IS_ISCSI_PERSONALITY(dev)					\
 	((dev)->hw_info.personality == QED_PCI_ISCSI)
+#define QED_IS_NVMETCP_PERSONALITY(dev)					\
+	((dev)->hw_info.personality == QED_PCI_NVMETCP)
 
 	/* Resource Allocation scheme results */
 	u32				resc_start[QED_MAX_RESC];
@@ -592,6 +596,7 @@ struct qed_hwfn {
 	struct qed_ooo_info		*p_ooo_info;
 	struct qed_rdma_info		*p_rdma_info;
 	struct qed_iscsi_info		*p_iscsi_info;
+	struct qed_nvmetcp_info		*p_nvmetcp_info;
 	struct qed_fcoe_info		*p_fcoe_info;
 	struct qed_pf_params		pf_params;
 
@@ -828,6 +833,7 @@ struct qed_dev {
 		struct qed_eth_cb_ops		*eth;
 		struct qed_fcoe_cb_ops		*fcoe;
 		struct qed_iscsi_cb_ops		*iscsi;
+		struct qed_nvmetcp_cb_ops	*nvmetcp;
 	} protocol_ops;
 	void				*ops_cookie;
 
diff --git a/drivers/net/ethernet/qlogic/qed/qed_cxt.c b/drivers/net/ethernet/qlogic/qed/qed_cxt.c
index fcabbaa518df..5a0a3cbcc1c1 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_cxt.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_cxt.c
@@ -2072,7 +2072,6 @@ int qed_cxt_set_pf_params(struct qed_hwfn *p_hwfn, u32 rdma_tasks)
 						    PROTOCOLID_FCOE,
 						    p_params->num_cons,
 						    0);
-
 			qed_cxt_set_proto_tid_count(p_hwfn, PROTOCOLID_FCOE,
 						    QED_CXT_FCOE_TID_SEG, 0,
 						    p_params->num_tasks, true);
@@ -2093,7 +2092,6 @@ int qed_cxt_set_pf_params(struct qed_hwfn *p_hwfn, u32 rdma_tasks)
 						    PROTOCOLID_TCP_ULP,
 						    p_params->num_cons,
 						    0);
-
 			qed_cxt_set_proto_tid_count(p_hwfn,
 						    PROTOCOLID_TCP_ULP,
 						    QED_CXT_TCP_ULP_TID_SEG,
@@ -2106,6 +2104,29 @@ int qed_cxt_set_pf_params(struct qed_hwfn *p_hwfn, u32 rdma_tasks)
 		}
 		break;
 	}
+	case QED_PCI_NVMETCP:
+	{
+		struct qed_nvmetcp_pf_params *p_params;
+
+		p_params = &p_hwfn->pf_params.nvmetcp_pf_params;
+
+		if (p_params->num_cons && p_params->num_tasks) {
+			qed_cxt_set_proto_cid_count(p_hwfn,
+						    PROTOCOLID_TCP_ULP,
+						    p_params->num_cons,
+						    0);
+			qed_cxt_set_proto_tid_count(p_hwfn,
+						    PROTOCOLID_TCP_ULP,
+						    QED_CXT_TCP_ULP_TID_SEG,
+						    0,
+						    p_params->num_tasks,
+						    true);
+		} else {
+			DP_INFO(p_hwfn->cdev,
+				"NvmeTCP personality used without setting params!\n");
+		}
+		break;
+	}
 	default:
 		return -EINVAL;
 	}
@@ -2129,6 +2150,7 @@ int qed_cxt_get_tid_mem_info(struct qed_hwfn *p_hwfn,
 		seg = QED_CXT_FCOE_TID_SEG;
 		break;
 	case QED_PCI_ISCSI:
+	case QED_PCI_NVMETCP:
 		proto = PROTOCOLID_TCP_ULP;
 		seg = QED_CXT_TCP_ULP_TID_SEG;
 		break;
@@ -2455,6 +2477,7 @@ int qed_cxt_get_task_ctx(struct qed_hwfn *p_hwfn,
 		seg = QED_CXT_FCOE_TID_SEG;
 		break;
 	case QED_PCI_ISCSI:
+	case QED_PCI_NVMETCP:
 		proto = PROTOCOLID_TCP_ULP;
 		seg = QED_CXT_TCP_ULP_TID_SEG;
 		break;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_dev.c b/drivers/net/ethernet/qlogic/qed/qed_dev.c
index c231d0e56571..932b892f1ef1 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_dev.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_dev.c
@@ -37,6 +37,7 @@
 #include "qed_sriov.h"
 #include "qed_vf.h"
 #include "qed_rdma.h"
+#include "qed_nvmetcp.h"
 
 static DEFINE_SPINLOCK(qm_lock);
 
@@ -667,7 +668,8 @@ qed_llh_set_engine_affin(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
 	}
 
 	/* Storage PF is bound to a single engine while L2 PF uses both */
-	if (QED_IS_FCOE_PERSONALITY(p_hwfn) || QED_IS_ISCSI_PERSONALITY(p_hwfn))
+	if (QED_IS_FCOE_PERSONALITY(p_hwfn) || QED_IS_ISCSI_PERSONALITY(p_hwfn) ||
+	    QED_IS_NVMETCP_PERSONALITY(p_hwfn))
 		eng = cdev->fir_affin ? QED_ENG1 : QED_ENG0;
 	else			/* L2_PERSONALITY */
 		eng = QED_BOTH_ENG;
@@ -1164,6 +1166,9 @@ void qed_llh_remove_mac_filter(struct qed_dev *cdev,
 	if (!test_bit(QED_MF_LLH_MAC_CLSS, &cdev->mf_bits))
 		goto out;
 
+	if (QED_IS_NVMETCP_PERSONALITY(p_hwfn))
+		return;
+
 	ether_addr_copy(filter.mac.addr, mac_addr);
 	rc = qed_llh_shadow_remove_filter(cdev, ppfid, &filter, &filter_idx,
 					  &ref_cnt);
@@ -1381,6 +1386,11 @@ void qed_resc_free(struct qed_dev *cdev)
 			qed_ooo_free(p_hwfn);
 		}
 
+		if (p_hwfn->hw_info.personality == QED_PCI_NVMETCP) {
+			qed_nvmetcp_free(p_hwfn);
+			qed_ooo_free(p_hwfn);
+		}
+
 		if (QED_IS_RDMA_PERSONALITY(p_hwfn) && rdma_info) {
 			qed_spq_unregister_async_cb(p_hwfn, rdma_info->proto);
 			qed_rdma_info_free(p_hwfn);
@@ -1423,6 +1433,7 @@ static u32 qed_get_pq_flags(struct qed_hwfn *p_hwfn)
 		flags |= PQ_FLAGS_OFLD;
 		break;
 	case QED_PCI_ISCSI:
+	case QED_PCI_NVMETCP:
 		flags |= PQ_FLAGS_ACK | PQ_FLAGS_OOO | PQ_FLAGS_OFLD;
 		break;
 	case QED_PCI_ETH_ROCE:
@@ -2263,7 +2274,8 @@ int qed_resc_alloc(struct qed_dev *cdev)
 			 * at the same time
 			 */
 			n_eqes += num_cons + 2 * MAX_NUM_VFS_BB + n_srq;
-		} else if (p_hwfn->hw_info.personality == QED_PCI_ISCSI) {
+		} else if (p_hwfn->hw_info.personality == QED_PCI_ISCSI ||
+			   p_hwfn->hw_info.personality == QED_PCI_NVMETCP) {
 			num_cons =
 			    qed_cxt_get_proto_cid_count(p_hwfn,
 							PROTOCOLID_TCP_ULP,
@@ -2313,6 +2325,15 @@ int qed_resc_alloc(struct qed_dev *cdev)
 				goto alloc_err;
 		}
 
+		if (p_hwfn->hw_info.personality == QED_PCI_NVMETCP) {
+			rc = qed_nvmetcp_alloc(p_hwfn);
+			if (rc)
+				goto alloc_err;
+			rc = qed_ooo_alloc(p_hwfn);
+			if (rc)
+				goto alloc_err;
+		}
+
 		if (QED_IS_RDMA_PERSONALITY(p_hwfn)) {
 			rc = qed_rdma_info_alloc(p_hwfn);
 			if (rc)
@@ -2393,6 +2414,11 @@ void qed_resc_setup(struct qed_dev *cdev)
 			qed_iscsi_setup(p_hwfn);
 			qed_ooo_setup(p_hwfn);
 		}
+
+		if (p_hwfn->hw_info.personality == QED_PCI_NVMETCP) {
+			qed_nvmetcp_setup(p_hwfn);
+			qed_ooo_setup(p_hwfn);
+		}
 	}
 }
 
@@ -2854,7 +2880,8 @@ static int qed_hw_init_pf(struct qed_hwfn *p_hwfn,
 
 	/* Protocol Configuration */
 	STORE_RT_REG(p_hwfn, PRS_REG_SEARCH_TCP_RT_OFFSET,
-		     (p_hwfn->hw_info.personality == QED_PCI_ISCSI) ? 1 : 0);
+		     ((p_hwfn->hw_info.personality == QED_PCI_ISCSI) ||
+			 (p_hwfn->hw_info.personality == QED_PCI_NVMETCP)) ? 1 : 0);
 	STORE_RT_REG(p_hwfn, PRS_REG_SEARCH_FCOE_RT_OFFSET,
 		     (p_hwfn->hw_info.personality == QED_PCI_FCOE) ? 1 : 0);
 	STORE_RT_REG(p_hwfn, PRS_REG_SEARCH_ROCE_RT_OFFSET, 0);
@@ -3535,14 +3562,21 @@ static void qed_hw_set_feat(struct qed_hwfn *p_hwfn)
 		feat_num[QED_ISCSI_CQ] = min_t(u32, sb_cnt.cnt,
 					       RESC_NUM(p_hwfn,
 							QED_CMDQS_CQS));
+
+	if (QED_IS_NVMETCP_PERSONALITY(p_hwfn))
+		feat_num[QED_NVMETCP_CQ] = min_t(u32, sb_cnt.cnt,
+						 RESC_NUM(p_hwfn,
+							  QED_CMDQS_CQS));
+
 	DP_VERBOSE(p_hwfn,
 		   NETIF_MSG_PROBE,
-		   "#PF_L2_QUEUES=%d VF_L2_QUEUES=%d #ROCE_CNQ=%d FCOE_CQ=%d ISCSI_CQ=%d #SBS=%d\n",
+		   "#PF_L2_QUEUES=%d VF_L2_QUEUES=%d #ROCE_CNQ=%d FCOE_CQ=%d ISCSI_CQ=%d NVMETCP_CQ=%d #SBS=%d\n",
 		   (int)FEAT_NUM(p_hwfn, QED_PF_L2_QUE),
 		   (int)FEAT_NUM(p_hwfn, QED_VF_L2_QUE),
 		   (int)FEAT_NUM(p_hwfn, QED_RDMA_CNQ),
 		   (int)FEAT_NUM(p_hwfn, QED_FCOE_CQ),
 		   (int)FEAT_NUM(p_hwfn, QED_ISCSI_CQ),
+		   (int)FEAT_NUM(p_hwfn, QED_NVMETCP_CQ),
 		   (int)sb_cnt.cnt);
 }
 
@@ -3734,7 +3768,8 @@ int qed_hw_get_dflt_resc(struct qed_hwfn *p_hwfn,
 		break;
 	case QED_BDQ:
 		if (p_hwfn->hw_info.personality != QED_PCI_ISCSI &&
-		    p_hwfn->hw_info.personality != QED_PCI_FCOE)
+		    p_hwfn->hw_info.personality != QED_PCI_FCOE &&
+			p_hwfn->hw_info.personality != QED_PCI_NVMETCP)
 			*p_resc_num = 0;
 		else
 			*p_resc_num = 1;
@@ -3755,7 +3790,8 @@ int qed_hw_get_dflt_resc(struct qed_hwfn *p_hwfn,
 			*p_resc_start = 0;
 		else if (p_hwfn->cdev->num_ports_in_engine == 4)
 			*p_resc_start = p_hwfn->port_id;
-		else if (p_hwfn->hw_info.personality == QED_PCI_ISCSI)
+		else if (p_hwfn->hw_info.personality == QED_PCI_ISCSI ||
+			 p_hwfn->hw_info.personality == QED_PCI_NVMETCP)
 			*p_resc_start = p_hwfn->port_id;
 		else if (p_hwfn->hw_info.personality == QED_PCI_FCOE)
 			*p_resc_start = p_hwfn->port_id + 2;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_hsi.h b/drivers/net/ethernet/qlogic/qed/qed_hsi.h
index 9dbeb2efdc51..fb1baa2da2d0 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_hsi.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_hsi.h
@@ -20,6 +20,7 @@
 #include <linux/qed/fcoe_common.h>
 #include <linux/qed/eth_common.h>
 #include <linux/qed/iscsi_common.h>
+#include <linux/qed/nvmetcp_common.h>
 #include <linux/qed/iwarp_common.h>
 #include <linux/qed/rdma_common.h>
 #include <linux/qed/roce_common.h>
@@ -12147,7 +12148,8 @@ struct public_func {
 #define FUNC_MF_CFG_PROTOCOL_ISCSI              0x00000010
 #define FUNC_MF_CFG_PROTOCOL_FCOE               0x00000020
 #define FUNC_MF_CFG_PROTOCOL_ROCE               0x00000030
-#define FUNC_MF_CFG_PROTOCOL_MAX	0x00000030
+#define FUNC_MF_CFG_PROTOCOL_NVMETCP    0x00000040
+#define FUNC_MF_CFG_PROTOCOL_MAX	0x00000040
 
 #define FUNC_MF_CFG_MIN_BW_MASK		0x0000ff00
 #define FUNC_MF_CFG_MIN_BW_SHIFT	8
diff --git a/drivers/net/ethernet/qlogic/qed/qed_ll2.c b/drivers/net/ethernet/qlogic/qed/qed_ll2.c
index 286e53927866..02a4610d9330 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_ll2.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_ll2.c
@@ -960,7 +960,8 @@ static int qed_sp_ll2_rx_queue_start(struct qed_hwfn *p_hwfn,
 
 	if (test_bit(QED_MF_LL2_NON_UNICAST, &p_hwfn->cdev->mf_bits) &&
 	    p_ramrod->main_func_queue && conn_type != QED_LL2_TYPE_ROCE &&
-	    conn_type != QED_LL2_TYPE_IWARP) {
+	    conn_type != QED_LL2_TYPE_IWARP &&
+		(!QED_IS_NVMETCP_PERSONALITY(p_hwfn))) {
 		p_ramrod->mf_si_bcast_accept_all = 1;
 		p_ramrod->mf_si_mcast_accept_all = 1;
 	} else {
@@ -1047,7 +1048,8 @@ static int qed_sp_ll2_tx_queue_start(struct qed_hwfn *p_hwfn,
 		p_ramrod->conn_type = PROTOCOLID_IWARP;
 		break;
 	case QED_LL2_TYPE_OOO:
-		if (p_hwfn->hw_info.personality == QED_PCI_ISCSI)
+		if (p_hwfn->hw_info.personality == QED_PCI_ISCSI ||
+		    p_hwfn->hw_info.personality == QED_PCI_NVMETCP)
 			p_ramrod->conn_type = PROTOCOLID_TCP_ULP;
 		else
 			p_ramrod->conn_type = PROTOCOLID_IWARP;
@@ -1634,7 +1636,8 @@ int qed_ll2_establish_connection(void *cxt, u8 connection_handle)
 	if (rc)
 		goto out;
 
-	if (!QED_IS_RDMA_PERSONALITY(p_hwfn))
+	if (!QED_IS_RDMA_PERSONALITY(p_hwfn) &&
+	    !QED_IS_NVMETCP_PERSONALITY(p_hwfn))
 		qed_wr(p_hwfn, p_ptt, PRS_REG_USE_LIGHT_L2, 1);
 
 	qed_ll2_establish_connection_ooo(p_hwfn, p_ll2_conn);
@@ -2376,7 +2379,8 @@ static int qed_ll2_start_ooo(struct qed_hwfn *p_hwfn,
 static bool qed_ll2_is_storage_eng1(struct qed_dev *cdev)
 {
 	return (QED_IS_FCOE_PERSONALITY(QED_LEADING_HWFN(cdev)) ||
-		QED_IS_ISCSI_PERSONALITY(QED_LEADING_HWFN(cdev))) &&
+		QED_IS_ISCSI_PERSONALITY(QED_LEADING_HWFN(cdev)) ||
+		QED_IS_NVMETCP_PERSONALITY(QED_LEADING_HWFN(cdev))) &&
 		(QED_AFFIN_HWFN(cdev) != QED_LEADING_HWFN(cdev));
 }
 
@@ -2402,11 +2406,13 @@ static int qed_ll2_stop(struct qed_dev *cdev)
 
 	if (cdev->ll2->handle == QED_LL2_UNUSED_HANDLE)
 		return 0;
+	if (!QED_IS_NVMETCP_PERSONALITY(p_hwfn))
+		qed_llh_remove_mac_filter(cdev, 0, cdev->ll2_mac_address);
 
 	qed_llh_remove_mac_filter(cdev, 0, cdev->ll2_mac_address);
 	eth_zero_addr(cdev->ll2_mac_address);
 
-	if (QED_IS_ISCSI_PERSONALITY(p_hwfn))
+	if (QED_IS_ISCSI_PERSONALITY(p_hwfn) || QED_IS_NVMETCP_PERSONALITY(p_hwfn))
 		qed_ll2_stop_ooo(p_hwfn);
 
 	/* In CMT mode, LL2 is always started on engine 0 for a storage PF */
@@ -2442,6 +2448,7 @@ static int __qed_ll2_start(struct qed_hwfn *p_hwfn,
 		conn_type = QED_LL2_TYPE_FCOE;
 		break;
 	case QED_PCI_ISCSI:
+	case QED_PCI_NVMETCP:
 		conn_type = QED_LL2_TYPE_TCP_ULP;
 		break;
 	case QED_PCI_ETH_ROCE:
@@ -2567,7 +2574,7 @@ static int qed_ll2_start(struct qed_dev *cdev, struct qed_ll2_params *params)
 		}
 	}
 
-	if (QED_IS_ISCSI_PERSONALITY(p_hwfn)) {
+	if (QED_IS_ISCSI_PERSONALITY(p_hwfn) || QED_IS_NVMETCP_PERSONALITY(p_hwfn)) {
 		DP_VERBOSE(cdev, QED_MSG_STORAGE, "Starting OOO LL2 queue\n");
 		rc = qed_ll2_start_ooo(p_hwfn, params);
 		if (rc) {
@@ -2576,10 +2583,13 @@ static int qed_ll2_start(struct qed_dev *cdev, struct qed_ll2_params *params)
 		}
 	}
 
-	rc = qed_llh_add_mac_filter(cdev, 0, params->ll2_mac_address);
-	if (rc) {
-		DP_NOTICE(cdev, "Failed to add an LLH filter\n");
-		goto err3;
+	if (!QED_IS_NVMETCP_PERSONALITY(p_hwfn)) {
+		rc = qed_llh_add_mac_filter(cdev, 0, params->ll2_mac_address);
+		if (rc) {
+			DP_NOTICE(cdev, "Failed to add an LLH filter\n");
+			goto err3;
+		}
+
 	}
 
 	ether_addr_copy(cdev->ll2_mac_address, params->ll2_mac_address);
@@ -2587,7 +2597,7 @@ static int qed_ll2_start(struct qed_dev *cdev, struct qed_ll2_params *params)
 	return 0;
 
 err3:
-	if (QED_IS_ISCSI_PERSONALITY(p_hwfn))
+	if (QED_IS_ISCSI_PERSONALITY(p_hwfn) || QED_IS_NVMETCP_PERSONALITY(p_hwfn))
 		qed_ll2_stop_ooo(p_hwfn);
 err2:
 	if (b_is_storage_eng1)
diff --git a/drivers/net/ethernet/qlogic/qed/qed_mcp.c b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
index cd882c453394..4387292c37e2 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_mcp.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
@@ -2446,6 +2446,9 @@ qed_mcp_get_shmem_proto(struct qed_hwfn *p_hwfn,
 	case FUNC_MF_CFG_PROTOCOL_ISCSI:
 		*p_proto = QED_PCI_ISCSI;
 		break;
+	case FUNC_MF_CFG_PROTOCOL_NVMETCP:
+		*p_proto = QED_PCI_NVMETCP;
+		break;
 	case FUNC_MF_CFG_PROTOCOL_FCOE:
 		*p_proto = QED_PCI_FCOE;
 		break;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_mng_tlv.c b/drivers/net/ethernet/qlogic/qed/qed_mng_tlv.c
index 3e3192a3ad9b..6190adf965bc 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_mng_tlv.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_mng_tlv.c
@@ -1306,7 +1306,8 @@ int qed_mfw_process_tlv_req(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
 	}
 
 	if ((tlv_group & QED_MFW_TLV_ISCSI) &&
-	    p_hwfn->hw_info.personality != QED_PCI_ISCSI) {
+	    p_hwfn->hw_info.personality != QED_PCI_ISCSI &&
+		p_hwfn->hw_info.personality != QED_PCI_NVMETCP) {
 		DP_VERBOSE(p_hwfn, QED_MSG_SP,
 			   "Skipping iSCSI TLVs for non-iSCSI function\n");
 		tlv_group &= ~QED_MFW_TLV_ISCSI;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
new file mode 100644
index 000000000000..cb9c71109b2d
--- /dev/null
+++ b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
@@ -0,0 +1,266 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause)
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#include <linux/types.h>
+#include <asm/byteorder.h>
+#include <asm/param.h>
+#include <linux/delay.h>
+#include <linux/dma-mapping.h>
+#include <linux/etherdevice.h>
+#include <linux/kernel.h>
+#include <linux/log2.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/stddef.h>
+#include <linux/string.h>
+#include <linux/errno.h>
+#include <linux/list.h>
+#include <linux/qed/qed_nvmetcp_if.h>
+#include "qed.h"
+#include "qed_cxt.h"
+#include "qed_dev_api.h"
+#include "qed_hsi.h"
+#include "qed_hw.h"
+#include "qed_int.h"
+#include "qed_nvmetcp.h"
+#include "qed_ll2.h"
+#include "qed_mcp.h"
+#include "qed_sp.h"
+#include "qed_reg_addr.h"
+
+static int qed_nvmetcp_async_event(struct qed_hwfn *p_hwfn, u8 fw_event_code,
+				   u16 echo, union event_ring_data *data,
+				   u8 fw_return_code)
+{
+	if (p_hwfn->p_nvmetcp_info->event_cb) {
+		struct qed_nvmetcp_info *p_nvmetcp = p_hwfn->p_nvmetcp_info;
+
+		return p_nvmetcp->event_cb(p_nvmetcp->event_context,
+					 fw_event_code, data);
+	} else {
+		DP_NOTICE(p_hwfn, "nvmetcp async completion is not set\n");
+
+		return -EINVAL;
+	}
+}
+
+static int qed_sp_nvmetcp_func_start(struct qed_hwfn *p_hwfn,
+				     enum spq_mode comp_mode,
+				     struct qed_spq_comp_cb *p_comp_addr,
+				     void *event_context,
+				     nvmetcp_event_cb_t async_event_cb)
+{
+	struct nvmetcp_init_ramrod_params *p_ramrod = NULL;
+	struct qed_nvmetcp_pf_params *p_params = NULL;
+	struct scsi_init_func_queues *p_queue = NULL;
+	struct nvmetcp_spe_func_init *p_init = NULL;
+	struct qed_sp_init_data init_data = {};
+	struct qed_spq_entry *p_ent = NULL;
+	int rc = 0;
+	u16 val;
+	u8 i;
+
+	/* Get SPQ entry */
+	init_data.cid = qed_spq_get_cid(p_hwfn);
+	init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+	init_data.comp_mode = comp_mode;
+	init_data.p_comp_data = p_comp_addr;
+	rc = qed_sp_init_request(p_hwfn, &p_ent,
+				 NVMETCP_RAMROD_CMD_ID_INIT_FUNC,
+				 PROTOCOLID_TCP_ULP, &init_data);
+	if (rc)
+		return rc;
+
+	p_ramrod = &p_ent->ramrod.nvmetcp_init;
+	p_init = &p_ramrod->nvmetcp_init_spe;
+	p_params = &p_hwfn->pf_params.nvmetcp_pf_params;
+	p_queue = &p_init->q_params;
+	p_init->num_sq_pages_in_ring = p_params->num_sq_pages_in_ring;
+	p_init->num_r2tq_pages_in_ring = p_params->num_r2tq_pages_in_ring;
+	p_init->num_uhq_pages_in_ring = p_params->num_uhq_pages_in_ring;
+	p_init->ll2_rx_queue_id = RESC_START(p_hwfn, QED_LL2_RAM_QUEUE) +
+					p_params->ll2_ooo_queue_id;
+	SET_FIELD(p_init->flags, NVMETCP_SPE_FUNC_INIT_NVMETCP_MODE, 1);
+	p_init->func_params.log_page_size = ilog2(PAGE_SIZE);
+	p_init->func_params.num_tasks = cpu_to_le16(p_params->num_tasks);
+	p_init->debug_flags = p_params->debug_mode;
+	DMA_REGPAIR_LE(p_queue->glbl_q_params_addr,
+		       p_params->glbl_q_params_addr);
+	p_queue->cq_num_entries = cpu_to_le16(QED_NVMETCP_FW_CQ_SIZE);
+	p_queue->num_queues = p_params->num_queues;
+	val = RESC_START(p_hwfn, QED_CMDQS_CQS);
+	p_queue->queue_relative_offset = cpu_to_le16((u16)val);
+	p_queue->cq_sb_pi = p_params->gl_rq_pi;
+
+	for (i = 0; i < p_params->num_queues; i++) {
+		val = qed_get_igu_sb_id(p_hwfn, i);
+		p_queue->cq_cmdq_sb_num_arr[i] = cpu_to_le16(val);
+	}
+
+	SET_FIELD(p_queue->q_validity,
+		  SCSI_INIT_FUNC_QUEUES_CMD_VALID, 0);
+	p_queue->cmdq_num_entries = 0;
+	p_queue->bdq_resource_id = (u8)RESC_START(p_hwfn, QED_BDQ);
+	p_ramrod->tcp_init.two_msl_timer = cpu_to_le32(QED_TCP_TWO_MSL_TIMER);
+	p_ramrod->tcp_init.tx_sws_timer = cpu_to_le16(QED_TCP_SWS_TIMER);
+	p_init->half_way_close_timeout = cpu_to_le16(QED_TCP_HALF_WAY_CLOSE_TIMEOUT);
+	p_ramrod->tcp_init.max_fin_rt = QED_TCP_MAX_FIN_RT;
+	SET_FIELD(p_ramrod->nvmetcp_init_spe.params,
+		  NVMETCP_SPE_FUNC_INIT_MAX_SYN_RT, QED_TCP_MAX_FIN_RT);
+	p_hwfn->p_nvmetcp_info->event_context = event_context;
+	p_hwfn->p_nvmetcp_info->event_cb = async_event_cb;
+	qed_spq_register_async_cb(p_hwfn, PROTOCOLID_TCP_ULP,
+				  qed_nvmetcp_async_event);
+
+	return qed_spq_post(p_hwfn, p_ent, NULL);
+}
+
+static int qed_sp_nvmetcp_func_stop(struct qed_hwfn *p_hwfn,
+				    enum spq_mode comp_mode,
+				    struct qed_spq_comp_cb *p_comp_addr)
+{
+	struct qed_spq_entry *p_ent = NULL;
+	struct qed_sp_init_data init_data;
+	int rc;
+
+	/* Get SPQ entry */
+	memset(&init_data, 0, sizeof(init_data));
+	init_data.cid = qed_spq_get_cid(p_hwfn);
+	init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+	init_data.comp_mode = comp_mode;
+	init_data.p_comp_data = p_comp_addr;
+	rc = qed_sp_init_request(p_hwfn, &p_ent,
+				 NVMETCP_RAMROD_CMD_ID_DESTROY_FUNC,
+				 PROTOCOLID_TCP_ULP, &init_data);
+	if (rc)
+		return rc;
+
+	rc = qed_spq_post(p_hwfn, p_ent, NULL);
+	qed_spq_unregister_async_cb(p_hwfn, PROTOCOLID_TCP_ULP);
+
+	return rc;
+}
+
+static int qed_fill_nvmetcp_dev_info(struct qed_dev *cdev,
+				     struct qed_dev_nvmetcp_info *info)
+{
+	struct qed_hwfn *hwfn = QED_AFFIN_HWFN(cdev);
+	int rc;
+
+	memset(info, 0, sizeof(*info));
+	rc = qed_fill_dev_info(cdev, &info->common);
+	info->port_id = MFW_PORT(hwfn);
+	info->num_cqs = FEAT_NUM(hwfn, QED_NVMETCP_CQ);
+
+	return rc;
+}
+
+static void qed_register_nvmetcp_ops(struct qed_dev *cdev,
+				     struct qed_nvmetcp_cb_ops *ops,
+				     void *cookie)
+{
+	cdev->protocol_ops.nvmetcp = ops;
+	cdev->ops_cookie = cookie;
+}
+
+static int qed_nvmetcp_stop(struct qed_dev *cdev)
+{
+	int rc;
+
+	if (!(cdev->flags & QED_FLAG_STORAGE_STARTED)) {
+		DP_NOTICE(cdev, "nvmetcp already stopped\n");
+
+		return 0;
+	}
+
+	if (!hash_empty(cdev->connections)) {
+		DP_NOTICE(cdev,
+			  "Can't stop nvmetcp - not all connections were returned\n");
+
+		return -EINVAL;
+	}
+
+	/* Stop the nvmetcp */
+	rc = qed_sp_nvmetcp_func_stop(QED_AFFIN_HWFN(cdev), QED_SPQ_MODE_EBLOCK,
+				      NULL);
+	cdev->flags &= ~QED_FLAG_STORAGE_STARTED;
+
+	return rc;
+}
+
+static int qed_nvmetcp_start(struct qed_dev *cdev,
+			     struct qed_nvmetcp_tid *tasks,
+			     void *event_context,
+			     nvmetcp_event_cb_t async_event_cb)
+{
+	struct qed_tid_mem *tid_info;
+	int rc;
+
+	if (cdev->flags & QED_FLAG_STORAGE_STARTED) {
+		DP_NOTICE(cdev, "nvmetcp already started;\n");
+
+		return 0;
+	}
+
+	rc = qed_sp_nvmetcp_func_start(QED_AFFIN_HWFN(cdev),
+				       QED_SPQ_MODE_EBLOCK, NULL,
+				       event_context, async_event_cb);
+	if (rc) {
+		DP_NOTICE(cdev, "Failed to start nvmetcp\n");
+
+		return rc;
+	}
+
+	cdev->flags |= QED_FLAG_STORAGE_STARTED;
+	hash_init(cdev->connections);
+
+	if (!tasks)
+		return 0;
+
+	tid_info = kzalloc(sizeof(*tid_info), GFP_KERNEL);
+	if (!tid_info) {
+		qed_nvmetcp_stop(cdev);
+
+		return -ENOMEM;
+	}
+
+	rc = qed_cxt_get_tid_mem_info(QED_AFFIN_HWFN(cdev), tid_info);
+	if (rc) {
+		DP_NOTICE(cdev, "Failed to gather task information\n");
+		qed_nvmetcp_stop(cdev);
+		kfree(tid_info);
+
+		return rc;
+	}
+
+	/* Fill task information */
+	tasks->size = tid_info->tid_size;
+	tasks->num_tids_per_block = tid_info->num_tids_per_block;
+	memcpy(tasks->blocks, tid_info->blocks,
+	       MAX_TID_BLOCKS_NVMETCP * sizeof(u8 *));
+	kfree(tid_info);
+
+	return 0;
+}
+
+static const struct qed_nvmetcp_ops qed_nvmetcp_ops_pass = {
+	.common = &qed_common_ops_pass,
+	.ll2 = &qed_ll2_ops_pass,
+	.fill_dev_info = &qed_fill_nvmetcp_dev_info,
+	.register_ops = &qed_register_nvmetcp_ops,
+	.start = &qed_nvmetcp_start,
+	.stop = &qed_nvmetcp_stop,
+
+	/* Placeholder - Connection level ops */
+};
+
+const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void)
+{
+	return &qed_nvmetcp_ops_pass;
+}
+EXPORT_SYMBOL(qed_get_nvmetcp_ops);
+
+void qed_put_nvmetcp_ops(void)
+{
+}
+EXPORT_SYMBOL(qed_put_nvmetcp_ops);
diff --git a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h
new file mode 100644
index 000000000000..774b46ade408
--- /dev/null
+++ b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h
@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#ifndef _QED_NVMETCP_H
+#define _QED_NVMETCP_H
+
+#include <linux/types.h>
+#include <linux/list.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/qed/tcp_common.h>
+#include <linux/qed/qed_nvmetcp_if.h>
+#include <linux/qed/qed_chain.h>
+#include "qed.h"
+#include "qed_hsi.h"
+#include "qed_mcp.h"
+#include "qed_sp.h"
+
+#define QED_NVMETCP_FW_CQ_SIZE (4 * 1024)
+
+/* tcp parameters */
+#define QED_TCP_TWO_MSL_TIMER 4000
+#define QED_TCP_HALF_WAY_CLOSE_TIMEOUT 10
+#define QED_TCP_MAX_FIN_RT 2
+#define QED_TCP_SWS_TIMER 5000
+
+struct qed_nvmetcp_info {
+	spinlock_t lock; /* Connection resources. */
+	struct list_head free_list;
+	u16 max_num_outstanding_tasks;
+	void *event_context;
+	nvmetcp_event_cb_t event_cb;
+};
+
+#if IS_ENABLED(CONFIG_QED_NVMETCP)
+int qed_nvmetcp_alloc(struct qed_hwfn *p_hwfn);
+void qed_nvmetcp_setup(struct qed_hwfn *p_hwfn);
+void qed_nvmetcp_free(struct qed_hwfn *p_hwfn);
+
+#else /* IS_ENABLED(CONFIG_QED_NVMETCP) */
+static inline int qed_nvmetcp_alloc(struct qed_hwfn *p_hwfn)
+{
+	return -EINVAL;
+}
+
+static inline void qed_nvmetcp_setup(struct qed_hwfn *p_hwfn) {}
+static inline void qed_nvmetcp_free(struct qed_hwfn *p_hwfn) {}
+
+#endif /* IS_ENABLED(CONFIG_QED_NVMETCP) */
+
+#endif
diff --git a/drivers/net/ethernet/qlogic/qed/qed_ooo.c b/drivers/net/ethernet/qlogic/qed/qed_ooo.c
index 599da0d7366b..b8c5641b29a8 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_ooo.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_ooo.c
@@ -16,7 +16,7 @@
 #include "qed_ll2.h"
 #include "qed_ooo.h"
 #include "qed_cxt.h"
-
+#include "qed_nvmetcp.h"
 static struct qed_ooo_archipelago
 *qed_ooo_seek_archipelago(struct qed_hwfn *p_hwfn,
 			  struct qed_ooo_info
@@ -83,6 +83,7 @@ int qed_ooo_alloc(struct qed_hwfn *p_hwfn)
 
 	switch (p_hwfn->hw_info.personality) {
 	case QED_PCI_ISCSI:
+	case QED_PCI_NVMETCP:
 		proto = PROTOCOLID_TCP_ULP;
 		break;
 	case QED_PCI_ETH_RDMA:
diff --git a/drivers/net/ethernet/qlogic/qed/qed_sp.h b/drivers/net/ethernet/qlogic/qed/qed_sp.h
index 993f1357b6fc..525159e747a5 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_sp.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_sp.h
@@ -100,6 +100,8 @@ union ramrod_data {
 	struct iscsi_spe_conn_mac_update iscsi_conn_mac_update;
 	struct iscsi_spe_conn_termination iscsi_conn_terminate;
 
+	struct nvmetcp_init_ramrod_params nvmetcp_init;
+
 	struct vf_start_ramrod_data vf_start;
 	struct vf_stop_ramrod_data vf_stop;
 };
diff --git a/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c b/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
index ee7dc0a7da6c..b4ed54ffef9b 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
@@ -385,6 +385,7 @@ int qed_sp_pf_start(struct qed_hwfn *p_hwfn,
 		p_ramrod->personality = PERSONALITY_FCOE;
 		break;
 	case QED_PCI_ISCSI:
+	case QED_PCI_NVMETCP:
 		p_ramrod->personality = PERSONALITY_TCP_ULP;
 		break;
 	case QED_PCI_ETH_ROCE:
diff --git a/include/linux/qed/nvmetcp_common.h b/include/linux/qed/nvmetcp_common.h
new file mode 100644
index 000000000000..e9ccfc07041d
--- /dev/null
+++ b/include/linux/qed/nvmetcp_common.h
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#ifndef __NVMETCP_COMMON__
+#define __NVMETCP_COMMON__
+
+#include "tcp_common.h"
+
+/* NVMeTCP firmware function init parameters */
+struct nvmetcp_spe_func_init {
+	__le16 half_way_close_timeout;
+	u8 num_sq_pages_in_ring;
+	u8 num_r2tq_pages_in_ring;
+	u8 num_uhq_pages_in_ring;
+	u8 ll2_rx_queue_id;
+	u8 flags;
+#define NVMETCP_SPE_FUNC_INIT_COUNTERS_EN_MASK 0x1
+#define NVMETCP_SPE_FUNC_INIT_COUNTERS_EN_SHIFT 0
+#define NVMETCP_SPE_FUNC_INIT_NVMETCP_MODE_MASK 0x1
+#define NVMETCP_SPE_FUNC_INIT_NVMETCP_MODE_SHIFT 1
+#define NVMETCP_SPE_FUNC_INIT_RESERVED0_MASK 0x3F
+#define NVMETCP_SPE_FUNC_INIT_RESERVED0_SHIFT 2
+	u8 debug_flags;
+	__le16 reserved1;
+	u8 params;
+#define NVMETCP_SPE_FUNC_INIT_MAX_SYN_RT_MASK	0xF
+#define NVMETCP_SPE_FUNC_INIT_MAX_SYN_RT_SHIFT	0
+#define NVMETCP_SPE_FUNC_INIT_RESERVED1_MASK	0xF
+#define NVMETCP_SPE_FUNC_INIT_RESERVED1_SHIFT	4
+	u8 reserved2[5];
+	struct scsi_init_func_params func_params;
+	struct scsi_init_func_queues q_params;
+};
+
+/* NVMeTCP init params passed by driver to FW in NVMeTCP init ramrod. */
+struct nvmetcp_init_ramrod_params {
+	struct nvmetcp_spe_func_init nvmetcp_init_spe;
+	struct tcp_init_params tcp_init;
+};
+
+/* NVMeTCP Ramrod Command IDs */
+enum nvmetcp_ramrod_cmd_id {
+	NVMETCP_RAMROD_CMD_ID_UNUSED = 0,
+	NVMETCP_RAMROD_CMD_ID_INIT_FUNC = 1,
+	NVMETCP_RAMROD_CMD_ID_DESTROY_FUNC = 2,
+	MAX_NVMETCP_RAMROD_CMD_ID
+};
+
+struct nvmetcp_glbl_queue_entry {
+	struct regpair cq_pbl_addr;
+	struct regpair reserved;
+};
+
+#endif /* __NVMETCP_COMMON__ */
diff --git a/include/linux/qed/qed_if.h b/include/linux/qed/qed_if.h
index 68d17a4fbf20..850b98991670 100644
--- a/include/linux/qed/qed_if.h
+++ b/include/linux/qed/qed_if.h
@@ -542,6 +542,22 @@ struct qed_iscsi_pf_params {
 	u8 bdq_pbl_num_entries[3];
 };
 
+struct qed_nvmetcp_pf_params {
+	u64 glbl_q_params_addr;
+	u16 cq_num_entries;
+	u16 num_cons;
+	u16 num_tasks;
+	u8 num_sq_pages_in_ring;
+	u8 num_r2tq_pages_in_ring;
+	u8 num_uhq_pages_in_ring;
+	u8 num_queues;
+	u8 gl_rq_pi;
+	u8 gl_cmd_pi;
+	u8 debug_mode;
+	u8 ll2_ooo_queue_id;
+	u16 min_rto;
+};
+
 struct qed_rdma_pf_params {
 	/* Supplied to QED during resource allocation (may affect the ILT and
 	 * the doorbell BAR).
@@ -560,6 +576,7 @@ struct qed_pf_params {
 	struct qed_eth_pf_params eth_pf_params;
 	struct qed_fcoe_pf_params fcoe_pf_params;
 	struct qed_iscsi_pf_params iscsi_pf_params;
+	struct qed_nvmetcp_pf_params nvmetcp_pf_params;
 	struct qed_rdma_pf_params rdma_pf_params;
 };
 
@@ -662,6 +679,7 @@ enum qed_sb_type {
 enum qed_protocol {
 	QED_PROTOCOL_ETH,
 	QED_PROTOCOL_ISCSI,
+	QED_PROTOCOL_NVMETCP = QED_PROTOCOL_ISCSI,
 	QED_PROTOCOL_FCOE,
 };
 
diff --git a/include/linux/qed/qed_nvmetcp_if.h b/include/linux/qed/qed_nvmetcp_if.h
new file mode 100644
index 000000000000..76868bdf0883
--- /dev/null
+++ b/include/linux/qed/qed_nvmetcp_if.h
@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#ifndef _QED_NVMETCP_IF_H
+#define _QED_NVMETCP_IF_H
+#include <linux/types.h>
+#include <linux/qed/qed_if.h>
+
+#define QED_NVMETCP_MAX_IO_SIZE	0x800000
+
+typedef int (*nvmetcp_event_cb_t) (void *context,
+				   u8 fw_event_code, void *fw_handle);
+
+struct qed_dev_nvmetcp_info {
+	struct qed_dev_info common;
+	u8 port_id;  /* Physical port */
+	u8 num_cqs;
+};
+
+#define MAX_TID_BLOCKS_NVMETCP (512)
+struct qed_nvmetcp_tid {
+	u32 size;		/* In bytes per task */
+	u32 num_tids_per_block;
+	u8 *blocks[MAX_TID_BLOCKS_NVMETCP];
+};
+
+struct qed_nvmetcp_cb_ops {
+	struct qed_common_cb_ops common;
+};
+
+/**
+ * struct qed_nvmetcp_ops - qed NVMeTCP operations.
+ * @common:		common operations pointer
+ * @ll2:		light L2 operations pointer
+ * @fill_dev_info:	fills NVMeTCP specific information
+ *			@param cdev
+ *			@param info
+ *			@return 0 on success, otherwise error value.
+ * @register_ops:	register nvmetcp operations
+ *			@param cdev
+ *			@param ops - specified using qed_nvmetcp_cb_ops
+ *			@param cookie - driver private
+ * @start:		nvmetcp in FW
+ *			@param cdev
+ *			@param tasks - qed will fill information about tasks
+ *			return 0 on success, otherwise error value.
+ * @stop:		nvmetcp in FW
+ *			@param cdev
+ *			return 0 on success, otherwise error value.
+ */
+struct qed_nvmetcp_ops {
+	const struct qed_common_ops *common;
+
+	const struct qed_ll2_ops *ll2;
+
+	int (*fill_dev_info)(struct qed_dev *cdev,
+			     struct qed_dev_nvmetcp_info *info);
+
+	void (*register_ops)(struct qed_dev *cdev,
+			     struct qed_nvmetcp_cb_ops *ops, void *cookie);
+
+	int (*start)(struct qed_dev *cdev,
+		     struct qed_nvmetcp_tid *tasks,
+		     void *event_context, nvmetcp_event_cb_t async_event_cb);
+
+	int (*stop)(struct qed_dev *cdev);
+};
+
+const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void);
+void qed_put_nvmetcp_ops(void);
+#endif
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 10:16:51 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 20:16:51 +0300
Subject: [PATCH 3/7] qed: Add NVMeTCP Offload Connection Level FW and HW HSI
In-Reply-To: <20210602171655.23581-1-smalin@marvell.com>
References: <20210602171655.23581-1-smalin@marvell.com>
Message-ID: <20210602171655.23581-4-smalin@marvell.com>

This patch introduces the NVMeTCP HSI and HSI functionality in order to
initialize and interact with the HW device as part of the connection level
HSI.

This includes:
- Connection offload: offload a TCP connection to the FW.
- Connection update: update the ICReq-ICResp params
- Connection clear SQ: outstanding IOs FW flush.
- Connection termination: terminate the TCP connection and flush the FW.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c | 557 +++++++++++++++++-
 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h |  52 ++
 drivers/net/ethernet/qlogic/qed/qed_sp.h      |   3 +
 include/linux/qed/nvmetcp_common.h            | 143 +++++
 include/linux/qed/qed_nvmetcp_if.h            |  94 +++
 5 files changed, 847 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
index cb9c71109b2d..7943804e88cd 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
@@ -243,6 +243,555 @@ static int qed_nvmetcp_start(struct qed_dev *cdev,
 	return 0;
 }
 
+static struct qed_hash_nvmetcp_con *qed_nvmetcp_get_hash(struct qed_dev *cdev,
+							 u32 handle)
+{
+	struct qed_hash_nvmetcp_con *hash_con = NULL;
+
+	if (!(cdev->flags & QED_FLAG_STORAGE_STARTED))
+		return NULL;
+
+	hash_for_each_possible(cdev->connections, hash_con, node, handle) {
+		if (hash_con->con->icid == handle)
+			break;
+	}
+
+	if (!hash_con || hash_con->con->icid != handle)
+		return NULL;
+
+	return hash_con;
+}
+
+static int qed_sp_nvmetcp_conn_offload(struct qed_hwfn *p_hwfn,
+				       struct qed_nvmetcp_conn *p_conn,
+				       enum spq_mode comp_mode,
+				       struct qed_spq_comp_cb *p_comp_addr)
+{
+	struct nvmetcp_spe_conn_offload *p_ramrod = NULL;
+	struct tcp_offload_params_opt2 *p_tcp = NULL;
+	struct qed_sp_init_data init_data = { 0 };
+	struct qed_spq_entry *p_ent = NULL;
+	dma_addr_t r2tq_pbl_addr;
+	dma_addr_t xhq_pbl_addr;
+	dma_addr_t uhq_pbl_addr;
+	u16 physical_q;
+	int rc = 0;
+	u8 i;
+
+	/* Get SPQ entry */
+	init_data.cid = p_conn->icid;
+	init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+	init_data.comp_mode = comp_mode;
+	init_data.p_comp_data = p_comp_addr;
+	rc = qed_sp_init_request(p_hwfn, &p_ent,
+				 NVMETCP_RAMROD_CMD_ID_OFFLOAD_CONN,
+				 PROTOCOLID_TCP_ULP, &init_data);
+	if (rc)
+		return rc;
+
+	p_ramrod = &p_ent->ramrod.nvmetcp_conn_offload;
+
+	/* Transmission PQ is the first of the PF */
+	physical_q = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_OFLD);
+	p_conn->physical_q0 = cpu_to_le16(physical_q);
+	p_ramrod->nvmetcp.physical_q0 = cpu_to_le16(physical_q);
+
+	/* nvmetcp Pure-ACK PQ */
+	physical_q = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_ACK);
+	p_conn->physical_q1 = cpu_to_le16(physical_q);
+	p_ramrod->nvmetcp.physical_q1 = cpu_to_le16(physical_q);
+	p_ramrod->conn_id = cpu_to_le16(p_conn->conn_id);
+	DMA_REGPAIR_LE(p_ramrod->nvmetcp.sq_pbl_addr, p_conn->sq_pbl_addr);
+	r2tq_pbl_addr = qed_chain_get_pbl_phys(&p_conn->r2tq);
+	DMA_REGPAIR_LE(p_ramrod->nvmetcp.r2tq_pbl_addr, r2tq_pbl_addr);
+	xhq_pbl_addr = qed_chain_get_pbl_phys(&p_conn->xhq);
+	DMA_REGPAIR_LE(p_ramrod->nvmetcp.xhq_pbl_addr, xhq_pbl_addr);
+	uhq_pbl_addr = qed_chain_get_pbl_phys(&p_conn->uhq);
+	DMA_REGPAIR_LE(p_ramrod->nvmetcp.uhq_pbl_addr, uhq_pbl_addr);
+	p_ramrod->nvmetcp.flags = p_conn->offl_flags;
+	p_ramrod->nvmetcp.default_cq = p_conn->default_cq;
+	p_ramrod->nvmetcp.initial_ack = 0;
+	DMA_REGPAIR_LE(p_ramrod->nvmetcp.nvmetcp.cccid_itid_table_addr,
+		       p_conn->nvmetcp_cccid_itid_table_addr);
+	p_ramrod->nvmetcp.nvmetcp.cccid_max_range =
+		 cpu_to_le16(p_conn->nvmetcp_cccid_max_range);
+	p_tcp = &p_ramrod->tcp;
+	qed_set_fw_mac_addr(&p_tcp->remote_mac_addr_hi,
+			    &p_tcp->remote_mac_addr_mid,
+			    &p_tcp->remote_mac_addr_lo, p_conn->remote_mac);
+	qed_set_fw_mac_addr(&p_tcp->local_mac_addr_hi,
+			    &p_tcp->local_mac_addr_mid,
+			    &p_tcp->local_mac_addr_lo, p_conn->local_mac);
+	p_tcp->vlan_id = cpu_to_le16(p_conn->vlan_id);
+	p_tcp->flags = cpu_to_le16(p_conn->tcp_flags);
+	p_tcp->ip_version = p_conn->ip_version;
+	if (p_tcp->ip_version == TCP_IPV6) {
+		for (i = 0; i < 4; i++) {
+			p_tcp->remote_ip[i] = cpu_to_le32(p_conn->remote_ip[i]);
+			p_tcp->local_ip[i] = cpu_to_le32(p_conn->local_ip[i]);
+		}
+	} else {
+		p_tcp->remote_ip[0] = cpu_to_le32(p_conn->remote_ip[0]);
+		p_tcp->local_ip[0] = cpu_to_le32(p_conn->local_ip[0]);
+	}
+
+	p_tcp->flow_label = cpu_to_le32(p_conn->flow_label);
+	p_tcp->ttl = p_conn->ttl;
+	p_tcp->tos_or_tc = p_conn->tos_or_tc;
+	p_tcp->remote_port = cpu_to_le16(p_conn->remote_port);
+	p_tcp->local_port = cpu_to_le16(p_conn->local_port);
+	p_tcp->mss = cpu_to_le16(p_conn->mss);
+	p_tcp->rcv_wnd_scale = p_conn->rcv_wnd_scale;
+	p_tcp->connect_mode = p_conn->connect_mode;
+	p_tcp->cwnd = cpu_to_le32(p_conn->cwnd);
+	p_tcp->ka_max_probe_cnt = p_conn->ka_max_probe_cnt;
+	p_tcp->ka_timeout = cpu_to_le32(p_conn->ka_timeout);
+	p_tcp->max_rt_time = cpu_to_le32(p_conn->max_rt_time);
+	p_tcp->ka_interval = cpu_to_le32(p_conn->ka_interval);
+
+	return qed_spq_post(p_hwfn, p_ent, NULL);
+}
+
+static int qed_sp_nvmetcp_conn_update(struct qed_hwfn *p_hwfn,
+				      struct qed_nvmetcp_conn *p_conn,
+				      enum spq_mode comp_mode,
+				      struct qed_spq_comp_cb *p_comp_addr)
+{
+	struct nvmetcp_conn_update_ramrod_params *p_ramrod = NULL;
+	struct qed_spq_entry *p_ent = NULL;
+	struct qed_sp_init_data init_data;
+	int rc = -EINVAL;
+	u32 dval;
+
+	/* Get SPQ entry */
+	memset(&init_data, 0, sizeof(init_data));
+	init_data.cid = p_conn->icid;
+	init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+	init_data.comp_mode = comp_mode;
+	init_data.p_comp_data = p_comp_addr;
+
+	rc = qed_sp_init_request(p_hwfn, &p_ent,
+				 NVMETCP_RAMROD_CMD_ID_UPDATE_CONN,
+				 PROTOCOLID_TCP_ULP, &init_data);
+	if (rc)
+		return rc;
+
+	p_ramrod = &p_ent->ramrod.nvmetcp_conn_update;
+	p_ramrod->conn_id = cpu_to_le16(p_conn->conn_id);
+	p_ramrod->flags = p_conn->update_flag;
+	p_ramrod->max_seq_size = cpu_to_le32(p_conn->max_seq_size);
+	dval = p_conn->max_recv_pdu_length;
+	p_ramrod->max_recv_pdu_length = cpu_to_le32(dval);
+	dval = p_conn->max_send_pdu_length;
+	p_ramrod->max_send_pdu_length = cpu_to_le32(dval);
+	p_ramrod->first_seq_length = cpu_to_le32(p_conn->first_seq_length);
+
+	return qed_spq_post(p_hwfn, p_ent, NULL);
+}
+
+static int qed_sp_nvmetcp_conn_terminate(struct qed_hwfn *p_hwfn,
+					 struct qed_nvmetcp_conn *p_conn,
+					 enum spq_mode comp_mode,
+					 struct qed_spq_comp_cb *p_comp_addr)
+{
+	struct nvmetcp_spe_conn_termination *p_ramrod = NULL;
+	struct qed_spq_entry *p_ent = NULL;
+	struct qed_sp_init_data init_data;
+	int rc = -EINVAL;
+
+	/* Get SPQ entry */
+	memset(&init_data, 0, sizeof(init_data));
+	init_data.cid = p_conn->icid;
+	init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+	init_data.comp_mode = comp_mode;
+	init_data.p_comp_data = p_comp_addr;
+	rc = qed_sp_init_request(p_hwfn, &p_ent,
+				 NVMETCP_RAMROD_CMD_ID_TERMINATION_CONN,
+				 PROTOCOLID_TCP_ULP, &init_data);
+	if (rc)
+		return rc;
+
+	p_ramrod = &p_ent->ramrod.nvmetcp_conn_terminate;
+	p_ramrod->conn_id = cpu_to_le16(p_conn->conn_id);
+	p_ramrod->abortive = p_conn->abortive_dsconnect;
+
+	return qed_spq_post(p_hwfn, p_ent, NULL);
+}
+
+static int qed_sp_nvmetcp_conn_clear_sq(struct qed_hwfn *p_hwfn,
+					struct qed_nvmetcp_conn *p_conn,
+					enum spq_mode comp_mode,
+					struct qed_spq_comp_cb *p_comp_addr)
+{
+	struct qed_spq_entry *p_ent = NULL;
+	struct qed_sp_init_data init_data;
+	int rc = -EINVAL;
+
+	/* Get SPQ entry */
+	memset(&init_data, 0, sizeof(init_data));
+	init_data.cid = p_conn->icid;
+	init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+	init_data.comp_mode = comp_mode;
+	init_data.p_comp_data = p_comp_addr;
+	rc = qed_sp_init_request(p_hwfn, &p_ent,
+				 NVMETCP_RAMROD_CMD_ID_CLEAR_SQ,
+				 PROTOCOLID_TCP_ULP, &init_data);
+	if (rc)
+		return rc;
+
+	return qed_spq_post(p_hwfn, p_ent, NULL);
+}
+
+static void __iomem *qed_nvmetcp_get_db_addr(struct qed_hwfn *p_hwfn, u32 cid)
+{
+	return (u8 __iomem *)p_hwfn->doorbells +
+			     qed_db_addr(cid, DQ_DEMS_LEGACY);
+}
+
+static int qed_nvmetcp_allocate_connection(struct qed_hwfn *p_hwfn,
+					   struct qed_nvmetcp_conn **p_out_conn)
+{
+	struct qed_chain_init_params params = {
+		.mode		= QED_CHAIN_MODE_PBL,
+		.intended_use	= QED_CHAIN_USE_TO_CONSUME_PRODUCE,
+		.cnt_type	= QED_CHAIN_CNT_TYPE_U16,
+	};
+	struct qed_nvmetcp_pf_params *p_params = NULL;
+	struct qed_nvmetcp_conn *p_conn = NULL;
+	int rc = 0;
+
+	/* Try finding a free connection that can be used */
+	spin_lock_bh(&p_hwfn->p_nvmetcp_info->lock);
+	if (!list_empty(&p_hwfn->p_nvmetcp_info->free_list))
+		p_conn = list_first_entry(&p_hwfn->p_nvmetcp_info->free_list,
+					  struct qed_nvmetcp_conn, list_entry);
+	if (p_conn) {
+		list_del(&p_conn->list_entry);
+		spin_unlock_bh(&p_hwfn->p_nvmetcp_info->lock);
+		*p_out_conn = p_conn;
+
+		return 0;
+	}
+	spin_unlock_bh(&p_hwfn->p_nvmetcp_info->lock);
+
+	/* Need to allocate a new connection */
+	p_params = &p_hwfn->pf_params.nvmetcp_pf_params;
+	p_conn = kzalloc(sizeof(*p_conn), GFP_KERNEL);
+	if (!p_conn)
+		return -ENOMEM;
+
+	params.num_elems = p_params->num_r2tq_pages_in_ring *
+			   QED_CHAIN_PAGE_SIZE / sizeof(struct nvmetcp_wqe);
+	params.elem_size = sizeof(struct nvmetcp_wqe);
+	rc = qed_chain_alloc(p_hwfn->cdev, &p_conn->r2tq, &params);
+	if (rc)
+		goto nomem_r2tq;
+
+	params.num_elems = p_params->num_uhq_pages_in_ring *
+			   QED_CHAIN_PAGE_SIZE / sizeof(struct iscsi_uhqe);
+	params.elem_size = sizeof(struct iscsi_uhqe);
+	rc = qed_chain_alloc(p_hwfn->cdev, &p_conn->uhq, &params);
+	if (rc)
+		goto nomem_uhq;
+
+	params.elem_size = sizeof(struct iscsi_xhqe);
+	rc = qed_chain_alloc(p_hwfn->cdev, &p_conn->xhq, &params);
+	if (rc)
+		goto nomem;
+
+	p_conn->free_on_delete = true;
+	*p_out_conn = p_conn;
+
+	return 0;
+
+nomem:
+	qed_chain_free(p_hwfn->cdev, &p_conn->uhq);
+nomem_uhq:
+	qed_chain_free(p_hwfn->cdev, &p_conn->r2tq);
+nomem_r2tq:
+	kfree(p_conn);
+
+	return -ENOMEM;
+}
+
+static int qed_nvmetcp_acquire_connection(struct qed_hwfn *p_hwfn,
+					  struct qed_nvmetcp_conn **p_out_conn)
+{
+	struct qed_nvmetcp_conn *p_conn = NULL;
+	int rc = 0;
+	u32 icid;
+
+	spin_lock_bh(&p_hwfn->p_nvmetcp_info->lock);
+	rc = qed_cxt_acquire_cid(p_hwfn, PROTOCOLID_TCP_ULP, &icid);
+	spin_unlock_bh(&p_hwfn->p_nvmetcp_info->lock);
+
+	if (rc)
+		return rc;
+
+	rc = qed_nvmetcp_allocate_connection(p_hwfn, &p_conn);
+	if (rc) {
+		spin_lock_bh(&p_hwfn->p_nvmetcp_info->lock);
+		qed_cxt_release_cid(p_hwfn, icid);
+		spin_unlock_bh(&p_hwfn->p_nvmetcp_info->lock);
+
+		return rc;
+	}
+
+	p_conn->icid = icid;
+	p_conn->conn_id = (u16)icid;
+	p_conn->fw_cid = (p_hwfn->hw_info.opaque_fid << 16) | icid;
+	*p_out_conn = p_conn;
+
+	return rc;
+}
+
+static void qed_nvmetcp_release_connection(struct qed_hwfn *p_hwfn,
+					   struct qed_nvmetcp_conn *p_conn)
+{
+	spin_lock_bh(&p_hwfn->p_nvmetcp_info->lock);
+	list_add_tail(&p_conn->list_entry, &p_hwfn->p_nvmetcp_info->free_list);
+	qed_cxt_release_cid(p_hwfn, p_conn->icid);
+	spin_unlock_bh(&p_hwfn->p_nvmetcp_info->lock);
+}
+
+static void qed_nvmetcp_free_connection(struct qed_hwfn *p_hwfn,
+					struct qed_nvmetcp_conn *p_conn)
+{
+	qed_chain_free(p_hwfn->cdev, &p_conn->xhq);
+	qed_chain_free(p_hwfn->cdev, &p_conn->uhq);
+	qed_chain_free(p_hwfn->cdev, &p_conn->r2tq);
+	kfree(p_conn);
+}
+
+int qed_nvmetcp_alloc(struct qed_hwfn *p_hwfn)
+{
+	struct qed_nvmetcp_info *p_nvmetcp_info;
+
+	p_nvmetcp_info = kzalloc(sizeof(*p_nvmetcp_info), GFP_KERNEL);
+	if (!p_nvmetcp_info)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&p_nvmetcp_info->free_list);
+	p_hwfn->p_nvmetcp_info = p_nvmetcp_info;
+
+	return 0;
+}
+
+void qed_nvmetcp_setup(struct qed_hwfn *p_hwfn)
+{
+	spin_lock_init(&p_hwfn->p_nvmetcp_info->lock);
+}
+
+void qed_nvmetcp_free(struct qed_hwfn *p_hwfn)
+{
+	struct qed_nvmetcp_conn *p_conn = NULL;
+
+	if (!p_hwfn->p_nvmetcp_info)
+		return;
+
+	while (!list_empty(&p_hwfn->p_nvmetcp_info->free_list)) {
+		p_conn = list_first_entry(&p_hwfn->p_nvmetcp_info->free_list,
+					  struct qed_nvmetcp_conn, list_entry);
+		if (p_conn) {
+			list_del(&p_conn->list_entry);
+			qed_nvmetcp_free_connection(p_hwfn, p_conn);
+		}
+	}
+
+	kfree(p_hwfn->p_nvmetcp_info);
+	p_hwfn->p_nvmetcp_info = NULL;
+}
+
+static int qed_nvmetcp_acquire_conn(struct qed_dev *cdev,
+				    u32 *handle,
+				    u32 *fw_cid, void __iomem **p_doorbell)
+{
+	struct qed_hash_nvmetcp_con *hash_con;
+	int rc;
+
+	/* Allocate a hashed connection */
+	hash_con = kzalloc(sizeof(*hash_con), GFP_ATOMIC);
+	if (!hash_con)
+		return -ENOMEM;
+
+	/* Acquire the connection */
+	rc = qed_nvmetcp_acquire_connection(QED_AFFIN_HWFN(cdev),
+					    &hash_con->con);
+	if (rc) {
+		DP_NOTICE(cdev, "Failed to acquire Connection\n");
+		kfree(hash_con);
+
+		return rc;
+	}
+
+	/* Added the connection to hash table */
+	*handle = hash_con->con->icid;
+	*fw_cid = hash_con->con->fw_cid;
+	hash_add(cdev->connections, &hash_con->node, *handle);
+	if (p_doorbell)
+		*p_doorbell = qed_nvmetcp_get_db_addr(QED_AFFIN_HWFN(cdev),
+						      *handle);
+
+	return 0;
+}
+
+static int qed_nvmetcp_release_conn(struct qed_dev *cdev, u32 handle)
+{
+	struct qed_hash_nvmetcp_con *hash_con;
+
+	hash_con = qed_nvmetcp_get_hash(cdev, handle);
+	if (!hash_con) {
+		DP_NOTICE(cdev, "Failed to find connection for handle %d\n",
+			  handle);
+
+		return -EINVAL;
+	}
+
+	hlist_del(&hash_con->node);
+	qed_nvmetcp_release_connection(QED_AFFIN_HWFN(cdev), hash_con->con);
+	kfree(hash_con);
+
+	return 0;
+}
+
+static int qed_nvmetcp_offload_conn(struct qed_dev *cdev, u32 handle,
+				    struct qed_nvmetcp_params_offload *conn_info)
+{
+	struct qed_hash_nvmetcp_con *hash_con;
+	struct qed_nvmetcp_conn *con;
+
+	hash_con = qed_nvmetcp_get_hash(cdev, handle);
+	if (!hash_con) {
+		DP_NOTICE(cdev, "Failed to find connection for handle %d\n",
+			  handle);
+
+		return -EINVAL;
+	}
+
+	/* Update the connection with information from the params */
+	con = hash_con->con;
+
+	/* FW initializations */
+	con->layer_code = NVMETCP_SLOW_PATH_LAYER_CODE;
+	con->sq_pbl_addr = conn_info->sq_pbl_addr;
+	con->nvmetcp_cccid_max_range = conn_info->nvmetcp_cccid_max_range;
+	con->nvmetcp_cccid_itid_table_addr = conn_info->nvmetcp_cccid_itid_table_addr;
+	con->default_cq = conn_info->default_cq;
+	SET_FIELD(con->offl_flags, NVMETCP_CONN_OFFLOAD_PARAMS_TARGET_MODE, 0);
+	SET_FIELD(con->offl_flags, NVMETCP_CONN_OFFLOAD_PARAMS_NVMETCP_MODE, 1);
+	SET_FIELD(con->offl_flags, NVMETCP_CONN_OFFLOAD_PARAMS_TCP_ON_CHIP_1B, 1);
+
+	/* Networking and TCP stack initializations */
+	ether_addr_copy(con->local_mac, conn_info->src.mac);
+	ether_addr_copy(con->remote_mac, conn_info->dst.mac);
+	memcpy(con->local_ip, conn_info->src.ip, sizeof(con->local_ip));
+	memcpy(con->remote_ip, conn_info->dst.ip, sizeof(con->remote_ip));
+	con->local_port = conn_info->src.port;
+	con->remote_port = conn_info->dst.port;
+	con->vlan_id = conn_info->vlan_id;
+
+	if (conn_info->timestamp_en)
+		SET_FIELD(con->tcp_flags, TCP_OFFLOAD_PARAMS_OPT2_TS_EN, 1);
+
+	if (conn_info->delayed_ack_en)
+		SET_FIELD(con->tcp_flags, TCP_OFFLOAD_PARAMS_OPT2_DA_EN, 1);
+
+	if (conn_info->tcp_keep_alive_en)
+		SET_FIELD(con->tcp_flags, TCP_OFFLOAD_PARAMS_OPT2_KA_EN, 1);
+
+	if (conn_info->ecn_en)
+		SET_FIELD(con->tcp_flags, TCP_OFFLOAD_PARAMS_OPT2_ECN_EN, 1);
+
+	con->ip_version = conn_info->ip_version;
+	con->flow_label = QED_TCP_FLOW_LABEL;
+	con->ka_max_probe_cnt = conn_info->ka_max_probe_cnt;
+	con->ka_timeout = conn_info->ka_timeout;
+	con->ka_interval = conn_info->ka_interval;
+	con->max_rt_time = conn_info->max_rt_time;
+	con->ttl = conn_info->ttl;
+	con->tos_or_tc = conn_info->tos_or_tc;
+	con->mss = conn_info->mss;
+	con->cwnd = conn_info->cwnd;
+	con->rcv_wnd_scale = conn_info->rcv_wnd_scale;
+	con->connect_mode = 0;
+
+	return qed_sp_nvmetcp_conn_offload(QED_AFFIN_HWFN(cdev), con,
+					 QED_SPQ_MODE_EBLOCK, NULL);
+}
+
+static int qed_nvmetcp_update_conn(struct qed_dev *cdev,
+				   u32 handle,
+				   struct qed_nvmetcp_params_update *conn_info)
+{
+	struct qed_hash_nvmetcp_con *hash_con;
+	struct qed_nvmetcp_conn *con;
+
+	hash_con = qed_nvmetcp_get_hash(cdev, handle);
+	if (!hash_con) {
+		DP_NOTICE(cdev, "Failed to find connection for handle %d\n",
+			  handle);
+
+		return -EINVAL;
+	}
+
+	/* Update the connection with information from the params */
+	con = hash_con->con;
+	SET_FIELD(con->update_flag,
+		  ISCSI_CONN_UPDATE_RAMROD_PARAMS_INITIAL_R2T, 0);
+	SET_FIELD(con->update_flag,
+		  ISCSI_CONN_UPDATE_RAMROD_PARAMS_IMMEDIATE_DATA, 1);
+	if (conn_info->hdr_digest_en)
+		SET_FIELD(con->update_flag, ISCSI_CONN_UPDATE_RAMROD_PARAMS_HD_EN, 1);
+
+	if (conn_info->data_digest_en)
+		SET_FIELD(con->update_flag, ISCSI_CONN_UPDATE_RAMROD_PARAMS_DD_EN, 1);
+
+	/* Placeholder - initialize pfv, cpda, hpda */
+
+	con->max_seq_size = conn_info->max_io_size;
+	con->max_recv_pdu_length = conn_info->max_recv_pdu_length;
+	con->max_send_pdu_length = conn_info->max_send_pdu_length;
+	con->first_seq_length = conn_info->max_io_size;
+
+	return qed_sp_nvmetcp_conn_update(QED_AFFIN_HWFN(cdev), con,
+					QED_SPQ_MODE_EBLOCK, NULL);
+}
+
+static int qed_nvmetcp_clear_conn_sq(struct qed_dev *cdev, u32 handle)
+{
+	struct qed_hash_nvmetcp_con *hash_con;
+
+	hash_con = qed_nvmetcp_get_hash(cdev, handle);
+	if (!hash_con) {
+		DP_NOTICE(cdev, "Failed to find connection for handle %d\n",
+			  handle);
+
+		return -EINVAL;
+	}
+
+	return qed_sp_nvmetcp_conn_clear_sq(QED_AFFIN_HWFN(cdev), hash_con->con,
+					    QED_SPQ_MODE_EBLOCK, NULL);
+}
+
+static int qed_nvmetcp_destroy_conn(struct qed_dev *cdev,
+				    u32 handle, u8 abrt_conn)
+{
+	struct qed_hash_nvmetcp_con *hash_con;
+
+	hash_con = qed_nvmetcp_get_hash(cdev, handle);
+	if (!hash_con) {
+		DP_NOTICE(cdev, "Failed to find connection for handle %d\n",
+			  handle);
+
+		return -EINVAL;
+	}
+
+	hash_con->con->abortive_dsconnect = abrt_conn;
+
+	return qed_sp_nvmetcp_conn_terminate(QED_AFFIN_HWFN(cdev), hash_con->con,
+					   QED_SPQ_MODE_EBLOCK, NULL);
+}
+
 static const struct qed_nvmetcp_ops qed_nvmetcp_ops_pass = {
 	.common = &qed_common_ops_pass,
 	.ll2 = &qed_ll2_ops_pass,
@@ -250,8 +799,12 @@ static const struct qed_nvmetcp_ops qed_nvmetcp_ops_pass = {
 	.register_ops = &qed_register_nvmetcp_ops,
 	.start = &qed_nvmetcp_start,
 	.stop = &qed_nvmetcp_stop,
-
-	/* Placeholder - Connection level ops */
+	.acquire_conn = &qed_nvmetcp_acquire_conn,
+	.release_conn = &qed_nvmetcp_release_conn,
+	.offload_conn = &qed_nvmetcp_offload_conn,
+	.update_conn = &qed_nvmetcp_update_conn,
+	.destroy_conn = &qed_nvmetcp_destroy_conn,
+	.clear_sq = &qed_nvmetcp_clear_conn_sq,
 };
 
 const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void)
diff --git a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h
index 774b46ade408..e5e9d075bf4f 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.h
@@ -19,6 +19,7 @@
 #define QED_NVMETCP_FW_CQ_SIZE (4 * 1024)
 
 /* tcp parameters */
+#define QED_TCP_FLOW_LABEL 0
 #define QED_TCP_TWO_MSL_TIMER 4000
 #define QED_TCP_HALF_WAY_CLOSE_TIMEOUT 10
 #define QED_TCP_MAX_FIN_RT 2
@@ -32,6 +33,57 @@ struct qed_nvmetcp_info {
 	nvmetcp_event_cb_t event_cb;
 };
 
+struct qed_hash_nvmetcp_con {
+	struct hlist_node node;
+	struct qed_nvmetcp_conn *con;
+};
+
+struct qed_nvmetcp_conn {
+	struct list_head list_entry;
+	bool free_on_delete;
+	u16 conn_id;
+	u32 icid;
+	u32 fw_cid;
+	u8 layer_code;
+	u8 offl_flags;
+	u8 connect_mode;
+	dma_addr_t sq_pbl_addr;
+	struct qed_chain r2tq;
+	struct qed_chain xhq;
+	struct qed_chain uhq;
+	u8 local_mac[6];
+	u8 remote_mac[6];
+	u8 ip_version;
+	u8 ka_max_probe_cnt;
+	u16 vlan_id;
+	u16 tcp_flags;
+	u32 remote_ip[4];
+	u32 local_ip[4];
+	u32 flow_label;
+	u32 ka_timeout;
+	u32 ka_interval;
+	u32 max_rt_time;
+	u8 ttl;
+	u8 tos_or_tc;
+	u16 remote_port;
+	u16 local_port;
+	u16 mss;
+	u8 rcv_wnd_scale;
+	u32 rcv_wnd;
+	u32 cwnd;
+	u8 update_flag;
+	u8 default_cq;
+	u8 abortive_dsconnect;
+	u32 max_seq_size;
+	u32 max_recv_pdu_length;
+	u32 max_send_pdu_length;
+	u32 first_seq_length;
+	u16 physical_q0;
+	u16 physical_q1;
+	u16 nvmetcp_cccid_max_range;
+	dma_addr_t nvmetcp_cccid_itid_table_addr;
+};
+
 #if IS_ENABLED(CONFIG_QED_NVMETCP)
 int qed_nvmetcp_alloc(struct qed_hwfn *p_hwfn);
 void qed_nvmetcp_setup(struct qed_hwfn *p_hwfn);
diff --git a/drivers/net/ethernet/qlogic/qed/qed_sp.h b/drivers/net/ethernet/qlogic/qed/qed_sp.h
index 525159e747a5..60ff3222bf55 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_sp.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_sp.h
@@ -101,6 +101,9 @@ union ramrod_data {
 	struct iscsi_spe_conn_termination iscsi_conn_terminate;
 
 	struct nvmetcp_init_ramrod_params nvmetcp_init;
+	struct nvmetcp_spe_conn_offload nvmetcp_conn_offload;
+	struct nvmetcp_conn_update_ramrod_params nvmetcp_conn_update;
+	struct nvmetcp_spe_conn_termination nvmetcp_conn_terminate;
 
 	struct vf_start_ramrod_data vf_start;
 	struct vf_stop_ramrod_data vf_stop;
diff --git a/include/linux/qed/nvmetcp_common.h b/include/linux/qed/nvmetcp_common.h
index e9ccfc07041d..c8836b71b866 100644
--- a/include/linux/qed/nvmetcp_common.h
+++ b/include/linux/qed/nvmetcp_common.h
@@ -6,6 +6,8 @@
 
 #include "tcp_common.h"
 
+#define NVMETCP_SLOW_PATH_LAYER_CODE (6)
+
 /* NVMeTCP firmware function init parameters */
 struct nvmetcp_spe_func_init {
 	__le16 half_way_close_timeout;
@@ -43,6 +45,10 @@ enum nvmetcp_ramrod_cmd_id {
 	NVMETCP_RAMROD_CMD_ID_UNUSED = 0,
 	NVMETCP_RAMROD_CMD_ID_INIT_FUNC = 1,
 	NVMETCP_RAMROD_CMD_ID_DESTROY_FUNC = 2,
+	NVMETCP_RAMROD_CMD_ID_OFFLOAD_CONN = 3,
+	NVMETCP_RAMROD_CMD_ID_UPDATE_CONN = 4,
+	NVMETCP_RAMROD_CMD_ID_TERMINATION_CONN = 5,
+	NVMETCP_RAMROD_CMD_ID_CLEAR_SQ = 6,
 	MAX_NVMETCP_RAMROD_CMD_ID
 };
 
@@ -51,4 +57,141 @@ struct nvmetcp_glbl_queue_entry {
 	struct regpair reserved;
 };
 
+/* NVMeTCP conn level EQEs */
+enum nvmetcp_eqe_opcode {
+	NVMETCP_EVENT_TYPE_INIT_FUNC = 0, /* Response after init Ramrod */
+	NVMETCP_EVENT_TYPE_DESTROY_FUNC, /* Response after destroy Ramrod */
+	NVMETCP_EVENT_TYPE_OFFLOAD_CONN,/* Response after option 2 offload Ramrod */
+	NVMETCP_EVENT_TYPE_UPDATE_CONN, /* Response after update Ramrod */
+	NVMETCP_EVENT_TYPE_CLEAR_SQ, /* Response after clear sq Ramrod */
+	NVMETCP_EVENT_TYPE_TERMINATE_CONN, /* Response after termination Ramrod */
+	NVMETCP_EVENT_TYPE_RESERVED0,
+	NVMETCP_EVENT_TYPE_RESERVED1,
+	NVMETCP_EVENT_TYPE_ASYN_CONNECT_COMPLETE, /* Connect completed (A-syn EQE) */
+	NVMETCP_EVENT_TYPE_ASYN_TERMINATE_DONE, /* Termination completed (A-syn EQE) */
+	NVMETCP_EVENT_TYPE_START_OF_ERROR_TYPES = 10, /* Separate EQs from err EQs */
+	NVMETCP_EVENT_TYPE_ASYN_ABORT_RCVD, /* TCP RST packet receive (A-syn EQE) */
+	NVMETCP_EVENT_TYPE_ASYN_CLOSE_RCVD, /* TCP FIN packet receive (A-syn EQE) */
+	NVMETCP_EVENT_TYPE_ASYN_SYN_RCVD, /* TCP SYN+ACK packet receive (A-syn EQE) */
+	NVMETCP_EVENT_TYPE_ASYN_MAX_RT_TIME, /* TCP max retransmit time (A-syn EQE) */
+	NVMETCP_EVENT_TYPE_ASYN_MAX_RT_CNT, /* TCP max retransmit count (A-syn EQE) */
+	NVMETCP_EVENT_TYPE_ASYN_MAX_KA_PROBES_CNT, /* TCP ka probes count (A-syn EQE) */
+	NVMETCP_EVENT_TYPE_ASYN_FIN_WAIT2, /* TCP fin wait 2 (A-syn EQE) */
+	NVMETCP_EVENT_TYPE_NVMETCP_CONN_ERROR, /* NVMeTCP error response (A-syn EQE) */
+	NVMETCP_EVENT_TYPE_TCP_CONN_ERROR, /* NVMeTCP error - tcp error (A-syn EQE) */
+	MAX_NVMETCP_EQE_OPCODE
+};
+
+struct nvmetcp_conn_offload_section {
+	struct regpair cccid_itid_table_addr; /* CCCID to iTID table address */
+	__le16 cccid_max_range; /* CCCID max value - used for validation */
+	__le16 reserved[3];
+};
+
+/* NVMe TCP connection offload params passed by driver to FW in NVMeTCP offload ramrod */
+struct nvmetcp_conn_offload_params {
+	struct regpair sq_pbl_addr;
+	struct regpair r2tq_pbl_addr;
+	struct regpair xhq_pbl_addr;
+	struct regpair uhq_pbl_addr;
+	__le16 physical_q0;
+	__le16 physical_q1;
+	u8 flags;
+#define NVMETCP_CONN_OFFLOAD_PARAMS_TCP_ON_CHIP_1B_MASK 0x1
+#define NVMETCP_CONN_OFFLOAD_PARAMS_TCP_ON_CHIP_1B_SHIFT 0
+#define NVMETCP_CONN_OFFLOAD_PARAMS_TARGET_MODE_MASK 0x1
+#define NVMETCP_CONN_OFFLOAD_PARAMS_TARGET_MODE_SHIFT 1
+#define NVMETCP_CONN_OFFLOAD_PARAMS_RESTRICTED_MODE_MASK 0x1
+#define NVMETCP_CONN_OFFLOAD_PARAMS_RESTRICTED_MODE_SHIFT 2
+#define NVMETCP_CONN_OFFLOAD_PARAMS_NVMETCP_MODE_MASK 0x1
+#define NVMETCP_CONN_OFFLOAD_PARAMS_NVMETCP_MODE_SHIFT 3
+#define NVMETCP_CONN_OFFLOAD_PARAMS_RESERVED1_MASK 0xF
+#define NVMETCP_CONN_OFFLOAD_PARAMS_RESERVED1_SHIFT 4
+	u8 default_cq;
+	__le16 reserved0;
+	__le32 reserved1;
+	__le32 initial_ack;
+
+	struct nvmetcp_conn_offload_section nvmetcp; /* NVMe/TCP section */
+};
+
+/* NVMe TCP and TCP connection offload params passed by driver to FW in NVMeTCP offload ramrod. */
+struct nvmetcp_spe_conn_offload {
+	__le16 reserved;
+	__le16 conn_id;
+	__le32 fw_cid;
+	struct nvmetcp_conn_offload_params nvmetcp;
+	struct tcp_offload_params_opt2 tcp;
+};
+
+/* NVMeTCP connection update params passed by driver to FW in NVMETCP update ramrod. */
+struct nvmetcp_conn_update_ramrod_params {
+	__le16 reserved0;
+	__le16 conn_id;
+	__le32 reserved1;
+	u8 flags;
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_HD_EN_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_HD_EN_SHIFT 0
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_DD_EN_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_DD_EN_SHIFT 1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED0_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED0_SHIFT 2
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED1_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED1_DATA_SHIFT 3
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED2_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED2_SHIFT 4
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED3_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED3_SHIFT 5
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED4_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED4_SHIFT 6
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED5_MASK 0x1
+#define NVMETCP_CONN_UPDATE_RAMROD_PARAMS_RESERVED5_SHIFT 7
+	u8 reserved3[3];
+	__le32 max_seq_size;
+	__le32 max_send_pdu_length;
+	__le32 max_recv_pdu_length;
+	__le32 first_seq_length;
+	__le32 reserved4[5];
+};
+
+/* NVMeTCP connection termination request */
+struct nvmetcp_spe_conn_termination {
+	__le16 reserved0;
+	__le16 conn_id;
+	__le32 reserved1;
+	u8 abortive;
+	u8 reserved2[7];
+	struct regpair reserved3;
+	struct regpair reserved4;
+};
+
+struct nvmetcp_dif_flags {
+	u8 flags;
+};
+
+enum nvmetcp_wqe_type {
+	NVMETCP_WQE_TYPE_NORMAL,
+	NVMETCP_WQE_TYPE_TASK_CLEANUP,
+	NVMETCP_WQE_TYPE_MIDDLE_PATH,
+	NVMETCP_WQE_TYPE_IC,
+	MAX_NVMETCP_WQE_TYPE
+};
+
+struct nvmetcp_wqe {
+	__le16 task_id;
+	u8 flags;
+#define NVMETCP_WQE_WQE_TYPE_MASK 0x7 /* [use nvmetcp_wqe_type] */
+#define NVMETCP_WQE_WQE_TYPE_SHIFT 0
+#define NVMETCP_WQE_NUM_SGES_MASK 0xF
+#define NVMETCP_WQE_NUM_SGES_SHIFT 3
+#define NVMETCP_WQE_RESPONSE_MASK 0x1
+#define NVMETCP_WQE_RESPONSE_SHIFT 7
+	struct nvmetcp_dif_flags prot_flags;
+	__le32 contlen_cdbsize;
+#define NVMETCP_WQE_CONT_LEN_MASK 0xFFFFFF
+#define NVMETCP_WQE_CONT_LEN_SHIFT 0
+#define NVMETCP_WQE_CDB_SIZE_OR_NVMETCP_CMD_MASK 0xFF
+#define NVMETCP_WQE_CDB_SIZE_OR_NVMETCP_CMD_SHIFT 24
+};
+
 #endif /* __NVMETCP_COMMON__ */
diff --git a/include/linux/qed/qed_nvmetcp_if.h b/include/linux/qed/qed_nvmetcp_if.h
index 76868bdf0883..5baf1c5ce798 100644
--- a/include/linux/qed/qed_nvmetcp_if.h
+++ b/include/linux/qed/qed_nvmetcp_if.h
@@ -24,6 +24,50 @@ struct qed_nvmetcp_tid {
 	u8 *blocks[MAX_TID_BLOCKS_NVMETCP];
 };
 
+struct qed_nvmetcp_id_params {
+	u8 mac[ETH_ALEN];
+	u32 ip[4];
+	u16 port;
+};
+
+struct qed_nvmetcp_params_offload {
+	/* FW initializations */
+	dma_addr_t sq_pbl_addr;
+	dma_addr_t nvmetcp_cccid_itid_table_addr;
+	u16 nvmetcp_cccid_max_range;
+	u8 default_cq;
+
+	/* Networking and TCP stack initializations */
+	struct qed_nvmetcp_id_params src;
+	struct qed_nvmetcp_id_params dst;
+	u32 ka_timeout;
+	u32 ka_interval;
+	u32 max_rt_time;
+	u32 cwnd;
+	u16 mss;
+	u16 vlan_id;
+	bool timestamp_en;
+	bool delayed_ack_en;
+	bool tcp_keep_alive_en;
+	bool ecn_en;
+	u8 ip_version;
+	u8 ka_max_probe_cnt;
+	u8 ttl;
+	u8 tos_or_tc;
+	u8 rcv_wnd_scale;
+};
+
+struct qed_nvmetcp_params_update {
+	u32 max_io_size;
+	u32 max_recv_pdu_length;
+	u32 max_send_pdu_length;
+
+	/* Placeholder: pfv, cpda, hpda */
+
+	bool hdr_digest_en;
+	bool data_digest_en;
+};
+
 struct qed_nvmetcp_cb_ops {
 	struct qed_common_cb_ops common;
 };
@@ -47,6 +91,38 @@ struct qed_nvmetcp_cb_ops {
  * @stop:		nvmetcp in FW
  *			@param cdev
  *			return 0 on success, otherwise error value.
+ * @acquire_conn:	acquire a new nvmetcp connection
+ *			@param cdev
+ *			@param handle - qed will fill handle that should be
+ *				used henceforth as identifier of the
+ *				connection.
+ *			@param p_doorbell - qed will fill the address of the
+ *				doorbell.
+ *			@return 0 on sucesss, otherwise error value.
+ * @release_conn:	release a previously acquired nvmetcp connection
+ *			@param cdev
+ *			@param handle - the connection handle.
+ *			@return 0 on success, otherwise error value.
+ * @offload_conn:	configures an offloaded connection
+ *			@param cdev
+ *			@param handle - the connection handle.
+ *			@param conn_info - the configuration to use for the
+ *				offload.
+ *			@return 0 on success, otherwise error value.
+ * @update_conn:	updates an offloaded connection
+ *			@param cdev
+ *			@param handle - the connection handle.
+ *			@param conn_info - the configuration to use for the
+ *				offload.
+ *			@return 0 on success, otherwise error value.
+ * @destroy_conn:	stops an offloaded connection
+ *			@param cdev
+ *			@param handle - the connection handle.
+ *			@return 0 on success, otherwise error value.
+ * @clear_sq:		clear all task in sq
+ *			@param cdev
+ *			@param handle - the connection handle.
+ *			@return 0 on success, otherwise error value.
  */
 struct qed_nvmetcp_ops {
 	const struct qed_common_ops *common;
@@ -64,6 +140,24 @@ struct qed_nvmetcp_ops {
 		     void *event_context, nvmetcp_event_cb_t async_event_cb);
 
 	int (*stop)(struct qed_dev *cdev);
+
+	int (*acquire_conn)(struct qed_dev *cdev,
+			    u32 *handle,
+			    u32 *fw_cid, void __iomem **p_doorbell);
+
+	int (*release_conn)(struct qed_dev *cdev, u32 handle);
+
+	int (*offload_conn)(struct qed_dev *cdev,
+			    u32 handle,
+			    struct qed_nvmetcp_params_offload *conn_info);
+
+	int (*update_conn)(struct qed_dev *cdev,
+			   u32 handle,
+			   struct qed_nvmetcp_params_update *conn_info);
+
+	int (*destroy_conn)(struct qed_dev *cdev, u32 handle, u8 abrt_conn);
+
+	int (*clear_sq)(struct qed_dev *cdev, u32 handle);
 };
 
 const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void);
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 10:16:49 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 20:16:49 +0300
Subject: [PATCH 1/7] qed: Add TCP_ULP FW resource layout
In-Reply-To: <20210602171655.23581-1-smalin@marvell.com>
References: <20210602171655.23581-1-smalin@marvell.com>
Message-ID: <20210602171655.23581-2-smalin@marvell.com>

From: Omkar Kulkarni <okulkarni at marvell.com>

Add TCP_ULP as a storage common TCP offload FW resource layout.
This will be used by the core driver (QED) for both the NVMeTCP and iSCSI.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/net/ethernet/qlogic/qed/qed_cxt.c     | 18 ++++++++---------
 drivers/net/ethernet/qlogic/qed/qed_cxt.h     |  2 +-
 drivers/net/ethernet/qlogic/qed/qed_dev.c     |  2 +-
 drivers/net/ethernet/qlogic/qed/qed_hsi.h     |  2 +-
 drivers/net/ethernet/qlogic/qed/qed_iscsi.c   | 20 +++++++++----------
 drivers/net/ethernet/qlogic/qed/qed_ll2.c     |  8 ++++----
 drivers/net/ethernet/qlogic/qed/qed_ooo.c     |  2 +-
 .../net/ethernet/qlogic/qed/qed_sp_commands.c |  2 +-
 include/linux/qed/common_hsi.h                |  2 +-
 include/linux/qed/qed_ll2_if.h                |  2 +-
 10 files changed, 30 insertions(+), 30 deletions(-)

diff --git a/drivers/net/ethernet/qlogic/qed/qed_cxt.c b/drivers/net/ethernet/qlogic/qed/qed_cxt.c
index 0a22f8ce9a2c..fcabbaa518df 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_cxt.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_cxt.c
@@ -94,14 +94,14 @@ struct src_ent {
 
 static bool src_proto(enum protocol_type type)
 {
-	return type == PROTOCOLID_ISCSI ||
+	return type == PROTOCOLID_TCP_ULP ||
 	       type == PROTOCOLID_FCOE ||
 	       type == PROTOCOLID_IWARP;
 }
 
 static bool tm_cid_proto(enum protocol_type type)
 {
-	return type == PROTOCOLID_ISCSI ||
+	return type == PROTOCOLID_TCP_ULP ||
 	       type == PROTOCOLID_FCOE ||
 	       type == PROTOCOLID_ROCE ||
 	       type == PROTOCOLID_IWARP;
@@ -2090,13 +2090,13 @@ int qed_cxt_set_pf_params(struct qed_hwfn *p_hwfn, u32 rdma_tasks)
 
 		if (p_params->num_cons && p_params->num_tasks) {
 			qed_cxt_set_proto_cid_count(p_hwfn,
-						    PROTOCOLID_ISCSI,
+						    PROTOCOLID_TCP_ULP,
 						    p_params->num_cons,
 						    0);
 
 			qed_cxt_set_proto_tid_count(p_hwfn,
-						    PROTOCOLID_ISCSI,
-						    QED_CXT_ISCSI_TID_SEG,
+						    PROTOCOLID_TCP_ULP,
+						    QED_CXT_TCP_ULP_TID_SEG,
 						    0,
 						    p_params->num_tasks,
 						    true);
@@ -2129,8 +2129,8 @@ int qed_cxt_get_tid_mem_info(struct qed_hwfn *p_hwfn,
 		seg = QED_CXT_FCOE_TID_SEG;
 		break;
 	case QED_PCI_ISCSI:
-		proto = PROTOCOLID_ISCSI;
-		seg = QED_CXT_ISCSI_TID_SEG;
+		proto = PROTOCOLID_TCP_ULP;
+		seg = QED_CXT_TCP_ULP_TID_SEG;
 		break;
 	default:
 		return -EINVAL;
@@ -2455,8 +2455,8 @@ int qed_cxt_get_task_ctx(struct qed_hwfn *p_hwfn,
 		seg = QED_CXT_FCOE_TID_SEG;
 		break;
 	case QED_PCI_ISCSI:
-		proto = PROTOCOLID_ISCSI;
-		seg = QED_CXT_ISCSI_TID_SEG;
+		proto = PROTOCOLID_TCP_ULP;
+		seg = QED_CXT_TCP_ULP_TID_SEG;
 		break;
 	default:
 		return -EINVAL;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_cxt.h b/drivers/net/ethernet/qlogic/qed/qed_cxt.h
index 056e79620a0e..8adb7ed0c12d 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_cxt.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_cxt.h
@@ -50,7 +50,7 @@ int qed_cxt_get_cid_info(struct qed_hwfn *p_hwfn,
 int qed_cxt_get_tid_mem_info(struct qed_hwfn *p_hwfn,
 			     struct qed_tid_mem *p_info);
 
-#define QED_CXT_ISCSI_TID_SEG	PROTOCOLID_ISCSI
+#define QED_CXT_TCP_ULP_TID_SEG	PROTOCOLID_TCP_ULP
 #define QED_CXT_ROCE_TID_SEG	PROTOCOLID_ROCE
 #define QED_CXT_FCOE_TID_SEG	PROTOCOLID_FCOE
 enum qed_cxt_elem_type {
diff --git a/drivers/net/ethernet/qlogic/qed/qed_dev.c b/drivers/net/ethernet/qlogic/qed/qed_dev.c
index d2f5855b2ea7..c231d0e56571 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_dev.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_dev.c
@@ -2266,7 +2266,7 @@ int qed_resc_alloc(struct qed_dev *cdev)
 		} else if (p_hwfn->hw_info.personality == QED_PCI_ISCSI) {
 			num_cons =
 			    qed_cxt_get_proto_cid_count(p_hwfn,
-							PROTOCOLID_ISCSI,
+							PROTOCOLID_TCP_ULP,
 							NULL);
 			n_eqes += 2 * num_cons;
 		}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_hsi.h b/drivers/net/ethernet/qlogic/qed/qed_hsi.h
index 559df9f4d656..9dbeb2efdc51 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_hsi.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_hsi.h
@@ -1118,7 +1118,7 @@ struct outer_tag_config_struct {
 /* personality per PF */
 enum personality_type {
 	BAD_PERSONALITY_TYP,
-	PERSONALITY_ISCSI,
+	PERSONALITY_TCP_ULP,
 	PERSONALITY_FCOE,
 	PERSONALITY_RDMA_AND_ETH,
 	PERSONALITY_RDMA,
diff --git a/drivers/net/ethernet/qlogic/qed/qed_iscsi.c b/drivers/net/ethernet/qlogic/qed/qed_iscsi.c
index 448567a1f520..db926d8b3033 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_iscsi.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_iscsi.c
@@ -158,7 +158,7 @@ qed_sp_iscsi_func_start(struct qed_hwfn *p_hwfn,
 
 	rc = qed_sp_init_request(p_hwfn, &p_ent,
 				 ISCSI_RAMROD_CMD_ID_INIT_FUNC,
-				 PROTOCOLID_ISCSI, &init_data);
+				 PROTOCOLID_TCP_ULP, &init_data);
 	if (rc)
 		return rc;
 
@@ -250,7 +250,7 @@ qed_sp_iscsi_func_start(struct qed_hwfn *p_hwfn,
 	p_hwfn->p_iscsi_info->event_context = event_context;
 	p_hwfn->p_iscsi_info->event_cb = async_event_cb;
 
-	qed_spq_register_async_cb(p_hwfn, PROTOCOLID_ISCSI,
+	qed_spq_register_async_cb(p_hwfn, PROTOCOLID_TCP_ULP,
 				  qed_iscsi_async_event);
 
 	return qed_spq_post(p_hwfn, p_ent, NULL);
@@ -286,7 +286,7 @@ static int qed_sp_iscsi_conn_offload(struct qed_hwfn *p_hwfn,
 
 	rc = qed_sp_init_request(p_hwfn, &p_ent,
 				 ISCSI_RAMROD_CMD_ID_OFFLOAD_CONN,
-				 PROTOCOLID_ISCSI, &init_data);
+				 PROTOCOLID_TCP_ULP, &init_data);
 	if (rc)
 		return rc;
 
@@ -465,7 +465,7 @@ static int qed_sp_iscsi_conn_update(struct qed_hwfn *p_hwfn,
 
 	rc = qed_sp_init_request(p_hwfn, &p_ent,
 				 ISCSI_RAMROD_CMD_ID_UPDATE_CONN,
-				 PROTOCOLID_ISCSI, &init_data);
+				 PROTOCOLID_TCP_ULP, &init_data);
 	if (rc)
 		return rc;
 
@@ -506,7 +506,7 @@ qed_sp_iscsi_mac_update(struct qed_hwfn *p_hwfn,
 
 	rc = qed_sp_init_request(p_hwfn, &p_ent,
 				 ISCSI_RAMROD_CMD_ID_MAC_UPDATE,
-				 PROTOCOLID_ISCSI, &init_data);
+				 PROTOCOLID_TCP_ULP, &init_data);
 	if (rc)
 		return rc;
 
@@ -548,7 +548,7 @@ static int qed_sp_iscsi_conn_terminate(struct qed_hwfn *p_hwfn,
 
 	rc = qed_sp_init_request(p_hwfn, &p_ent,
 				 ISCSI_RAMROD_CMD_ID_TERMINATION_CONN,
-				 PROTOCOLID_ISCSI, &init_data);
+				 PROTOCOLID_TCP_ULP, &init_data);
 	if (rc)
 		return rc;
 
@@ -582,7 +582,7 @@ static int qed_sp_iscsi_conn_clear_sq(struct qed_hwfn *p_hwfn,
 
 	rc = qed_sp_init_request(p_hwfn, &p_ent,
 				 ISCSI_RAMROD_CMD_ID_CLEAR_SQ,
-				 PROTOCOLID_ISCSI, &init_data);
+				 PROTOCOLID_TCP_ULP, &init_data);
 	if (rc)
 		return rc;
 
@@ -606,13 +606,13 @@ static int qed_sp_iscsi_func_stop(struct qed_hwfn *p_hwfn,
 
 	rc = qed_sp_init_request(p_hwfn, &p_ent,
 				 ISCSI_RAMROD_CMD_ID_DESTROY_FUNC,
-				 PROTOCOLID_ISCSI, &init_data);
+				 PROTOCOLID_TCP_ULP, &init_data);
 	if (rc)
 		return rc;
 
 	rc = qed_spq_post(p_hwfn, p_ent, NULL);
 
-	qed_spq_unregister_async_cb(p_hwfn, PROTOCOLID_ISCSI);
+	qed_spq_unregister_async_cb(p_hwfn, PROTOCOLID_TCP_ULP);
 	return rc;
 }
 
@@ -786,7 +786,7 @@ static int qed_iscsi_acquire_connection(struct qed_hwfn *p_hwfn,
 	u32 icid;
 
 	spin_lock_bh(&p_hwfn->p_iscsi_info->lock);
-	rc = qed_cxt_acquire_cid(p_hwfn, PROTOCOLID_ISCSI, &icid);
+	rc = qed_cxt_acquire_cid(p_hwfn, PROTOCOLID_TCP_ULP, &icid);
 	spin_unlock_bh(&p_hwfn->p_iscsi_info->lock);
 	if (rc)
 		return rc;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_ll2.c b/drivers/net/ethernet/qlogic/qed/qed_ll2.c
index 49783f365079..286e53927866 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_ll2.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_ll2.c
@@ -1037,8 +1037,8 @@ static int qed_sp_ll2_tx_queue_start(struct qed_hwfn *p_hwfn,
 	case QED_LL2_TYPE_FCOE:
 		p_ramrod->conn_type = PROTOCOLID_FCOE;
 		break;
-	case QED_LL2_TYPE_ISCSI:
-		p_ramrod->conn_type = PROTOCOLID_ISCSI;
+	case QED_LL2_TYPE_TCP_ULP:
+		p_ramrod->conn_type = PROTOCOLID_TCP_ULP;
 		break;
 	case QED_LL2_TYPE_ROCE:
 		p_ramrod->conn_type = PROTOCOLID_ROCE;
@@ -1048,7 +1048,7 @@ static int qed_sp_ll2_tx_queue_start(struct qed_hwfn *p_hwfn,
 		break;
 	case QED_LL2_TYPE_OOO:
 		if (p_hwfn->hw_info.personality == QED_PCI_ISCSI)
-			p_ramrod->conn_type = PROTOCOLID_ISCSI;
+			p_ramrod->conn_type = PROTOCOLID_TCP_ULP;
 		else
 			p_ramrod->conn_type = PROTOCOLID_IWARP;
 		break;
@@ -2442,7 +2442,7 @@ static int __qed_ll2_start(struct qed_hwfn *p_hwfn,
 		conn_type = QED_LL2_TYPE_FCOE;
 		break;
 	case QED_PCI_ISCSI:
-		conn_type = QED_LL2_TYPE_ISCSI;
+		conn_type = QED_LL2_TYPE_TCP_ULP;
 		break;
 	case QED_PCI_ETH_ROCE:
 		conn_type = QED_LL2_TYPE_ROCE;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_ooo.c b/drivers/net/ethernet/qlogic/qed/qed_ooo.c
index 88353aa404dc..599da0d7366b 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_ooo.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_ooo.c
@@ -83,7 +83,7 @@ int qed_ooo_alloc(struct qed_hwfn *p_hwfn)
 
 	switch (p_hwfn->hw_info.personality) {
 	case QED_PCI_ISCSI:
-		proto = PROTOCOLID_ISCSI;
+		proto = PROTOCOLID_TCP_ULP;
 		break;
 	case QED_PCI_ETH_RDMA:
 	case QED_PCI_ETH_IWARP:
diff --git a/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c b/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
index aa71adcf31ee..ee7dc0a7da6c 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
@@ -385,7 +385,7 @@ int qed_sp_pf_start(struct qed_hwfn *p_hwfn,
 		p_ramrod->personality = PERSONALITY_FCOE;
 		break;
 	case QED_PCI_ISCSI:
-		p_ramrod->personality = PERSONALITY_ISCSI;
+		p_ramrod->personality = PERSONALITY_TCP_ULP;
 		break;
 	case QED_PCI_ETH_ROCE:
 	case QED_PCI_ETH_IWARP:
diff --git a/include/linux/qed/common_hsi.h b/include/linux/qed/common_hsi.h
index 977807e1be53..0a3807e927c5 100644
--- a/include/linux/qed/common_hsi.h
+++ b/include/linux/qed/common_hsi.h
@@ -702,7 +702,7 @@ enum mf_mode {
 
 /* Per-protocol connection types */
 enum protocol_type {
-	PROTOCOLID_ISCSI,
+	PROTOCOLID_TCP_ULP,
 	PROTOCOLID_FCOE,
 	PROTOCOLID_ROCE,
 	PROTOCOLID_CORE,
diff --git a/include/linux/qed/qed_ll2_if.h b/include/linux/qed/qed_ll2_if.h
index ea273ba1c991..ff808d248883 100644
--- a/include/linux/qed/qed_ll2_if.h
+++ b/include/linux/qed/qed_ll2_if.h
@@ -18,7 +18,7 @@
 
 enum qed_ll2_conn_type {
 	QED_LL2_TYPE_FCOE,
-	QED_LL2_TYPE_ISCSI,
+	QED_LL2_TYPE_TCP_ULP,
 	QED_LL2_TYPE_TEST,
 	QED_LL2_TYPE_OOO,
 	QED_LL2_TYPE_RESERVED2,
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 10:16:52 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 20:16:52 +0300
Subject: [PATCH 4/7] qed: Add support of HW filter block
In-Reply-To: <20210602171655.23581-1-smalin@marvell.com>
References: <20210602171655.23581-1-smalin@marvell.com>
Message-ID: <20210602171655.23581-5-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

This patch introduces the functionality of HW filter block.
It adds and removes filters based on source and target TCP port.

It also add functionality to clear all filters at once.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/net/ethernet/qlogic/qed/qed.h         |  8 ++
 drivers/net/ethernet/qlogic/qed/qed_dev.c     | 90 +++++++++++++++++++
 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c |  5 ++
 include/linux/qed/qed_nvmetcp_if.h            | 24 +++++
 4 files changed, 127 insertions(+)

diff --git a/drivers/net/ethernet/qlogic/qed/qed.h b/drivers/net/ethernet/qlogic/qed/qed.h
index bc9bdb9d1bb9..b590c70539b5 100644
--- a/drivers/net/ethernet/qlogic/qed/qed.h
+++ b/drivers/net/ethernet/qlogic/qed/qed.h
@@ -49,6 +49,8 @@ extern const struct qed_common_ops qed_common_ops_pass;
 #define QED_MIN_WIDS		(4)
 #define QED_PF_DEMS_SIZE        (4)
 
+#define QED_LLH_DONT_CARE 0
+
 /* cau states */
 enum qed_coalescing_mode {
 	QED_COAL_MODE_DISABLE,
@@ -1005,4 +1007,10 @@ int qed_mfw_fill_tlv_data(struct qed_hwfn *hwfn,
 void qed_hw_info_set_offload_tc(struct qed_hw_info *p_info, u8 tc);
 
 void qed_periodic_db_rec_start(struct qed_hwfn *p_hwfn);
+
+int qed_llh_add_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port);
+int qed_llh_add_dst_tcp_port_filter(struct qed_dev *cdev, u16 dest_port);
+void qed_llh_remove_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port);
+void qed_llh_remove_dst_tcp_port_filter(struct qed_dev *cdev, u16 src_port);
+void qed_llh_clear_all_filters(struct qed_dev *cdev);
 #endif /* _QED_H */
diff --git a/drivers/net/ethernet/qlogic/qed/qed_dev.c b/drivers/net/ethernet/qlogic/qed/qed_dev.c
index 932b892f1ef1..0410c3604abd 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_dev.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_dev.c
@@ -5362,3 +5362,93 @@ void qed_set_fw_mac_addr(__le16 *fw_msb,
 	((u8 *)fw_lsb)[0] = mac[5];
 	((u8 *)fw_lsb)[1] = mac[4];
 }
+
+static int qed_llh_shadow_remove_all_filters(struct qed_dev *cdev, u8 ppfid)
+{
+	struct qed_llh_info *p_llh_info = cdev->p_llh_info;
+	struct qed_llh_filter_info *p_filters;
+	int rc;
+
+	rc = qed_llh_shadow_sanity(cdev, ppfid, 0, "remove_all");
+	if (rc)
+		return rc;
+
+	p_filters = p_llh_info->pp_filters[ppfid];
+	memset(p_filters, 0, NIG_REG_LLH_FUNC_FILTER_EN_SIZE *
+	       sizeof(*p_filters));
+
+	return 0;
+}
+
+static void qed_llh_clear_ppfid_filters(struct qed_dev *cdev, u8 ppfid)
+{
+	struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev);
+	struct qed_ptt *p_ptt = qed_ptt_acquire(p_hwfn);
+	u8 filter_idx, abs_ppfid;
+	int rc = 0;
+
+	if (!p_ptt)
+		return;
+
+	if (!test_bit(QED_MF_LLH_PROTO_CLSS, &cdev->mf_bits) &&
+	    !test_bit(QED_MF_LLH_MAC_CLSS, &cdev->mf_bits))
+		goto out;
+
+	rc = qed_llh_abs_ppfid(cdev, ppfid, &abs_ppfid);
+	if (rc)
+		goto out;
+
+	rc = qed_llh_shadow_remove_all_filters(cdev, ppfid);
+	if (rc)
+		goto out;
+
+	for (filter_idx = 0; filter_idx < NIG_REG_LLH_FUNC_FILTER_EN_SIZE;
+	     filter_idx++) {
+		rc = qed_llh_remove_filter(p_hwfn, p_ptt,
+					   abs_ppfid, filter_idx);
+		if (rc)
+			goto out;
+	}
+out:
+	qed_ptt_release(p_hwfn, p_ptt);
+}
+
+int qed_llh_add_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port)
+{
+	return qed_llh_add_protocol_filter(cdev, 0,
+					   QED_LLH_FILTER_TCP_SRC_PORT,
+					   src_port, QED_LLH_DONT_CARE);
+}
+
+void qed_llh_remove_src_tcp_port_filter(struct qed_dev *cdev, u16 src_port)
+{
+	qed_llh_remove_protocol_filter(cdev, 0,
+				       QED_LLH_FILTER_TCP_SRC_PORT,
+				       src_port, QED_LLH_DONT_CARE);
+}
+
+int qed_llh_add_dst_tcp_port_filter(struct qed_dev *cdev, u16 dest_port)
+{
+	return qed_llh_add_protocol_filter(cdev, 0,
+					   QED_LLH_FILTER_TCP_DEST_PORT,
+					   QED_LLH_DONT_CARE, dest_port);
+}
+
+void qed_llh_remove_dst_tcp_port_filter(struct qed_dev *cdev, u16 dest_port)
+{
+	qed_llh_remove_protocol_filter(cdev, 0,
+				       QED_LLH_FILTER_TCP_DEST_PORT,
+				       QED_LLH_DONT_CARE, dest_port);
+}
+
+void qed_llh_clear_all_filters(struct qed_dev *cdev)
+{
+	u8 ppfid;
+
+	if (!test_bit(QED_MF_LLH_PROTO_CLSS, &cdev->mf_bits) &&
+	    !test_bit(QED_MF_LLH_MAC_CLSS, &cdev->mf_bits))
+		return;
+
+	for (ppfid = 0; ppfid < cdev->p_llh_info->num_ppfid; ppfid++)
+		qed_llh_clear_ppfid_filters(cdev, ppfid);
+}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
index 7943804e88cd..d4d609a4d3a3 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
@@ -805,6 +805,11 @@ static const struct qed_nvmetcp_ops qed_nvmetcp_ops_pass = {
 	.update_conn = &qed_nvmetcp_update_conn,
 	.destroy_conn = &qed_nvmetcp_destroy_conn,
 	.clear_sq = &qed_nvmetcp_clear_conn_sq,
+	.add_src_tcp_port_filter = &qed_llh_add_src_tcp_port_filter,
+	.remove_src_tcp_port_filter = &qed_llh_remove_src_tcp_port_filter,
+	.add_dst_tcp_port_filter = &qed_llh_add_dst_tcp_port_filter,
+	.remove_dst_tcp_port_filter = &qed_llh_remove_dst_tcp_port_filter,
+	.clear_all_filters = &qed_llh_clear_all_filters
 };
 
 const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void)
diff --git a/include/linux/qed/qed_nvmetcp_if.h b/include/linux/qed/qed_nvmetcp_if.h
index 5baf1c5ce798..5180edad24e5 100644
--- a/include/linux/qed/qed_nvmetcp_if.h
+++ b/include/linux/qed/qed_nvmetcp_if.h
@@ -123,6 +123,20 @@ struct qed_nvmetcp_cb_ops {
  *			@param cdev
  *			@param handle - the connection handle.
  *			@return 0 on success, otherwise error value.
+ * @add_src_tcp_port_filter: Add source tcp port filter
+ *			@param cdev
+ *			@param src_port
+ * @remove_src_tcp_port_filter: Remove source tcp port filter
+ *			@param cdev
+ *			@param src_port
+ * @add_dst_tcp_port_filter: Add destination tcp port filter
+ *			@param cdev
+ *			@param dest_port
+ * @remove_dst_tcp_port_filter: Remove destination tcp port filter
+ *			@param cdev
+ *			@param dest_port
+ * @clear_all_filters: Clear all filters.
+ *			@param cdev
  */
 struct qed_nvmetcp_ops {
 	const struct qed_common_ops *common;
@@ -158,6 +172,16 @@ struct qed_nvmetcp_ops {
 	int (*destroy_conn)(struct qed_dev *cdev, u32 handle, u8 abrt_conn);
 
 	int (*clear_sq)(struct qed_dev *cdev, u32 handle);
+
+	int (*add_src_tcp_port_filter)(struct qed_dev *cdev, u16 src_port);
+
+	void (*remove_src_tcp_port_filter)(struct qed_dev *cdev, u16 src_port);
+
+	int (*add_dst_tcp_port_filter)(struct qed_dev *cdev, u16 dest_port);
+
+	void (*remove_dst_tcp_port_filter)(struct qed_dev *cdev, u16 dest_port);
+
+	void (*clear_all_filters)(struct qed_dev *cdev);
 };
 
 const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void);
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 10:16:53 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 20:16:53 +0300
Subject: [PATCH 5/7] qed: Add NVMeTCP Offload IO Level FW and HW HSI
In-Reply-To: <20210602171655.23581-1-smalin@marvell.com>
References: <20210602171655.23581-1-smalin@marvell.com>
Message-ID: <20210602171655.23581-6-smalin@marvell.com>

This patch introduces the NVMeTCP Offload FW and HW  HSI in order
to initialize the IO level configuration into a per IO HW
resource ("task") as part of the IO path flow.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 include/linux/qed/nvmetcp_common.h | 335 ++++++++++++++++++++++++++++-
 include/linux/qed/qed_nvmetcp_if.h |  31 +++
 2 files changed, 365 insertions(+), 1 deletion(-)

diff --git a/include/linux/qed/nvmetcp_common.h b/include/linux/qed/nvmetcp_common.h
index c8836b71b866..ad745a9c2264 100644
--- a/include/linux/qed/nvmetcp_common.h
+++ b/include/linux/qed/nvmetcp_common.h
@@ -7,6 +7,7 @@
 #include "tcp_common.h"
 
 #define NVMETCP_SLOW_PATH_LAYER_CODE (6)
+#define NVMETCP_WQE_NUM_SGES_SLOWIO (0xf)
 
 /* NVMeTCP firmware function init parameters */
 struct nvmetcp_spe_func_init {
@@ -194,4 +195,336 @@ struct nvmetcp_wqe {
 #define NVMETCP_WQE_CDB_SIZE_OR_NVMETCP_CMD_SHIFT 24
 };
 
-#endif /* __NVMETCP_COMMON__ */
+struct nvmetcp_host_cccid_itid_entry {
+	__le16 itid;
+};
+
+struct nvmetcp_connect_done_results {
+	__le16 icid;
+	__le16 conn_id;
+	struct tcp_ulp_connect_done_params params;
+};
+
+struct nvmetcp_eqe_data {
+	__le16 icid;
+	__le16 conn_id;
+	__le16 reserved;
+	u8 error_code;
+	u8 error_pdu_opcode_reserved;
+#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_MASK 0x3F
+#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_SHIFT  0
+#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_VALID_MASK  0x1
+#define NVMETCP_EQE_DATA_ERROR_PDU_OPCODE_VALID_SHIFT  6
+#define NVMETCP_EQE_DATA_RESERVED0_MASK 0x1
+#define NVMETCP_EQE_DATA_RESERVED0_SHIFT 7
+};
+
+enum nvmetcp_task_type {
+	NVMETCP_TASK_TYPE_HOST_WRITE,
+	NVMETCP_TASK_TYPE_HOST_READ,
+	NVMETCP_TASK_TYPE_INIT_CONN_REQUEST,
+	NVMETCP_TASK_TYPE_RESERVED0,
+	NVMETCP_TASK_TYPE_CLEANUP,
+	NVMETCP_TASK_TYPE_HOST_READ_NO_CQE,
+	MAX_NVMETCP_TASK_TYPE
+};
+
+struct nvmetcp_db_data {
+	u8 params;
+#define NVMETCP_DB_DATA_DEST_MASK 0x3 /* destination of doorbell (use enum db_dest) */
+#define NVMETCP_DB_DATA_DEST_SHIFT 0
+#define NVMETCP_DB_DATA_AGG_CMD_MASK 0x3 /* aggregative command to CM (use enum db_agg_cmd_sel) */
+#define NVMETCP_DB_DATA_AGG_CMD_SHIFT 2
+#define NVMETCP_DB_DATA_BYPASS_EN_MASK 0x1 /* enable QM bypass */
+#define NVMETCP_DB_DATA_BYPASS_EN_SHIFT 4
+#define NVMETCP_DB_DATA_RESERVED_MASK 0x1
+#define NVMETCP_DB_DATA_RESERVED_SHIFT 5
+#define NVMETCP_DB_DATA_AGG_VAL_SEL_MASK 0x3 /* aggregative value selection */
+#define NVMETCP_DB_DATA_AGG_VAL_SEL_SHIFT 6
+	u8 agg_flags; /* bit for every DQ counter flags in CM context that DQ can increment */
+	__le16 sq_prod;
+};
+
+struct nvmetcp_fw_nvmf_cqe {
+	__le32 reserved[4];
+};
+
+struct nvmetcp_icresp_mdata {
+	u8  digest;
+	u8  cpda;
+	__le16  pfv;
+	__le32 maxdata;
+	__le16 rsvd[4];
+};
+
+union nvmetcp_fw_cqe_data {
+	struct nvmetcp_fw_nvmf_cqe nvme_cqe;
+	struct nvmetcp_icresp_mdata icresp_mdata;
+};
+
+struct nvmetcp_fw_cqe {
+	__le16 conn_id;
+	u8 cqe_type;
+	u8 cqe_error_status_bits;
+#define CQE_ERROR_BITMAP_DIF_ERR_BITS_MASK 0x7
+#define CQE_ERROR_BITMAP_DIF_ERR_BITS_SHIFT 0
+#define CQE_ERROR_BITMAP_DATA_DIGEST_ERR_MASK 0x1
+#define CQE_ERROR_BITMAP_DATA_DIGEST_ERR_SHIFT 3
+#define CQE_ERROR_BITMAP_RCV_ON_INVALID_CONN_MASK 0x1
+#define CQE_ERROR_BITMAP_RCV_ON_INVALID_CONN_SHIFT 4
+	__le16 itid;
+	u8 task_type;
+	u8 fw_dbg_field;
+	u8 caused_conn_err;
+	u8 reserved0[3];
+	__le32 reserved1;
+	union nvmetcp_fw_cqe_data cqe_data;
+	struct regpair task_opaque;
+	__le32 reserved[6];
+};
+
+enum nvmetcp_fw_cqes_type {
+	NVMETCP_FW_CQE_TYPE_NORMAL = 1,
+	NVMETCP_FW_CQE_TYPE_RESERVED0,
+	NVMETCP_FW_CQE_TYPE_RESERVED1,
+	NVMETCP_FW_CQE_TYPE_CLEANUP,
+	NVMETCP_FW_CQE_TYPE_DUMMY,
+	MAX_NVMETCP_FW_CQES_TYPE
+};
+
+struct ystorm_nvmetcp_task_state {
+	struct scsi_cached_sges data_desc;
+	struct scsi_sgl_params sgl_params;
+	__le32 resrved0;
+	__le32 buffer_offset;
+	__le16 cccid;
+	struct nvmetcp_dif_flags dif_flags;
+	u8 flags;
+#define YSTORM_NVMETCP_TASK_STATE_LOCAL_COMP_MASK 0x1
+#define YSTORM_NVMETCP_TASK_STATE_LOCAL_COMP_SHIFT 0
+#define YSTORM_NVMETCP_TASK_STATE_SLOW_IO_MASK 0x1
+#define YSTORM_NVMETCP_TASK_STATE_SLOW_IO_SHIFT 1
+#define YSTORM_NVMETCP_TASK_STATE_SET_DIF_OFFSET_MASK 0x1
+#define YSTORM_NVMETCP_TASK_STATE_SET_DIF_OFFSET_SHIFT 2
+#define YSTORM_NVMETCP_TASK_STATE_SEND_W_RSP_MASK 0x1
+#define YSTORM_NVMETCP_TASK_STATE_SEND_W_RSP_SHIFT 3
+};
+
+struct ystorm_nvmetcp_task_rxmit_opt {
+	__le32 reserved[4];
+};
+
+struct nvmetcp_task_hdr {
+	__le32 reg[18];
+};
+
+struct nvmetcp_task_hdr_aligned {
+	struct nvmetcp_task_hdr task_hdr;
+	__le32 reserved[2];	/* HSI_COMMENT: Align to QREG */
+};
+
+struct e5_tdif_task_context {
+	__le32 reserved[16];
+};
+
+struct e5_rdif_task_context {
+	__le32 reserved[12];
+};
+
+struct ystorm_nvmetcp_task_st_ctx {
+	struct ystorm_nvmetcp_task_state state;
+	struct ystorm_nvmetcp_task_rxmit_opt rxmit_opt;
+	struct nvmetcp_task_hdr_aligned pdu_hdr;
+};
+
+struct mstorm_nvmetcp_task_st_ctx {
+	struct scsi_cached_sges data_desc;
+	struct scsi_sgl_params sgl_params;
+	__le32 rem_task_size;
+	__le32 data_buffer_offset;
+	u8 task_type;
+	struct nvmetcp_dif_flags dif_flags;
+	__le16 dif_task_icid;
+	struct regpair reserved0;
+	__le32 expected_itt;
+	__le32 reserved1;
+};
+
+struct ustorm_nvmetcp_task_st_ctx {
+	__le32 rem_rcv_len;
+	__le32 exp_data_transfer_len;
+	__le32 exp_data_sn;
+	struct regpair reserved0;
+	__le32 reg1_map;
+#define REG1_NUM_SGES_MASK 0xF
+#define REG1_NUM_SGES_SHIFT 0
+#define REG1_RESERVED1_MASK 0xFFFFFFF
+#define REG1_RESERVED1_SHIFT 4
+	u8 flags2;
+#define USTORM_NVMETCP_TASK_ST_CTX_AHS_EXIST_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_AHS_EXIST_SHIFT 0
+#define USTORM_NVMETCP_TASK_ST_CTX_RESERVED1_MASK 0x7F
+#define USTORM_NVMETCP_TASK_ST_CTX_RESERVED1_SHIFT 1
+	struct nvmetcp_dif_flags dif_flags;
+	__le16 reserved3;
+	__le16 tqe_opaque[2];
+	__le32 reserved5;
+	__le32 nvme_tcp_opaque_lo;
+	__le32 nvme_tcp_opaque_hi;
+	u8 task_type;
+	u8 error_flags;
+#define USTORM_NVMETCP_TASK_ST_CTX_DATA_DIGEST_ERROR_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_DATA_DIGEST_ERROR_SHIFT 0
+#define USTORM_NVMETCP_TASK_ST_CTX_DATA_TRUNCATED_ERROR_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_DATA_TRUNCATED_ERROR_SHIFT 1
+#define USTORM_NVMETCP_TASK_ST_CTX_UNDER_RUN_ERROR_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_UNDER_RUN_ERROR_SHIFT 2
+#define USTORM_NVMETCP_TASK_ST_CTX_NVME_TCP_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_NVME_TCP_SHIFT 3
+	u8 flags;
+#define USTORM_NVMETCP_TASK_ST_CTX_CQE_WRITE_MASK 0x3
+#define USTORM_NVMETCP_TASK_ST_CTX_CQE_WRITE_SHIFT 0
+#define USTORM_NVMETCP_TASK_ST_CTX_LOCAL_COMP_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_LOCAL_COMP_SHIFT 2
+#define USTORM_NVMETCP_TASK_ST_CTX_Q0_R2TQE_WRITE_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_Q0_R2TQE_WRITE_SHIFT 3
+#define USTORM_NVMETCP_TASK_ST_CTX_TOTAL_DATA_ACKED_DONE_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_TOTAL_DATA_ACKED_DONE_SHIFT 4
+#define USTORM_NVMETCP_TASK_ST_CTX_HQ_SCANNED_DONE_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_HQ_SCANNED_DONE_SHIFT 5
+#define USTORM_NVMETCP_TASK_ST_CTX_R2T2RECV_DONE_MASK 0x1
+#define USTORM_NVMETCP_TASK_ST_CTX_R2T2RECV_DONE_SHIFT 6
+	u8 cq_rss_number;
+};
+
+struct e5_ystorm_nvmetcp_task_ag_ctx {
+	u8 reserved /* cdu_validation */;
+	u8 byte1 /* state_and_core_id */;
+	__le16 word0 /* icid */;
+	u8 flags0;
+	u8 flags1;
+	u8 flags2;
+	u8 flags3;
+	__le32 TTT;
+	u8 byte2;
+	u8 byte3;
+	u8 byte4;
+	u8 e4_reserved7;
+};
+
+struct e5_mstorm_nvmetcp_task_ag_ctx {
+	u8 cdu_validation;
+	u8 byte1;
+	__le16 task_cid;
+	u8 flags0;
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_MASK 0xF
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_SHIFT 0
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_SHIFT 4
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_SHIFT 5
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_VALID_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_VALID_SHIFT 6
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_FLAG_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_FLAG_SHIFT 7
+	u8 flags1;
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_MASK 0x3
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_SHIFT 0
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1_MASK 0x3
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1_SHIFT 2
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF2_MASK 0x3
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF2_SHIFT 4
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_EN_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_TASK_CLEANUP_CF_EN_SHIFT 6
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1EN_MASK 0x1
+#define E5_MSTORM_NVMETCP_TASK_AG_CTX_CF1EN_SHIFT 7
+	u8 flags2;
+	u8 flags3;
+	__le32 reg0;
+	u8 byte2;
+	u8 byte3;
+	u8 byte4;
+	u8 e4_reserved7;
+};
+
+struct e5_ustorm_nvmetcp_task_ag_ctx {
+	u8 reserved;
+	u8 state_and_core_id;
+	__le16 icid;
+	u8 flags0;
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_MASK 0xF
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONNECTION_TYPE_SHIFT 0
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_EXIST_IN_QM0_SHIFT 4
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CONN_CLEAR_SQ_FLAG_SHIFT 5
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_SHIFT 6
+	u8 flags1;
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_RESERVED1_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_RESERVED1_SHIFT 0
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_SHIFT 2
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3_SHIFT 4
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_SHIFT 6
+	u8 flags2;
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_HQ_SCANNED_CF_EN_SHIFT 0
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DISABLE_DATA_ACKED_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DISABLE_DATA_ACKED_SHIFT 1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV_EN_SHIFT 2
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CF3EN_SHIFT 3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_EN_SHIFT 4
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_DATA_TOTAL_EXP_EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_DATA_TOTAL_EXP_EN_SHIFT 5
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_RULE1EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_RULE1EN_SHIFT 6
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_CONT_RCV_EXP_EN_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_CMP_CONT_RCV_EXP_EN_SHIFT 7
+	u8 flags3;
+	u8 flags4;
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED5_MASK 0x3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED5_SHIFT 0
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED6_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED6_SHIFT 2
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED7_MASK 0x1
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_E4_RESERVED7_SHIFT 3
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_TYPE_MASK 0xF
+#define E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_TYPE_SHIFT 4
+	u8 byte2;
+	u8 byte3;
+	u8 e4_reserved8;
+	__le32 dif_err_intervals;
+	__le32 dif_error_1st_interval;
+	__le32 rcv_cont_len;
+	__le32 exp_cont_len;
+	__le32 total_data_acked;
+	__le32 exp_data_acked;
+	__le16 word1;
+	__le16 next_tid;
+	__le32 hdr_residual_count;
+	__le32 exp_r2t_sn;
+};
+
+struct e5_nvmetcp_task_context {
+	struct ystorm_nvmetcp_task_st_ctx ystorm_st_context;
+	struct e5_ystorm_nvmetcp_task_ag_ctx ystorm_ag_context;
+	struct regpair ystorm_ag_padding[2];
+	struct e5_tdif_task_context tdif_context;
+	struct e5_mstorm_nvmetcp_task_ag_ctx mstorm_ag_context;
+	struct regpair mstorm_ag_padding[2];
+	struct e5_ustorm_nvmetcp_task_ag_ctx ustorm_ag_context;
+	struct regpair ustorm_ag_padding[2];
+	struct mstorm_nvmetcp_task_st_ctx mstorm_st_context;
+	struct regpair mstorm_st_padding[2];
+	struct ustorm_nvmetcp_task_st_ctx ustorm_st_context;
+	struct regpair ustorm_st_padding[2];
+	struct e5_rdif_task_context rdif_context;
+};
+
+#endif /* __NVMETCP_COMMON__*/
diff --git a/include/linux/qed/qed_nvmetcp_if.h b/include/linux/qed/qed_nvmetcp_if.h
index 5180edad24e5..606427ebb63c 100644
--- a/include/linux/qed/qed_nvmetcp_if.h
+++ b/include/linux/qed/qed_nvmetcp_if.h
@@ -5,6 +5,8 @@
 #define _QED_NVMETCP_IF_H
 #include <linux/types.h>
 #include <linux/qed/qed_if.h>
+#include <linux/qed/storage_common.h>
+#include <linux/qed/nvmetcp_common.h>
 
 #define QED_NVMETCP_MAX_IO_SIZE	0x800000
 
@@ -72,6 +74,35 @@ struct qed_nvmetcp_cb_ops {
 	struct qed_common_cb_ops common;
 };
 
+struct nvmetcp_sge {
+	struct regpair sge_addr; /* SGE address */
+	__le32 sge_len; /* SGE length */
+	__le32 reserved;
+};
+
+/* IO path HSI function SGL params */
+struct storage_sgl_task_params {
+	struct nvmetcp_sge *sgl;
+	struct regpair sgl_phys_addr;
+	u32 total_buffer_size;
+	u16 num_sges;
+	bool small_mid_sge;
+};
+
+/* IO path HSI function FW task context params */
+struct nvmetcp_task_params {
+	void *context; /* Output parameter - set/filled by the HSI function */
+	struct nvmetcp_wqe *sqe;
+	u32 tx_io_size; /* in bytes (Without DIF, if exists) */
+	u32 rx_io_size; /* in bytes (Without DIF, if exists) */
+	u16 conn_icid;
+	u16 itid;
+	struct regpair opq; /* qedn_task_ctx address */
+	u16 host_cccid;
+	u8 cq_rss_number;
+	bool send_write_incapsule;
+};
+
 /**
  * struct qed_nvmetcp_ops - qed NVMeTCP operations.
  * @common:		common operations pointer
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 10:16:54 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 20:16:54 +0300
Subject: [PATCH 6/7] qed: Add NVMeTCP Offload IO Level FW Initializations
In-Reply-To: <20210602171655.23581-1-smalin@marvell.com>
References: <20210602171655.23581-1-smalin@marvell.com>
Message-ID: <20210602171655.23581-7-smalin@marvell.com>

This patch introduces the NVMeTCP FW initializations which is used
to initialize the IO level configuration into a per IO HW
resource ("task") as part of the IO path flow.

This includes:
- Write IO FW initialization
- Read IO FW initialization.
- IC-Req and IC-Resp FW exchange.
- FW Cleanup flow (Flush IO).

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/net/ethernet/qlogic/qed/Makefile      |   5 +-
 drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c |   7 +-
 .../qlogic/qed/qed_nvmetcp_fw_funcs.c         | 376 ++++++++++++++++++
 .../qlogic/qed/qed_nvmetcp_fw_funcs.h         |  40 ++
 include/linux/qed/nvmetcp_common.h            |   1 +
 include/linux/qed/qed_nvmetcp_if.h            |  20 +
 6 files changed, 447 insertions(+), 2 deletions(-)
 create mode 100644 drivers/net/ethernet/qlogic/qed/qed_nvmetcp_fw_funcs.c
 create mode 100644 drivers/net/ethernet/qlogic/qed/qed_nvmetcp_fw_funcs.h

diff --git a/drivers/net/ethernet/qlogic/qed/Makefile b/drivers/net/ethernet/qlogic/qed/Makefile
index 7cb0db67ba5b..0d9c2fe0245d 100644
--- a/drivers/net/ethernet/qlogic/qed/Makefile
+++ b/drivers/net/ethernet/qlogic/qed/Makefile
@@ -28,7 +28,10 @@ qed-$(CONFIG_QED_ISCSI) += qed_iscsi.o
 qed-$(CONFIG_QED_LL2) += qed_ll2.o
 qed-$(CONFIG_QED_OOO) += qed_ooo.o
 
-qed-$(CONFIG_QED_NVMETCP) += qed_nvmetcp.o
+qed-$(CONFIG_QED_NVMETCP) +=	\
+	qed_nvmetcp.o		\
+	qed_nvmetcp_fw_funcs.o	\
+	qed_nvmetcp_ip_services.o
 
 qed-$(CONFIG_QED_RDMA) +=	\
 	qed_iwarp.o		\
diff --git a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
index d4d609a4d3a3..f19128c8d9cc 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp.c
@@ -27,6 +27,7 @@
 #include "qed_mcp.h"
 #include "qed_sp.h"
 #include "qed_reg_addr.h"
+#include "qed_nvmetcp_fw_funcs.h"
 
 static int qed_nvmetcp_async_event(struct qed_hwfn *p_hwfn, u8 fw_event_code,
 				   u16 echo, union event_ring_data *data,
@@ -809,7 +810,11 @@ static const struct qed_nvmetcp_ops qed_nvmetcp_ops_pass = {
 	.remove_src_tcp_port_filter = &qed_llh_remove_src_tcp_port_filter,
 	.add_dst_tcp_port_filter = &qed_llh_add_dst_tcp_port_filter,
 	.remove_dst_tcp_port_filter = &qed_llh_remove_dst_tcp_port_filter,
-	.clear_all_filters = &qed_llh_clear_all_filters
+	.clear_all_filters = &qed_llh_clear_all_filters,
+	.init_read_io = &init_nvmetcp_host_read_task,
+	.init_write_io = &init_nvmetcp_host_write_task,
+	.init_icreq_exchange = &init_nvmetcp_init_conn_req_task,
+	.init_task_cleanup = &init_cleanup_task_nvmetcp
 };
 
 const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void)
diff --git a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp_fw_funcs.c b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp_fw_funcs.c
new file mode 100644
index 000000000000..c1dd71d19f3f
--- /dev/null
+++ b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp_fw_funcs.c
@@ -0,0 +1,376 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause)
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/mm.h>
+#include <linux/types.h>
+#include <asm/byteorder.h>
+#include <linux/qed/common_hsi.h>
+#include <linux/qed/storage_common.h>
+#include <linux/qed/nvmetcp_common.h>
+#include <linux/qed/qed_nvmetcp_if.h>
+#include "qed_nvmetcp_fw_funcs.h"
+
+#define NVMETCP_NUM_SGES_IN_CACHE 0x4
+
+bool nvmetcp_is_slow_sgl(u16 num_sges, bool small_mid_sge)
+{
+	return (num_sges > SCSI_NUM_SGES_SLOW_SGL_THR && small_mid_sge);
+}
+
+void init_scsi_sgl_context(struct scsi_sgl_params *ctx_sgl_params,
+			   struct scsi_cached_sges *ctx_data_desc,
+			   struct storage_sgl_task_params *sgl_params)
+{
+	u8 num_sges_to_init = (u8)(sgl_params->num_sges > NVMETCP_NUM_SGES_IN_CACHE ?
+				   NVMETCP_NUM_SGES_IN_CACHE : sgl_params->num_sges);
+	u8 sge_index;
+
+	/* sgl params */
+	ctx_sgl_params->sgl_addr.lo = cpu_to_le32(sgl_params->sgl_phys_addr.lo);
+	ctx_sgl_params->sgl_addr.hi = cpu_to_le32(sgl_params->sgl_phys_addr.hi);
+	ctx_sgl_params->sgl_total_length = cpu_to_le32(sgl_params->total_buffer_size);
+	ctx_sgl_params->sgl_num_sges = cpu_to_le16(sgl_params->num_sges);
+
+	for (sge_index = 0; sge_index < num_sges_to_init; sge_index++) {
+		ctx_data_desc->sge[sge_index].sge_addr.lo =
+			cpu_to_le32(sgl_params->sgl[sge_index].sge_addr.lo);
+		ctx_data_desc->sge[sge_index].sge_addr.hi =
+			cpu_to_le32(sgl_params->sgl[sge_index].sge_addr.hi);
+		ctx_data_desc->sge[sge_index].sge_len =
+			cpu_to_le32(sgl_params->sgl[sge_index].sge_len);
+	}
+}
+
+static inline u32 calc_rw_task_size(struct nvmetcp_task_params *task_params,
+				    enum nvmetcp_task_type task_type)
+{
+	u32 io_size;
+
+	if (task_type == NVMETCP_TASK_TYPE_HOST_WRITE)
+		io_size = task_params->tx_io_size;
+	else
+		io_size = task_params->rx_io_size;
+
+	if (unlikely(!io_size))
+		return 0;
+
+	return io_size;
+}
+
+static inline void init_sqe(struct nvmetcp_task_params *task_params,
+			    struct storage_sgl_task_params *sgl_task_params,
+			    enum nvmetcp_task_type task_type)
+{
+	if (!task_params->sqe)
+		return;
+
+	memset(task_params->sqe, 0, sizeof(*task_params->sqe));
+	task_params->sqe->task_id = cpu_to_le16(task_params->itid);
+
+	switch (task_type) {
+	case NVMETCP_TASK_TYPE_HOST_WRITE: {
+		u32 buf_size = 0;
+		u32 num_sges = 0;
+
+		SET_FIELD(task_params->sqe->contlen_cdbsize,
+			  NVMETCP_WQE_CDB_SIZE_OR_NVMETCP_CMD, 1);
+		SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_WQE_TYPE,
+			  NVMETCP_WQE_TYPE_NORMAL);
+		if (task_params->tx_io_size) {
+			if (task_params->send_write_incapsule)
+				buf_size = calc_rw_task_size(task_params, task_type);
+
+			if (nvmetcp_is_slow_sgl(sgl_task_params->num_sges,
+						sgl_task_params->small_mid_sge))
+				num_sges = NVMETCP_WQE_NUM_SGES_SLOWIO;
+			else
+				num_sges = min((u16)sgl_task_params->num_sges,
+					       (u16)SCSI_NUM_SGES_SLOW_SGL_THR);
+		}
+		SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_NUM_SGES, num_sges);
+		SET_FIELD(task_params->sqe->contlen_cdbsize, NVMETCP_WQE_CONT_LEN, buf_size);
+	} break;
+
+	case NVMETCP_TASK_TYPE_HOST_READ: {
+		SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_WQE_TYPE,
+			  NVMETCP_WQE_TYPE_NORMAL);
+		SET_FIELD(task_params->sqe->contlen_cdbsize,
+			  NVMETCP_WQE_CDB_SIZE_OR_NVMETCP_CMD, 1);
+	} break;
+
+	case NVMETCP_TASK_TYPE_INIT_CONN_REQUEST: {
+		SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_WQE_TYPE,
+			  NVMETCP_WQE_TYPE_MIDDLE_PATH);
+
+		if (task_params->tx_io_size) {
+			SET_FIELD(task_params->sqe->contlen_cdbsize, NVMETCP_WQE_CONT_LEN,
+				  task_params->tx_io_size);
+			SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_NUM_SGES,
+				  min((u16)sgl_task_params->num_sges,
+				      (u16)SCSI_NUM_SGES_SLOW_SGL_THR));
+		}
+	} break;
+
+	case NVMETCP_TASK_TYPE_CLEANUP:
+		SET_FIELD(task_params->sqe->flags, NVMETCP_WQE_WQE_TYPE,
+			  NVMETCP_WQE_TYPE_TASK_CLEANUP);
+
+	default:
+		break;
+	}
+}
+
+/* The following function initializes of NVMeTCP task params */
+static inline void
+init_nvmetcp_task_params(struct e5_nvmetcp_task_context *context,
+			 struct nvmetcp_task_params *task_params,
+			 enum nvmetcp_task_type task_type)
+{
+	context->ystorm_st_context.state.cccid = task_params->host_cccid;
+	SET_FIELD(context->ustorm_st_context.error_flags, USTORM_NVMETCP_TASK_ST_CTX_NVME_TCP, 1);
+	context->ustorm_st_context.nvme_tcp_opaque_lo = cpu_to_le32(task_params->opq.lo);
+	context->ustorm_st_context.nvme_tcp_opaque_hi = cpu_to_le32(task_params->opq.hi);
+}
+
+/* The following function initializes default values to all tasks */
+static inline void
+init_default_nvmetcp_task(struct nvmetcp_task_params *task_params,
+			  void *pdu_header, void *nvme_cmd,
+			  enum nvmetcp_task_type task_type)
+{
+	struct e5_nvmetcp_task_context *context = task_params->context;
+	const u8 val_byte = context->mstorm_ag_context.cdu_validation;
+	u8 dw_index;
+
+	memset(context, 0, sizeof(*context));
+	init_nvmetcp_task_params(context, task_params,
+				 (enum nvmetcp_task_type)task_type);
+
+	/* Swapping requirements used below, will be removed in future FW versions */
+	if (task_type == NVMETCP_TASK_TYPE_HOST_WRITE ||
+	    task_type == NVMETCP_TASK_TYPE_HOST_READ) {
+		for (dw_index = 0;
+		     dw_index < QED_NVMETCP_CMN_HDR_SIZE / sizeof(u32);
+		     dw_index++)
+			context->ystorm_st_context.pdu_hdr.task_hdr.reg[dw_index] =
+				cpu_to_le32(__swab32(((u32 *)pdu_header)[dw_index]));
+
+		for (dw_index = QED_NVMETCP_CMN_HDR_SIZE / sizeof(u32);
+		     dw_index < QED_NVMETCP_CMD_HDR_SIZE / sizeof(u32);
+		     dw_index++)
+			context->ystorm_st_context.pdu_hdr.task_hdr.reg[dw_index] =
+				cpu_to_le32(__swab32(((u32 *)nvme_cmd)[dw_index - 2]));
+	} else {
+		for (dw_index = 0;
+		     dw_index < QED_NVMETCP_NON_IO_HDR_SIZE / sizeof(u32);
+		     dw_index++)
+			context->ystorm_st_context.pdu_hdr.task_hdr.reg[dw_index] =
+				cpu_to_le32(__swab32(((u32 *)pdu_header)[dw_index]));
+	}
+
+	/* M-Storm Context: */
+	context->mstorm_ag_context.cdu_validation = val_byte;
+	context->mstorm_st_context.task_type = (u8)(task_type);
+	context->mstorm_ag_context.task_cid = cpu_to_le16(task_params->conn_icid);
+
+	/* Ustorm Context: */
+	SET_FIELD(context->ustorm_ag_context.flags1, E5_USTORM_NVMETCP_TASK_AG_CTX_R2T2RECV, 1);
+	context->ustorm_st_context.task_type = (u8)(task_type);
+	context->ustorm_st_context.cq_rss_number = task_params->cq_rss_number;
+	context->ustorm_ag_context.icid = cpu_to_le16(task_params->conn_icid);
+}
+
+/* The following function initializes the U-Storm Task Contexts */
+static inline void
+init_ustorm_task_contexts(struct ustorm_nvmetcp_task_st_ctx *ustorm_st_context,
+			  struct e5_ustorm_nvmetcp_task_ag_ctx *ustorm_ag_context,
+			  u32 remaining_recv_len,
+			  u32 expected_data_transfer_len, u8 num_sges,
+			  bool tx_dif_conn_err_en)
+{
+	/* Remaining data to be received in bytes. Used in validations*/
+	ustorm_st_context->rem_rcv_len = cpu_to_le32(remaining_recv_len);
+	ustorm_ag_context->exp_data_acked = cpu_to_le32(expected_data_transfer_len);
+	ustorm_st_context->exp_data_transfer_len = cpu_to_le32(expected_data_transfer_len);
+	SET_FIELD(ustorm_st_context->reg1_map, REG1_NUM_SGES, num_sges);
+	SET_FIELD(ustorm_ag_context->flags2, E5_USTORM_NVMETCP_TASK_AG_CTX_DIF_ERROR_CF_EN,
+		  tx_dif_conn_err_en ? 1 : 0);
+}
+
+/* The following function initializes Local Completion Contexts: */
+static inline void
+set_local_completion_context(struct e5_nvmetcp_task_context *context)
+{
+	SET_FIELD(context->ystorm_st_context.state.flags,
+		  YSTORM_NVMETCP_TASK_STATE_LOCAL_COMP, 1);
+	SET_FIELD(context->ustorm_st_context.flags,
+		  USTORM_NVMETCP_TASK_ST_CTX_LOCAL_COMP, 1);
+}
+
+/* Common Fastpath task init function: */
+static inline void
+init_rw_nvmetcp_task(struct nvmetcp_task_params *task_params,
+		     enum nvmetcp_task_type task_type,
+		     void *pdu_header, void *nvme_cmd,
+		     struct storage_sgl_task_params *sgl_task_params)
+{
+	struct e5_nvmetcp_task_context *context = task_params->context;
+	u32 task_size = calc_rw_task_size(task_params, task_type);
+	bool slow_io = false;
+	u8 num_sges = 0;
+
+	init_default_nvmetcp_task(task_params, pdu_header, nvme_cmd, task_type);
+
+	/* Tx/Rx: */
+	if (task_params->tx_io_size) {
+		/* if data to transmit: */
+		init_scsi_sgl_context(&context->ystorm_st_context.state.sgl_params,
+				      &context->ystorm_st_context.state.data_desc,
+				      sgl_task_params);
+		slow_io = nvmetcp_is_slow_sgl(sgl_task_params->num_sges,
+					      sgl_task_params->small_mid_sge);
+		num_sges =
+			(u8)(!slow_io ? min((u32)sgl_task_params->num_sges,
+					    (u32)SCSI_NUM_SGES_SLOW_SGL_THR) :
+					    NVMETCP_WQE_NUM_SGES_SLOWIO);
+		if (slow_io) {
+			SET_FIELD(context->ystorm_st_context.state.flags,
+				  YSTORM_NVMETCP_TASK_STATE_SLOW_IO, 1);
+		}
+	} else if (task_params->rx_io_size) {
+		/* if data to receive: */
+		init_scsi_sgl_context(&context->mstorm_st_context.sgl_params,
+				      &context->mstorm_st_context.data_desc,
+				      sgl_task_params);
+		num_sges =
+			(u8)(!nvmetcp_is_slow_sgl(sgl_task_params->num_sges,
+						  sgl_task_params->small_mid_sge) ?
+						  min((u32)sgl_task_params->num_sges,
+						      (u32)SCSI_NUM_SGES_SLOW_SGL_THR) :
+						      NVMETCP_WQE_NUM_SGES_SLOWIO);
+		context->mstorm_st_context.rem_task_size = cpu_to_le32(task_size);
+	}
+
+	/* Ustorm context: */
+	init_ustorm_task_contexts(&context->ustorm_st_context,
+				  &context->ustorm_ag_context,
+				  /* Remaining Receive length is the Task Size */
+				  task_size,
+				  /* The size of the transmitted task */
+				  task_size,
+				  /* num_sges */
+				  num_sges,
+				  false);
+
+	/* Set exp_data_acked */
+	if (task_type == NVMETCP_TASK_TYPE_HOST_WRITE) {
+		if (task_params->send_write_incapsule)
+			context->ustorm_ag_context.exp_data_acked = task_size;
+		else
+			context->ustorm_ag_context.exp_data_acked = 0;
+	} else if (task_type == NVMETCP_TASK_TYPE_HOST_READ) {
+		context->ustorm_ag_context.exp_data_acked = 0;
+	}
+
+	context->ustorm_ag_context.exp_cont_len = 0;
+	init_sqe(task_params, sgl_task_params, task_type);
+}
+
+static void
+init_common_initiator_read_task(struct nvmetcp_task_params *task_params,
+				struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+				struct nvme_command *nvme_cmd,
+				struct storage_sgl_task_params *sgl_task_params)
+{
+	init_rw_nvmetcp_task(task_params, NVMETCP_TASK_TYPE_HOST_READ,
+			     cmd_pdu_header, nvme_cmd, sgl_task_params);
+}
+
+void init_nvmetcp_host_read_task(struct nvmetcp_task_params *task_params,
+				 struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+				 struct nvme_command *nvme_cmd,
+				 struct storage_sgl_task_params *sgl_task_params)
+{
+	init_common_initiator_read_task(task_params, (void *)cmd_pdu_header,
+					(void *)nvme_cmd, sgl_task_params);
+}
+
+static void
+init_common_initiator_write_task(struct nvmetcp_task_params *task_params,
+				 struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+				 struct nvme_command *nvme_cmd,
+				 struct storage_sgl_task_params *sgl_task_params)
+{
+	init_rw_nvmetcp_task(task_params, NVMETCP_TASK_TYPE_HOST_WRITE,
+			     cmd_pdu_header, nvme_cmd, sgl_task_params);
+}
+
+void init_nvmetcp_host_write_task(struct nvmetcp_task_params *task_params,
+				  struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+				  struct nvme_command *nvme_cmd,
+				  struct storage_sgl_task_params *sgl_task_params)
+{
+	init_common_initiator_write_task(task_params, (void *)cmd_pdu_header,
+					 (void *)nvme_cmd, sgl_task_params);
+}
+
+static void
+init_common_login_request_task(struct nvmetcp_task_params *task_params,
+			       void *login_req_pdu_header,
+			       struct storage_sgl_task_params *tx_sgl_task_params,
+			       struct storage_sgl_task_params *rx_sgl_task_params)
+{
+	struct e5_nvmetcp_task_context *context = task_params->context;
+
+	init_default_nvmetcp_task(task_params, (void *)login_req_pdu_header, NULL,
+				  NVMETCP_TASK_TYPE_INIT_CONN_REQUEST);
+
+	/* Ustorm Context: */
+	init_ustorm_task_contexts(&context->ustorm_st_context,
+				  &context->ustorm_ag_context,
+
+				  /* Remaining Receive length is the Task Size */
+				  task_params->rx_io_size ?
+				  rx_sgl_task_params->total_buffer_size : 0,
+
+				  /* The size of the transmitted task */
+				  task_params->tx_io_size ?
+				  tx_sgl_task_params->total_buffer_size : 0,
+				  0, /* num_sges */
+				  0); /* tx_dif_conn_err_en */
+
+	/* SGL context: */
+	if (task_params->tx_io_size)
+		init_scsi_sgl_context(&context->ystorm_st_context.state.sgl_params,
+				      &context->ystorm_st_context.state.data_desc,
+				      tx_sgl_task_params);
+	if (task_params->rx_io_size)
+		init_scsi_sgl_context(&context->mstorm_st_context.sgl_params,
+				      &context->mstorm_st_context.data_desc,
+				      rx_sgl_task_params);
+
+	context->mstorm_st_context.rem_task_size =
+		cpu_to_le32(task_params->rx_io_size ?
+				 rx_sgl_task_params->total_buffer_size : 0);
+	init_sqe(task_params, tx_sgl_task_params, NVMETCP_TASK_TYPE_INIT_CONN_REQUEST);
+}
+
+/* The following function initializes Login task in Host mode: */
+void init_nvmetcp_init_conn_req_task(struct nvmetcp_task_params *task_params,
+				     struct nvme_tcp_icreq_pdu *init_conn_req_pdu_hdr,
+				     struct storage_sgl_task_params *tx_sgl_task_params,
+				     struct storage_sgl_task_params *rx_sgl_task_params)
+{
+	init_common_login_request_task(task_params, init_conn_req_pdu_hdr,
+				       tx_sgl_task_params, rx_sgl_task_params);
+}
+
+void init_cleanup_task_nvmetcp(struct nvmetcp_task_params *task_params)
+{
+	init_sqe(task_params, NULL, NVMETCP_TASK_TYPE_CLEANUP);
+}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp_fw_funcs.h b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp_fw_funcs.h
new file mode 100644
index 000000000000..4c7ac2bd2ea5
--- /dev/null
+++ b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp_fw_funcs.h
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/* Copyright 2021 Marvell. All rights reserved. */
+
+#ifndef _QED_NVMETCP_FW_FUNCS_H
+#define _QED_NVMETCP_FW_FUNCS_H
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/mm.h>
+#include <linux/types.h>
+#include <asm/byteorder.h>
+#include <linux/qed/common_hsi.h>
+#include <linux/qed/storage_common.h>
+#include <linux/qed/nvmetcp_common.h>
+#include <linux/qed/qed_nvmetcp_if.h>
+
+#if IS_ENABLED(CONFIG_QED_NVMETCP)
+
+void init_nvmetcp_host_read_task(struct nvmetcp_task_params *task_params,
+				 struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+				 struct nvme_command *nvme_cmd,
+				 struct storage_sgl_task_params *sgl_task_params);
+void init_nvmetcp_host_write_task(struct nvmetcp_task_params *task_params,
+				  struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+				  struct nvme_command *nvme_cmd,
+				  struct storage_sgl_task_params *sgl_task_params);
+void init_nvmetcp_init_conn_req_task(struct nvmetcp_task_params *task_params,
+				     struct nvme_tcp_icreq_pdu *init_conn_req_pdu_hdr,
+				     struct storage_sgl_task_params *tx_sgl_task_params,
+				     struct storage_sgl_task_params *rx_sgl_task_params);
+void init_cleanup_task_nvmetcp(struct nvmetcp_task_params *task_params);
+
+#else /* IS_ENABLED(CONFIG_QED_NVMETCP) */
+
+#endif /* IS_ENABLED(CONFIG_QED_NVMETCP) */
+
+#endif /* _QED_NVMETCP_FW_FUNCS_H */
diff --git a/include/linux/qed/nvmetcp_common.h b/include/linux/qed/nvmetcp_common.h
index ad745a9c2264..5a2ab0606308 100644
--- a/include/linux/qed/nvmetcp_common.h
+++ b/include/linux/qed/nvmetcp_common.h
@@ -5,6 +5,7 @@
 #define __NVMETCP_COMMON__
 
 #include "tcp_common.h"
+#include <linux/nvme-tcp.h>
 
 #define NVMETCP_SLOW_PATH_LAYER_CODE (6)
 #define NVMETCP_WQE_NUM_SGES_SLOWIO (0xf)
diff --git a/include/linux/qed/qed_nvmetcp_if.h b/include/linux/qed/qed_nvmetcp_if.h
index 606427ebb63c..14671bc19ed1 100644
--- a/include/linux/qed/qed_nvmetcp_if.h
+++ b/include/linux/qed/qed_nvmetcp_if.h
@@ -9,6 +9,9 @@
 #include <linux/qed/nvmetcp_common.h>
 
 #define QED_NVMETCP_MAX_IO_SIZE	0x800000
+#define QED_NVMETCP_CMN_HDR_SIZE (sizeof(struct nvme_tcp_hdr))
+#define QED_NVMETCP_CMD_HDR_SIZE (sizeof(struct nvme_tcp_cmd_pdu))
+#define QED_NVMETCP_NON_IO_HDR_SIZE ((QED_NVMETCP_CMN_HDR_SIZE + 16))
 
 typedef int (*nvmetcp_event_cb_t) (void *context,
 				   u8 fw_event_code, void *fw_handle);
@@ -213,6 +216,23 @@ struct qed_nvmetcp_ops {
 	void (*remove_dst_tcp_port_filter)(struct qed_dev *cdev, u16 dest_port);
 
 	void (*clear_all_filters)(struct qed_dev *cdev);
+
+	void (*init_read_io)(struct nvmetcp_task_params *task_params,
+			     struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+			     struct nvme_command *nvme_cmd,
+			     struct storage_sgl_task_params *sgl_task_params);
+
+	void (*init_write_io)(struct nvmetcp_task_params *task_params,
+			      struct nvme_tcp_cmd_pdu *cmd_pdu_header,
+			      struct nvme_command *nvme_cmd,
+			      struct storage_sgl_task_params *sgl_task_params);
+
+	void (*init_icreq_exchange)(struct nvmetcp_task_params *task_params,
+				    struct nvme_tcp_icreq_pdu *init_conn_req_pdu_hdr,
+				    struct storage_sgl_task_params *tx_sgl_task_params,
+				    struct storage_sgl_task_params *rx_sgl_task_params);
+
+	void (*init_task_cleanup)(struct nvmetcp_task_params *task_params);
 };
 
 const struct qed_nvmetcp_ops *qed_get_nvmetcp_ops(void);
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 10:16:55 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 20:16:55 +0300
Subject: [PATCH 7/7] qed: Add IP services APIs support
In-Reply-To: <20210602171655.23581-1-smalin@marvell.com>
References: <20210602171655.23581-1-smalin@marvell.com>
Message-ID: <20210602171655.23581-8-smalin@marvell.com>

From: Nikolay Assa <nassa at marvell.com>

This patch introduces APIs which the NVMeTCP Offload device (qedn)
will use through the paired net-device (qede).
It includes APIs for:
- ipv4/ipv6 routing
- get VLAN from net-device
- TCP ports reservation

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Nikolay Assa <nassa at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 .../qlogic/qed/qed_nvmetcp_ip_services.c      | 238 ++++++++++++++++++
 .../linux/qed/qed_nvmetcp_ip_services_if.h    |  29 +++
 2 files changed, 267 insertions(+)
 create mode 100644 drivers/net/ethernet/qlogic/qed/qed_nvmetcp_ip_services.c
 create mode 100644 include/linux/qed/qed_nvmetcp_ip_services_if.h

diff --git a/drivers/net/ethernet/qlogic/qed/qed_nvmetcp_ip_services.c b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp_ip_services.c
new file mode 100644
index 000000000000..96a2077fd315
--- /dev/null
+++ b/drivers/net/ethernet/qlogic/qed/qed_nvmetcp_ip_services.c
@@ -0,0 +1,238 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause)
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#include <linux/types.h>
+#include <asm/byteorder.h>
+#include <asm/param.h>
+#include <linux/delay.h>
+#include <linux/pci.h>
+#include <linux/dma-mapping.h>
+#include <linux/etherdevice.h>
+#include <linux/kernel.h>
+#include <linux/stddef.h>
+#include <linux/errno.h>
+
+#include <net/tcp.h>
+
+#include <linux/qed/qed_nvmetcp_ip_services_if.h>
+
+#define QED_IP_RESOL_TIMEOUT  4
+
+int qed_route_ipv4(struct sockaddr_storage *local_addr,
+		   struct sockaddr_storage *remote_addr,
+		   struct sockaddr *hardware_address,
+		   struct net_device **ndev)
+{
+	struct neighbour *neigh = NULL;
+	__be32 *loc_ip, *rem_ip;
+	struct rtable *rt;
+	int rc = -ENXIO;
+	int retry;
+
+	loc_ip = &((struct sockaddr_in *)local_addr)->sin_addr.s_addr;
+	rem_ip = &((struct sockaddr_in *)remote_addr)->sin_addr.s_addr;
+	*ndev = NULL;
+	rt = ip_route_output(&init_net, *rem_ip, *loc_ip, 0/*tos*/, 0/*oif*/);
+	if (IS_ERR(rt)) {
+		pr_err("lookup route failed\n");
+		rc = PTR_ERR(rt);
+		goto return_err;
+	}
+
+	neigh = dst_neigh_lookup(&rt->dst, rem_ip);
+	if (!neigh) {
+		rc = -ENOMEM;
+		ip_rt_put(rt);
+		goto return_err;
+	}
+
+	*ndev = rt->dst.dev;
+	ip_rt_put(rt);
+
+	/* If not resolved, kick-off state machine towards resolution */
+	if (!(neigh->nud_state & NUD_VALID))
+		neigh_event_send(neigh, NULL);
+
+	/* query neighbor until resolved or timeout */
+	retry = QED_IP_RESOL_TIMEOUT;
+	while (!(neigh->nud_state & NUD_VALID) && retry > 0) {
+		msleep(1000);
+		retry--;
+	}
+
+	if (neigh->nud_state & NUD_VALID) {
+		/* copy resolved MAC address */
+		neigh_ha_snapshot(hardware_address->sa_data, neigh, *ndev);
+		hardware_address->sa_family = (*ndev)->type;
+		rc = 0;
+	}
+
+	neigh_release(neigh);
+	if (!(*loc_ip)) {
+		*loc_ip = inet_select_addr(*ndev, *rem_ip, RT_SCOPE_UNIVERSE);
+		local_addr->ss_family = AF_INET;
+	}
+
+return_err:
+
+	return rc;
+}
+EXPORT_SYMBOL(qed_route_ipv4);
+
+int qed_route_ipv6(struct sockaddr_storage *local_addr,
+		   struct sockaddr_storage *remote_addr,
+		   struct sockaddr *hardware_address,
+		   struct net_device **ndev)
+{
+	struct neighbour *neigh = NULL;
+	struct dst_entry *dst;
+	struct flowi6 fl6;
+	int rc = -ENXIO;
+	int retry;
+
+	memset(&fl6, 0, sizeof(fl6));
+	fl6.saddr = ((struct sockaddr_in6 *)local_addr)->sin6_addr;
+	fl6.daddr = ((struct sockaddr_in6 *)remote_addr)->sin6_addr;
+	dst = ip6_route_output(&init_net, NULL, &fl6);
+	if (!dst || dst->error) {
+		if (dst) {
+			dst_release(dst);
+			pr_err("lookup route failed %d\n", dst->error);
+		}
+
+		goto out;
+	}
+
+	neigh = dst_neigh_lookup(dst, &fl6.daddr);
+	if (neigh) {
+		*ndev = ip6_dst_idev(dst)->dev;
+
+		/* If not resolved, kick-off state machine towards resolution */
+		if (!(neigh->nud_state & NUD_VALID))
+			neigh_event_send(neigh, NULL);
+
+		/* query neighbor until resolved or timeout */
+		retry = QED_IP_RESOL_TIMEOUT;
+		while (!(neigh->nud_state & NUD_VALID) && retry > 0) {
+			msleep(1000);
+			retry--;
+		}
+
+		if (neigh->nud_state & NUD_VALID) {
+			neigh_ha_snapshot((u8 *)hardware_address->sa_data,
+					  neigh, *ndev);
+			hardware_address->sa_family = (*ndev)->type;
+			rc = 0;
+		}
+
+		neigh_release(neigh);
+
+		if (ipv6_addr_any(&fl6.saddr)) {
+			if (ipv6_dev_get_saddr(dev_net(*ndev), *ndev,
+					       &fl6.daddr, 0, &fl6.saddr)) {
+				pr_err("Unable to find source IP address\n");
+				goto out;
+			}
+
+			local_addr->ss_family = AF_INET6;
+			((struct sockaddr_in6 *)local_addr)->sin6_addr =
+								fl6.saddr;
+		}
+	}
+
+	dst_release(dst);
+
+out:
+
+	return rc;
+}
+EXPORT_SYMBOL(qed_route_ipv6);
+
+void qed_vlan_get_ndev(struct net_device **ndev, u16 *vlan_id)
+{
+	if (is_vlan_dev(*ndev)) {
+		*vlan_id = vlan_dev_vlan_id(*ndev);
+		*ndev = vlan_dev_real_dev(*ndev);
+	}
+}
+EXPORT_SYMBOL(qed_vlan_get_ndev);
+
+struct pci_dev *qed_validate_ndev(struct net_device *ndev)
+{
+	struct pci_dev *pdev = NULL;
+	struct net_device *upper;
+
+	for_each_pci_dev(pdev) {
+		if (pdev && pdev->driver &&
+		    !strcmp(pdev->driver->name, "qede")) {
+			upper = pci_get_drvdata(pdev);
+			if (upper->ifindex == ndev->ifindex)
+				return pdev;
+		}
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL(qed_validate_ndev);
+
+__be16 qed_get_in_port(struct sockaddr_storage *sa)
+{
+	return sa->ss_family == AF_INET
+		? ((struct sockaddr_in *)sa)->sin_port
+		: ((struct sockaddr_in6 *)sa)->sin6_port;
+}
+EXPORT_SYMBOL(qed_get_in_port);
+
+int qed_fetch_tcp_port(struct sockaddr_storage local_ip_addr,
+		       struct socket **sock, u16 *port)
+{
+	struct sockaddr_storage sa;
+	int rc = 0;
+
+	rc = sock_create(local_ip_addr.ss_family, SOCK_STREAM, IPPROTO_TCP,
+			 sock);
+	if (rc) {
+		pr_warn("failed to create socket: %d\n", rc);
+		goto err;
+	}
+
+	(*sock)->sk->sk_allocation = GFP_KERNEL;
+	sk_set_memalloc((*sock)->sk);
+
+	rc = kernel_bind(*sock, (struct sockaddr *)&local_ip_addr,
+			 sizeof(local_ip_addr));
+
+	if (rc) {
+		pr_warn("failed to bind socket: %d\n", rc);
+		goto err_sock;
+	}
+
+	rc = kernel_getsockname(*sock, (struct sockaddr *)&sa);
+	if (rc < 0) {
+		pr_warn("getsockname() failed: %d\n", rc);
+		goto err_sock;
+	}
+
+	*port = ntohs(qed_get_in_port(&sa));
+
+	return 0;
+
+err_sock:
+	sock_release(*sock);
+	sock = NULL;
+err:
+
+	return rc;
+}
+EXPORT_SYMBOL(qed_fetch_tcp_port);
+
+void qed_return_tcp_port(struct socket *sock)
+{
+	if (sock && sock->sk) {
+		tcp_set_state(sock->sk, TCP_CLOSE);
+		sock_release(sock);
+	}
+}
+EXPORT_SYMBOL(qed_return_tcp_port);
diff --git a/include/linux/qed/qed_nvmetcp_ip_services_if.h b/include/linux/qed/qed_nvmetcp_ip_services_if.h
new file mode 100644
index 000000000000..3604aee53796
--- /dev/null
+++ b/include/linux/qed/qed_nvmetcp_ip_services_if.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#ifndef _QED_IP_SERVICES_IF_H
+#define _QED_IP_SERVICES_IF_H
+
+#include <linux/types.h>
+#include <net/route.h>
+#include <net/ip6_route.h>
+#include <linux/inetdevice.h>
+
+int qed_route_ipv4(struct sockaddr_storage *local_addr,
+		   struct sockaddr_storage *remote_addr,
+		   struct sockaddr *hardware_address,
+		   struct net_device **ndev);
+int qed_route_ipv6(struct sockaddr_storage *local_addr,
+		   struct sockaddr_storage *remote_addr,
+		   struct sockaddr *hardware_address,
+		   struct net_device **ndev);
+void qed_vlan_get_ndev(struct net_device **ndev, u16 *vlan_id);
+struct pci_dev *qed_validate_ndev(struct net_device *ndev);
+void qed_return_tcp_port(struct socket *sock);
+int qed_fetch_tcp_port(struct sockaddr_storage local_ip_addr,
+		       struct socket **sock, u16 *port);
+__be16 qed_get_in_port(struct sockaddr_storage *sa);
+
+#endif /* _QED_IP_SERVICES_IF_H */
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 11:42:38 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 21:42:38 +0300
Subject: [PATCH 0/8] NVMeTCP Offload ULP
Message-ID: <20210602184246.14184-1-smalin@marvell.com>

With the goal of enabling a generic infrastructure that allows NVMe/TCP
offload devices like NICs to seamlessly plug into the NVMe-oF stack, this
patch series introduces the nvme-tcp-offload ULP host layer, which will
be a new transport type called "tcp-offload" and will serve as an 
abstraction layer to work with vendor specific nvme-tcp offload drivers.

NVMeTCP offload is a full offload of the NVMeTCP protocol, this includes 
both the TCP level and the NVMeTCP level.

The nvme-tcp-offload transport can co-exist with the existing tcp and 
other transports. The tcp offload was designed so that stack changes are 
kept to a bare minimum: only registering new transports.
All other APIs, ops etc. are identical to the regular tcp transport.
Representing the TCP offload as a new transport allows clear and manageable
differentiation between the connections which should use the offload path 
and those that are not offloaded (even on the same device).

The nvme-tcp-offload layers and API compared to nvme-tcp and nvme-rdma:

* NVMe layer: *

       [ nvme/nvme-fabrics/blk-mq ]
             |
        (nvme API and blk-mq API)
             |
             |			 
* Vendor agnostic transport layer: *

      [ nvme-rdma ] [ nvme-tcp ] [ nvme-tcp-offload ]
             |        |             |
           (Verbs) 
             |        |             |
             |     (Socket)
             |        |             |
             |        |        (nvme-tcp-offload API)
             |        |             |
             |        |             |
* Vendor Specific Driver: *

             |        |             |
           [ qedr ]       
                      |             |
                   [ qede ]
                                    |
                                  [ qedn ]


Performance:
============
With this implementation on top of the Marvell qedn driver (using the
Marvell FastLinQ NIC), we were able to demonstrate the following CPU
utilization improvement:

On AMD EPYC 7402, 2.80GHz, 28 cores:
- For 16K queued read IOs, 16jobs, 4qd (50Gbps line rate): 
  Improved the CPU utilization from 15.1% with NVMeTCP SW to 4.7% with 
  NVMeTCP offload.

On Intel(R) Xeon(R) Gold 5122 CPU, 3.60GHz, 16 cores: 
- For 512K queued read IOs, 16jobs, 4qd (25Gbps line rate): 
  Improved the CPU utilization from 16.3% with NVMeTCP SW to 1.1% with 
  NVMeTCP offload.

In addition, we were able to demonstrate the following latency improvement:
- For 200K read IOPS (16 jobs, 16 qd, with fio rate limiter):
  Improved the average latency from 105 usec with NVMeTCP SW to 39 usec 
  with NVMeTCP offload.
  
  Improved the 99.99 tail latency from 570 usec with NVMeTCP SW to 91 usec 
  with NVMeTCP offload.

The end-to-end offload latency was measured from fio while running against 
back end of null device.


Upstream plan:
==============
The RFC series "NVMeTCP Offload ULP and QEDN Device Driver" 
https://lore.kernel.org/netdev/20210531225222.16992-1-smalin at marvell.com/
was designed in a modular way so that part 1 (nvme-tcp-offload) and 
part 2 (qed) are independent and part 3 (qedn) depends on both parts 1+2.

- Part 1 (RFC patch 1-8): NVMeTCP Offload ULP
  The nvme-tcp-offload patches, will be sent to 
  'linux-nvme at lists.infradead.org'.
  
- Part 2 (RFC patches 9-15): QED NVMeTCP Offload
  The qed infrastructure, will be sent to 'netdev at vger.kernel.org'.

Once part 1 and 2 are accepted:

- Part 3 (RFC patches 16-27): QEDN NVMeTCP Offload
  The qedn patches, will be sent to 'linux-nvme at lists.infradead.org'.
 
Marvell is fully committed to maintain, test, and address issues with 
the new nvme-tcp-offload layer.


Usage:
======
With the Marvell NVMeTCP offload design, the network-device (qede) and the 
offload-device (qedn) are paired on each port - Logically similar to the 
RDMA model.
The user will interact with the network-device in order to configure 
the ip/vlan. The NVMeTCP configuration is populated as part of the 
nvme connect command.

Example:
Assign IP to the net-device (from any existing Linux tool):

    ip addr add 100.100.0.101/24 dev p1p1

This IP will be used by both net-device (qede) and offload-device (qedn).

In order to connect from "sw" nvme-tcp through the net-device (qede):

    nvme connect -t tcp -s 4420 -a 100.100.0.100 -n testnqn

In order to connect from "offload" nvme-tcp through the offload-device (qedn):

    nvme connect -t tcp_offload -s 4420 -a 100.100.0.100 -n testnqn
	
An alternative approach, and as a future enhancement that will not impact this 
series will be to modify nvme-cli with a new flag that will determine 
if "-t tcp" should be the regular nvme-tcp (which will be the default) 
or nvme-tcp-offload.
Exmaple:
    nvme connect -t tcp -s 4420 -a 100.100.0.100 -n testnqn -[new flag]


Queue Initialization Design:
============================
The nvme-tcp-offload ULP module shall register with the existing 
nvmf_transport_ops (.name = "tcp_offload"), nvme_ctrl_ops and blk_mq_ops.
The nvme-tcp-offload vendor driver shall register to nvme-tcp-offload ULP
with the following ops:
- claim_dev() - in order to resolve the route to the target according to
                the paired net_dev.
- create_queue() - in order to create offloaded nvme-tcp queue.

The nvme-tcp-offload ULP module shall manage all the controller level
functionalities, call claim_dev and based on the return values shall call
the relevant module create_queue in order to create the admin queue and
the IO queues.


IO-path Design:
===============
The nvme-tcp-offload shall work at the IO-level - the nvme-tcp-offload 
ULP module shall pass the request (the IO) to the nvme-tcp-offload vendor
driver and later, the nvme-tcp-offload vendor driver returns the request
completion (the IO completion).
No additional handling is needed in between; this design will reduce the
CPU utilization as we will describe below.

The nvme-tcp-offload vendor driver shall register to nvme-tcp-offload ULP
with the following IO-path ops:
- send_req() - in order to pass the request to the handling of the
               offload driver that shall pass it to the vendor specific device.
- poll_queue()

Once the IO completes, the nvme-tcp-offload vendor driver shall call 
command.done() that will invoke the nvme-tcp-offload ULP layer to
complete the request.


TCP events:
===========
The Marvell FastLinQ NIC HW engine handle all the TCP re-transmissions
and OOO events.


Teardown and errors:
====================
In case of NVMeTCP queue error the nvme-tcp-offload vendor driver shall
call the nvme_tcp_ofld_report_queue_err.
The nvme-tcp-offload vendor driver shall register to nvme-tcp-offload ULP
with the following teardown ops:
- drain_queue()
- destroy_queue()


The Marvell FastLinQ NIC HW engine:
====================================
The Marvell NIC HW engine is capable of offloading the entire TCP/IP
stack and managing up to 64K connections per PF, already implemented and 
upstream use cases for this include iWARP (by the Marvell qedr driver) 
and iSCSI (by the Marvell qedi driver).
In addition, the Marvell NIC HW engine offloads the NVMeTCP queue layer
and is able to manage the IO level also in case of TCP re-transmissions
and OOO events.
The HW engine enables direct data placement (including the data digest CRC
calculation and validation) and direct data transmission (including data
digest CRC calculation).


The Marvell qedn driver:
========================
The new driver will be added under "drivers/nvme/hw" and will be enabled
by the Kconfig "Marvell NVM Express over Fabrics TCP offload".
As part of the qedn init, the driver will register as a pci device driver 
and will work with the Marvell fastlinQ NIC.
As part of the probe, the driver will register to the nvme_tcp_offload
(ULP) and to the qed module (qed_nvmetcp_ops) - similar to other
"qed_*_ops" which are used by the qede, qedr, qedf and qedi device
drivers.


nvme-tcp-offload Future work:
=============================
- NVMF_OPT_HOST_IFACE Support.


Changes since RFC v1:
=====================
- nvme-tcp-offload: Fix nvme_tcp_ofld_ops return values.
- nvme-tcp-offload: Remove NVMF_TRTYPE_TCP_OFFLOAD.
- nvme-tcp-offload: Add nvme_tcp_ofld_poll() implementation.
- nvme-tcp-offload: Fix nvme_tcp_ofld_queue_rq() to check map_sg() and 
  send_req() return values.

Changes since RFC v2:
=====================
- nvme-tcp-offload: Fixes in controller and queue level (patches 3-6).
- qedn: Add the Marvell's NVMeTCP HW offload vendor driver init and probe
  (patches 8-11).
  
Changes since RFC v3:
=====================
- nvme-tcp-offload: Add the full implementation of the nvme-tcp-offload layer 
  including the new ops: setup_ctrl(), release_ctrl(), commit_rqs() and new 
  flows (ASYNC and timeout).
- nvme-tcp-offload: Add device maximums: max_hw_sectors, max_segments.
- nvme-tcp-offload: layer design and optimization changes.

Changes since RFC v4:
=====================
(Many thanks to Hannes Reinecke for his feedback)
- nvme_tcp_offload: Add num_hw_vectors in order to limit the number of queues.
- nvme_tcp_offload: Add per device private_data.
- nvme_tcp_offload: Fix header digest, data digest and tos initialization.

Changes since RFC v5:
=====================
(Many thanks to Sagi Grimberg for his feedback)
- nvme-fabrics: Expose nvmf_check_required_opts() globally (as a new patch).
- nvme_tcp_offload: Remove io-queues BLK_MQ_F_BLOCKING.
- nvme_tcp_offload: Fix the nvme_tcp_ofld_stop_queue (drain_queue) flow.
- nvme_tcp_offload: Fix the nvme_tcp_ofld_free_queue (destroy_queue) flow.
- nvme_tcp_offload: Change rwsem to mutex.
- nvme_tcp_offload: remove redundant fields.
- nvme_tcp_offload: Remove the "new" from setup_ctrl().
- nvme_tcp_offload: Remove the init_req() and commit_rqs() ops.
- nvme_tcp_offload: Minor fixes in nvme_tcp_ofld_create_ctrl() ansd 
  nvme_tcp_ofld_free_queue().
- nvme_tcp_offload: Patch 8 (timeout and async) was squeashed into 
  patch 7 (io level).

Changes since RFC v6:
=====================
- No changes in nvme_tcp_offload (only in qedn).


Arie Gershberg (2):
  nvme-tcp-offload: Add controller level implementation
  nvme-tcp-offload: Add controller level error recovery implementation

Dean Balandin (3):
  nvme-tcp-offload: Add device scan implementation
  nvme-tcp-offload: Add queue level implementation
  nvme-tcp-offload: Add IO level implementation

Prabhakar Kushwaha (2):
  nvme-fabrics: Move NVMF_ALLOWED_OPTS and NVMF_REQUIRED_OPTS
    definitions
  nvme-fabrics: Expose nvmf_check_required_opts() globally

Shai Malin (1):
  nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW offload ULP

 MAINTAINERS                     |    8 +
 drivers/nvme/host/Kconfig       |   17 +
 drivers/nvme/host/Makefile      |    3 +
 drivers/nvme/host/fabrics.c     |   12 +-
 drivers/nvme/host/fabrics.h     |    9 +
 drivers/nvme/host/tcp-offload.c | 1318 +++++++++++++++++++++++++++++++
 drivers/nvme/host/tcp-offload.h |  206 +++++
 7 files changed, 1564 insertions(+), 9 deletions(-)
 create mode 100644 drivers/nvme/host/tcp-offload.c
 create mode 100644 drivers/nvme/host/tcp-offload.h

-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 11:42:40 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 21:42:40 +0300
Subject: [PATCH 2/8] nvme-fabrics: Move NVMF_ALLOWED_OPTS and
 NVMF_REQUIRED_OPTS definitions
In-Reply-To: <20210602184246.14184-1-smalin@marvell.com>
References: <20210602184246.14184-1-smalin@marvell.com>
Message-ID: <20210602184246.14184-3-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

Move NVMF_ALLOWED_OPTS and NVMF_REQUIRED_OPTS definitions
to header file, so it can be used by the different HW devices.

NVMeTCP offload devices might have different limitations of the
allowed options, for example, a device that does not support all the
queue types. With tcp and rdma, only the nvme-tcp and nvme-rdma layers
handle those attributes and the HW devices do not create any limitations
for the allowed options.

An alternative design could be to add separate fields in
nvme_tcp_ofld_ops such as max_hw_sectors and max_segments that
we already have in this series.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Acked-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/fabrics.c | 7 -------
 drivers/nvme/host/fabrics.h | 7 +++++++
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 34a84d2086c7..e8f6b8f9fd35 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -947,13 +947,6 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
 
-#define NVMF_REQUIRED_OPTS	(NVMF_OPT_TRANSPORT | NVMF_OPT_NQN)
-#define NVMF_ALLOWED_OPTS	(NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \
-				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
-				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
-				 NVMF_OPT_DISABLE_SQFLOW |\
-				 NVMF_OPT_FAIL_FAST_TMO)
-
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
 {
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index d7f7974dc208..ce7fe3a842b1 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -68,6 +68,13 @@ enum {
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 };
 
+#define NVMF_REQUIRED_OPTS	(NVMF_OPT_TRANSPORT | NVMF_OPT_NQN)
+#define NVMF_ALLOWED_OPTS	(NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \
+				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
+				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
+				 NVMF_OPT_DISABLE_SQFLOW |\
+				 NVMF_OPT_FAIL_FAST_TMO)
+
 /**
  * struct nvmf_ctrl_options - Used to hold the options specified
  *			      with the parsing opts enum.
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 11:42:39 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 21:42:39 +0300
Subject: [PATCH 1/8] nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW
 offload ULP
In-Reply-To: <20210602184246.14184-1-smalin@marvell.com>
References: <20210602184246.14184-1-smalin@marvell.com>
Message-ID: <20210602184246.14184-2-smalin@marvell.com>

This patch will present the structure for the NVMeTCP offload common
layer driver. This module is added under "drivers/nvme/host/" and future
offload drivers which will register to it will be placed under
"drivers/nvme/hw".
This new driver will be enabled by the Kconfig "NVM Express over Fabrics
TCP offload commmon layer".
In order to support the new transport type, for host mode, no change is
needed.

Each new vendor-specific offload driver will register to this ULP during
its probe function, by filling out the nvme_tcp_ofld_dev->ops and
nvme_tcp_ofld_dev->private_data and calling nvme_tcp_ofld_register_dev
with the initialized struct.

The internal implementation:
- tcp-offload.h:
  Includes all common structs and ops to be used and shared by offload
  drivers.

- tcp-offload.c:
  Includes the init function which registers as a NVMf transport just
  like any other transport.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 MAINTAINERS                     |   8 ++
 drivers/nvme/host/Kconfig       |  17 +++
 drivers/nvme/host/Makefile      |   3 +
 drivers/nvme/host/tcp-offload.c | 124 ++++++++++++++++++++
 drivers/nvme/host/tcp-offload.h | 199 ++++++++++++++++++++++++++++++++
 5 files changed, 351 insertions(+)
 create mode 100644 drivers/nvme/host/tcp-offload.c
 create mode 100644 drivers/nvme/host/tcp-offload.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 5c90148f0369..64cdffb8aaed 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13003,6 +13003,14 @@ F:	drivers/nvme/host/
 F:	include/linux/nvme.h
 F:	include/uapi/linux/nvme_ioctl.h
 
+NVM EXPRESS TCP OFFLOAD TRANSPORT DRIVERS
+M:	Shai Malin <smalin at marvell.com>
+M:	Ariel Elior <aelior at marvell.com>
+L:	linux-nvme at lists.infradead.org
+S:	Supported
+F:	drivers/nvme/host/tcp-offload.c
+F:	drivers/nvme/host/tcp-offload.h
+
 NVM EXPRESS FC TRANSPORT DRIVERS
 M:	James Smart <james.smart at broadcom.com>
 L:	linux-nvme at lists.infradead.org
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 494675aeaaad..9c6f4d776daf 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -85,3 +85,20 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_TCP_OFFLOAD
+	tristate "NVM Express over Fabrics TCP offload common layer"
+	default m
+	depends on BLOCK
+	depends on INET
+	select NVME_CORE
+	select NVME_FABRICS
+	help
+	  This provides support for the NVMe over Fabrics protocol using
+	  the TCP offload transport. This allows you to use remote block devices
+	  exported using the NVMe protocol set.
+
+	  To configure a NVMe over Fabrics controller use the nvme-cli tool
+	  from https://github.com/linux-nvme/nvme-cli.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index cbc509784b2e..3c3fdf83ce38 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -8,6 +8,7 @@ obj-$(CONFIG_NVME_FABRICS)		+= nvme-fabrics.o
 obj-$(CONFIG_NVME_RDMA)			+= nvme-rdma.o
 obj-$(CONFIG_NVME_FC)			+= nvme-fc.o
 obj-$(CONFIG_NVME_TCP)			+= nvme-tcp.o
+obj-$(CONFIG_NVME_TCP_OFFLOAD)	+= nvme-tcp-offload.o
 
 nvme-core-y				:= core.o ioctl.o
 nvme-core-$(CONFIG_TRACING)		+= trace.o
@@ -26,3 +27,5 @@ nvme-rdma-y				+= rdma.o
 nvme-fc-y				+= fc.o
 
 nvme-tcp-y				+= tcp.o
+
+nvme-tcp-offload-y		+= tcp-offload.o
diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
new file mode 100644
index 000000000000..f7aa49f337dc
--- /dev/null
+++ b/drivers/nvme/host/tcp-offload.c
@@ -0,0 +1,124 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+/* Kernel includes */
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+/* Driver includes */
+#include "tcp-offload.h"
+
+static LIST_HEAD(nvme_tcp_ofld_devices);
+static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
+
+/**
+ * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
+ * function.
+ * @dev:	NVMeTCP offload device instance to be registered to the
+ *		common tcp offload instance.
+ *
+ * API function that registers the type of vendor specific driver
+ * being implemented to the common NVMe over TCP offload library. Part of
+ * the overall init sequence of starting up an offload driver.
+ */
+int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev)
+{
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+
+	if (!ops->claim_dev ||
+	    !ops->setup_ctrl ||
+	    !ops->release_ctrl ||
+	    !ops->create_queue ||
+	    !ops->drain_queue ||
+	    !ops->destroy_queue ||
+	    !ops->poll_queue ||
+	    !ops->send_req)
+		return -EINVAL;
+
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_add_tail(&dev->entry, &nvme_tcp_ofld_devices);
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_register_dev);
+
+/**
+ * nvme_tcp_ofld_unregister_dev() - NVMeTCP Offload Library unregistration
+ * function.
+ * @dev:	NVMeTCP offload device instance to be unregistered from the
+ *		common tcp offload instance.
+ *
+ * API function that unregisters the type of vendor specific driver being
+ * implemented from the common NVMe over TCP offload library.
+ * Part of the overall exit sequence of unloading the implemented driver.
+ */
+void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev)
+{
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_del(&dev->entry);
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
+
+/**
+ * nvme_tcp_ofld_report_queue_err() - NVMeTCP Offload report error event
+ * callback function. Pointed to by nvme_tcp_ofld_queue->report_err.
+ * @queue:	NVMeTCP offload queue instance on which the error has occurred.
+ *
+ * API function that allows the vendor specific offload driver to reports errors
+ * to the common offload layer, to invoke error recovery.
+ */
+int nvme_tcp_ofld_report_queue_err(struct nvme_tcp_ofld_queue *queue)
+{
+	/* Placeholder - invoke error recovery flow */
+
+	return 0;
+}
+
+/**
+ * nvme_tcp_ofld_req_done() - NVMeTCP Offload request done callback
+ * function. Pointed to by nvme_tcp_ofld_req->done.
+ * Handles both NVME_TCP_F_DATA_SUCCESS flag and NVMe CQ.
+ * @req:	NVMeTCP offload request to complete.
+ * @result:     The nvme_result.
+ * @status:     The completion status.
+ *
+ * API function that allows the vendor specific offload driver to report request
+ * completions to the common offload layer.
+ */
+void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
+			    union nvme_result *result,
+			    __le16 status)
+{
+	/* Placeholder - complete request with/without error */
+}
+
+static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
+	.name		= "tcp_offload",
+	.module		= THIS_MODULE,
+	.required_opts	= NVMF_OPT_TRADDR,
+	.allowed_opts	= NVMF_OPT_TRSVCID | NVMF_OPT_NR_WRITE_QUEUES  |
+			  NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO |
+			  NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HDR_DIGEST |
+			  NVMF_OPT_DATA_DIGEST | NVMF_OPT_NR_POLL_QUEUES |
+			  NVMF_OPT_TOS,
+};
+
+static int __init nvme_tcp_ofld_init_module(void)
+{
+	nvmf_register_transport(&nvme_tcp_ofld_transport);
+
+	return 0;
+}
+
+static void __exit nvme_tcp_ofld_cleanup_module(void)
+{
+	nvmf_unregister_transport(&nvme_tcp_ofld_transport);
+}
+
+module_init(nvme_tcp_ofld_init_module);
+module_exit(nvme_tcp_ofld_cleanup_module);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
new file mode 100644
index 000000000000..520a0ea6f4b8
--- /dev/null
+++ b/drivers/nvme/host/tcp-offload.h
@@ -0,0 +1,199 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+/* Linux includes */
+#include <linux/dma-mapping.h>
+#include <linux/scatterlist.h>
+#include <linux/types.h>
+#include <linux/nvme-tcp.h>
+
+/* Driver includes */
+#include "nvme.h"
+#include "fabrics.h"
+
+/* Forward declarations */
+struct nvme_tcp_ofld_ops;
+
+/* Representation of a vendor-specific device. This is the struct used to
+ * register to the offload layer by the vendor-specific driver during its probe
+ * function.
+ * Allocated by vendor-specific driver.
+ */
+struct nvme_tcp_ofld_dev {
+	struct list_head entry;
+	struct net_device *ndev;
+	struct nvme_tcp_ofld_ops *ops;
+
+	/* Vendor specific driver context */
+	int num_hw_vectors;
+};
+
+/* Per IO struct holding the nvme_request and command
+ * Allocated by blk-mq.
+ */
+struct nvme_tcp_ofld_req {
+	struct nvme_request req;
+	struct nvme_command nvme_cmd;
+	struct list_head queue_entry;
+	struct nvme_tcp_ofld_queue *queue;
+
+	/* Vendor specific driver context */
+	void *private_data;
+
+	/* async flag is used to distinguish between async and IO flow
+	 * in common send_req() of nvme_tcp_ofld_ops.
+	 */
+	bool async;
+
+	void (*done)(struct nvme_tcp_ofld_req *req,
+		     union nvme_result *result,
+		     __le16 status);
+};
+
+enum nvme_tcp_ofld_queue_flags {
+	NVME_TCP_OFLD_Q_ALLOCATED = 0,
+	NVME_TCP_OFLD_Q_LIVE = 1,
+};
+
+/* Allocated by nvme_tcp_ofld */
+struct nvme_tcp_ofld_queue {
+	/* Offload device associated to this queue */
+	struct nvme_tcp_ofld_dev *dev;
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	unsigned long flags;
+	size_t cmnd_capsule_len;
+
+	u8 hdr_digest;
+	u8 data_digest;
+	u8 tos;
+
+	/* Vendor specific driver context */
+	void *private_data;
+
+	/* Error callback function */
+	int (*report_err)(struct nvme_tcp_ofld_queue *queue);
+};
+
+/* Connectivity (routing) params used for establishing a connection */
+struct nvme_tcp_ofld_ctrl_con_params {
+	struct sockaddr_storage remote_ip_addr;
+
+	/* If NVMF_OPT_HOST_TRADDR is provided it will be set in local_ip_addr
+	 * in nvme_tcp_ofld_create_ctrl().
+	 * If NVMF_OPT_HOST_TRADDR is not provided the local_ip_addr will be
+	 * initialized by claim_dev().
+	 */
+	struct sockaddr_storage local_ip_addr;
+};
+
+/* Allocated by nvme_tcp_ofld */
+struct nvme_tcp_ofld_ctrl {
+	struct nvme_ctrl nctrl;
+	struct list_head list;
+	struct nvme_tcp_ofld_dev *dev;
+
+	/* admin and IO queues */
+	struct blk_mq_tag_set tag_set;
+	struct blk_mq_tag_set admin_tag_set;
+	struct nvme_tcp_ofld_queue *queues;
+
+	struct work_struct err_work;
+	struct delayed_work connect_work;
+
+	/*
+	 * Each entry in the array indicates the number of queues of
+	 * corresponding type.
+	 */
+	u32 io_queues[HCTX_MAX_TYPES];
+
+	/* Connectivity params */
+	struct nvme_tcp_ofld_ctrl_con_params conn_params;
+
+	/* Vendor specific driver context */
+	void *private_data;
+};
+
+struct nvme_tcp_ofld_ops {
+	const char *name;
+	struct module *module;
+
+	/* For vendor-specific driver to report what opts it supports.
+	 * It could be different than the ULP supported opts due to hardware
+	 * limitations. Also it could be different among different vendor
+	 * drivers.
+	 */
+	int required_opts; /* bitmap using enum nvmf_parsing_opts */
+	int allowed_opts; /* bitmap using enum nvmf_parsing_opts */
+
+	/* For vendor-specific max num of segments and IO sizes */
+	u32 max_hw_sectors;
+	u32 max_segments;
+
+	/**
+	 * claim_dev: Return True if addr is reachable via offload device.
+	 * @dev: The offload device to check.
+	 * @ctrl: The offload ctrl have the conn_params field. The
+	 * conn_params is to be filled with routing params by the lower
+	 * driver.
+	 */
+	int (*claim_dev)(struct nvme_tcp_ofld_dev *dev,
+			 struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * setup_ctrl: Setup device specific controller structures.
+	 * @ctrl: The offload ctrl.
+	 */
+	int (*setup_ctrl)(struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * release_ctrl: Release/Free device specific controller structures.
+	 * @ctrl: The offload ctrl.
+	 */
+	int (*release_ctrl)(struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * create_queue: Create offload queue and establish TCP + NVMeTCP
+	 * (icreq+icresp) connection. Return true on successful connection.
+	 * Based on nvme_tcp_alloc_queue.
+	 * @queue: The queue itself - used as input and output.
+	 * @qid: The queue ID associated with the requested queue.
+	 * @q_size: The queue depth.
+	 */
+	int (*create_queue)(struct nvme_tcp_ofld_queue *queue, int qid,
+			    size_t queue_size);
+
+	/**
+	 * drain_queue: Drain a given queue - blocking function call.
+	 * Return from this function ensures that no additional
+	 * completions will arrive on this queue and that the HW will
+	 * not access host memory.
+	 * @queue: The queue to drain.
+	 */
+	void (*drain_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * destroy_queue: Close the TCP + NVMeTCP connection of a given queue
+	 * and make sure its no longer active (no completions will arrive on the
+	 * queue).
+	 * @queue: The queue to destroy.
+	 */
+	void (*destroy_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * poll_queue: Poll a given queue for completions.
+	 * @queue: The queue to poll.
+	 */
+	int (*poll_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * send_req: Dispatch a request. Returns the execution status.
+	 * @req: Ptr to request to be sent.
+	 */
+	int (*send_req)(struct nvme_tcp_ofld_req *req);
+};
+
+/* Exported functions for lower vendor specific offload drivers */
+int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
+void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 11:42:41 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 21:42:41 +0300
Subject: [PATCH 3/8] nvme-fabrics: Expose nvmf_check_required_opts() globally
In-Reply-To: <20210602184246.14184-1-smalin@marvell.com>
References: <20210602184246.14184-1-smalin@marvell.com>
Message-ID: <20210602184246.14184-4-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

nvmf_check_required_opts() is used to check if user provided opts has
the required_opts or not. if not, it will log which options are not
provided.

It can be leveraged by nvme-tcp-offload to check if provided opts are
supported by this specific vendor driver or not.

So expose nvmf_check_required_opts() globally.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/fabrics.c | 5 +++--
 drivers/nvme/host/fabrics.h | 2 ++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index e8f6b8f9fd35..b4d5d46dae62 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -865,8 +865,8 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	return ret;
 }
 
-static int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
-		unsigned int required_opts)
+int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
+			     unsigned int required_opts)
 {
 	if ((opts->mask & required_opts) != required_opts) {
 		int i;
@@ -884,6 +884,7 @@ static int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(nvmf_check_required_opts);
 
 bool nvmf_ip_options_match(struct nvme_ctrl *ctrl,
 		struct nvmf_ctrl_options *opts)
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index ce7fe3a842b1..8399fcc063ef 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -193,5 +193,7 @@ int nvmf_get_address(struct nvme_ctrl *ctrl, char *buf, int size);
 bool nvmf_should_reconnect(struct nvme_ctrl *ctrl);
 bool nvmf_ip_options_match(struct nvme_ctrl *ctrl,
 		struct nvmf_ctrl_options *opts);
+int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
+			     unsigned int required_opts);
 
 #endif /* _NVME_FABRICS_H */
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 11:42:42 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 21:42:42 +0300
Subject: [PATCH 4/8] nvme-tcp-offload: Add device scan implementation
In-Reply-To: <20210602184246.14184-1-smalin@marvell.com>
References: <20210602184246.14184-1-smalin@marvell.com>
Message-ID: <20210602184246.14184-5-smalin@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

As part of create_ctrl(), it scans the registered devices and calls
the claim_dev op on each of them, to find the first devices that matches
the connection params. Once the correct devices is found (claim_dev
returns true), we raise the refcnt of that device and return that device
as the device to be used for ctrl currently being created.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/tcp-offload.c | 77 +++++++++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index f7aa49f337dc..5ee3bbca8770 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -13,6 +13,11 @@
 static LIST_HEAD(nvme_tcp_ofld_devices);
 static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
 
+static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nctrl)
+{
+	return container_of(nctrl, struct nvme_tcp_ofld_ctrl, nctrl);
+}
+
 /**
  * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
  * function.
@@ -96,6 +101,77 @@ void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
 	/* Placeholder - complete request with/without error */
 }
 
+static struct nvme_tcp_ofld_dev *
+nvme_tcp_ofld_lookup_dev(struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	struct nvme_tcp_ofld_dev *dev;
+
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_for_each_entry(dev, &nvme_tcp_ofld_devices, entry) {
+		if (dev->ops->claim_dev(dev, ctrl))
+			goto out;
+	}
+
+	dev = NULL;
+out:
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+
+	return dev;
+}
+
+static struct nvme_ctrl *
+nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	struct nvme_tcp_ofld_dev *dev;
+	struct nvme_ctrl *nctrl;
+	int rc = 0;
+
+	ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
+	if (!ctrl)
+		return ERR_PTR(-ENOMEM);
+
+	nctrl = &ctrl->nctrl;
+
+	/* Init nvme_tcp_ofld_ctrl and nvme_ctrl params based on received opts */
+
+	/* Find device that can reach the dest addr */
+	dev = nvme_tcp_ofld_lookup_dev(ctrl);
+	if (!dev) {
+		pr_info("no device found for addr %s:%s.\n",
+			opts->traddr, opts->trsvcid);
+		rc = -EINVAL;
+		goto out_free_ctrl;
+	}
+
+	/* Increase driver refcnt */
+	if (!try_module_get(dev->ops->module)) {
+		pr_err("try_module_get failed\n");
+		dev = NULL;
+		goto out_free_ctrl;
+	}
+
+	ctrl->dev = dev;
+
+	if (ctrl->dev->ops->max_hw_sectors)
+		nctrl->max_hw_sectors = ctrl->dev->ops->max_hw_sectors;
+	if (ctrl->dev->ops->max_segments)
+		nctrl->max_segments = ctrl->dev->ops->max_segments;
+
+	/* Init queues */
+
+	/* Call nvme_init_ctrl */
+
+	/* Setup ctrl */
+
+	return nctrl;
+
+out_free_ctrl:
+	kfree(ctrl);
+
+	return ERR_PTR(rc);
+}
+
 static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
 	.name		= "tcp_offload",
 	.module		= THIS_MODULE,
@@ -105,6 +181,7 @@ static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
 			  NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HDR_DIGEST |
 			  NVMF_OPT_DATA_DIGEST | NVMF_OPT_NR_POLL_QUEUES |
 			  NVMF_OPT_TOS,
+	.create_ctrl	= nvme_tcp_ofld_create_ctrl,
 };
 
 static int __init nvme_tcp_ofld_init_module(void)
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 11:42:43 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 21:42:43 +0300
Subject: [PATCH 5/8] nvme-tcp-offload: Add controller level implementation
In-Reply-To: <20210602184246.14184-1-smalin@marvell.com>
References: <20210602184246.14184-1-smalin@marvell.com>
Message-ID: <20210602184246.14184-6-smalin@marvell.com>

From: Arie Gershberg <agershberg at marvell.com>

In this patch we implement controller level functionality including:
- create_ctrl.
- delete_ctrl.
- free_ctrl.

The implementation is similar to other nvme fabrics modules, the main
difference being that the nvme-tcp-offload ULP calls the vendor specific
claim_dev() op with the given TCP/IP parameters to determine which device
will be used for this controller.
Once found, the vendor specific device and controller will be paired and
kept in a controller list managed by the ULP.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 drivers/nvme/host/tcp-offload.c | 481 +++++++++++++++++++++++++++++++-
 1 file changed, 476 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 5ee3bbca8770..d05dec9c0add 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -12,6 +12,10 @@
 
 static LIST_HEAD(nvme_tcp_ofld_devices);
 static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
+static LIST_HEAD(nvme_tcp_ofld_ctrl_list);
+static DEFINE_MUTEX(nvme_tcp_ofld_ctrl_mutex);
+static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops;
+static struct blk_mq_ops nvme_tcp_ofld_mq_ops;
 
 static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nctrl)
 {
@@ -119,21 +123,439 @@ nvme_tcp_ofld_lookup_dev(struct nvme_tcp_ofld_ctrl *ctrl)
 	return dev;
 }
 
+static struct blk_mq_tag_set *
+nvme_tcp_ofld_alloc_tagset(struct nvme_ctrl *nctrl, bool admin)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct blk_mq_tag_set *set;
+	int rc;
+
+	if (admin) {
+		set = &ctrl->admin_tag_set;
+		memset(set, 0, sizeof(*set));
+		set->ops = &nvme_tcp_ofld_admin_mq_ops;
+		set->queue_depth = NVME_AQ_MQ_TAG_DEPTH;
+		set->reserved_tags = NVMF_RESERVED_TAGS;
+		set->numa_node = nctrl->numa_node;
+		set->flags = BLK_MQ_F_BLOCKING;
+		set->cmd_size = sizeof(struct nvme_tcp_ofld_req);
+		set->driver_data = ctrl;
+		set->nr_hw_queues = 1;
+		set->timeout = NVME_ADMIN_TIMEOUT;
+	} else {
+		set = &ctrl->tag_set;
+		memset(set, 0, sizeof(*set));
+		set->ops = &nvme_tcp_ofld_mq_ops;
+		set->queue_depth = nctrl->sqsize + 1;
+		set->reserved_tags = NVMF_RESERVED_TAGS;
+		set->numa_node = nctrl->numa_node;
+		set->flags = BLK_MQ_F_SHOULD_MERGE;
+		set->cmd_size = sizeof(struct nvme_tcp_ofld_req);
+		set->driver_data = ctrl;
+		set->nr_hw_queues = nctrl->queue_count - 1;
+		set->timeout = NVME_IO_TIMEOUT;
+		set->nr_maps = nctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2;
+	}
+
+	rc = blk_mq_alloc_tag_set(set);
+	if (rc)
+		return ERR_PTR(rc);
+
+	return set;
+}
+
+static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
+					       bool new)
+{
+	int rc;
+
+	/* Placeholder - alloc_admin_queue */
+	if (new) {
+		nctrl->admin_tagset =
+				nvme_tcp_ofld_alloc_tagset(nctrl, true);
+		if (IS_ERR(nctrl->admin_tagset)) {
+			rc = PTR_ERR(nctrl->admin_tagset);
+			nctrl->admin_tagset = NULL;
+			goto out_destroy_queue;
+		}
+
+		nctrl->fabrics_q = blk_mq_init_queue(nctrl->admin_tagset);
+		if (IS_ERR(nctrl->fabrics_q)) {
+			rc = PTR_ERR(nctrl->fabrics_q);
+			nctrl->fabrics_q = NULL;
+			goto out_free_tagset;
+		}
+
+		nctrl->admin_q = blk_mq_init_queue(nctrl->admin_tagset);
+		if (IS_ERR(nctrl->admin_q)) {
+			rc = PTR_ERR(nctrl->admin_q);
+			nctrl->admin_q = NULL;
+			goto out_cleanup_fabrics_q;
+		}
+	}
+
+	/* Placeholder - nvme_tcp_ofld_start_queue */
+
+	rc = nvme_enable_ctrl(nctrl);
+	if (rc)
+		goto out_stop_queue;
+
+	blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	rc = nvme_init_ctrl_finish(nctrl);
+	if (rc)
+		goto out_quiesce_queue;
+
+	return 0;
+
+out_quiesce_queue:
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+
+out_stop_queue:
+	/* Placeholder - stop offload queue */
+	nvme_cancel_admin_tagset(nctrl);
+
+out_cleanup_fabrics_q:
+	if (new)
+		blk_cleanup_queue(nctrl->fabrics_q);
+out_free_tagset:
+	if (new)
+		blk_mq_free_tag_set(nctrl->admin_tagset);
+out_destroy_queue:
+	/* Placeholder - free admin queue */
+
+	return rc;
+}
+
+static int
+nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
+{
+	int rc;
+
+	/* Placeholder - alloc_io_queues */
+
+	if (new) {
+		nctrl->tagset = nvme_tcp_ofld_alloc_tagset(nctrl, false);
+		if (IS_ERR(nctrl->tagset)) {
+			rc = PTR_ERR(nctrl->tagset);
+			nctrl->tagset = NULL;
+			goto out_free_io_queues;
+		}
+
+		nctrl->connect_q = blk_mq_init_queue(nctrl->tagset);
+		if (IS_ERR(nctrl->connect_q)) {
+			rc = PTR_ERR(nctrl->connect_q);
+			nctrl->connect_q = NULL;
+			goto out_free_tag_set;
+		}
+	}
+
+	/* Placeholder - start_io_queues */
+
+	if (!new) {
+		nvme_start_queues(nctrl);
+		if (!nvme_wait_freeze_timeout(nctrl, NVME_IO_TIMEOUT)) {
+			/*
+			 * If we timed out waiting for freeze we are likely to
+			 * be stuck.  Fail the controller initialization just
+			 * to be safe.
+			 */
+			rc = -ENODEV;
+			goto out_wait_freeze_timed_out;
+		}
+		blk_mq_update_nr_hw_queues(nctrl->tagset, nctrl->queue_count - 1);
+		nvme_unfreeze(nctrl);
+	}
+
+	return 0;
+
+out_wait_freeze_timed_out:
+	nvme_stop_queues(nctrl);
+	nvme_sync_io_queues(nctrl);
+
+	/* Placeholder - Stop IO queues */
+
+	if (new)
+		blk_cleanup_queue(nctrl->connect_q);
+out_free_tag_set:
+	if (new)
+		blk_mq_free_tag_set(nctrl->tagset);
+out_free_io_queues:
+	/* Placeholder - free_io_queues */
+
+	return rc;
+}
+
+static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvmf_ctrl_options *opts = nctrl->opts;
+	int rc = 0;
+
+	rc = ctrl->dev->ops->setup_ctrl(ctrl);
+	if (rc)
+		return rc;
+
+	rc = nvme_tcp_ofld_configure_admin_queue(nctrl, new);
+	if (rc)
+		goto out_release_ctrl;
+
+	if (nctrl->icdoff) {
+		dev_err(nctrl->device, "icdoff is not supported!\n");
+		rc = -EINVAL;
+		goto destroy_admin;
+	}
+
+	if (!(nctrl->sgls & ((1 << 0) | (1 << 1)))) {
+		dev_err(nctrl->device, "Mandatory sgls are not supported!\n");
+		goto destroy_admin;
+	}
+
+	if (opts->queue_size > nctrl->sqsize + 1)
+		dev_warn(nctrl->device,
+			 "queue_size %zu > ctrl sqsize %u, clamping down\n",
+			 opts->queue_size, nctrl->sqsize + 1);
+
+	if (nctrl->sqsize + 1 > nctrl->maxcmd) {
+		dev_warn(nctrl->device,
+			 "sqsize %u > ctrl maxcmd %u, clamping down\n",
+			 nctrl->sqsize + 1, nctrl->maxcmd);
+		nctrl->sqsize = nctrl->maxcmd - 1;
+	}
+
+	if (nctrl->queue_count > 1) {
+		rc = nvme_tcp_ofld_configure_io_queues(nctrl, new);
+		if (rc)
+			goto destroy_admin;
+	}
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_LIVE)) {
+		/*
+		 * state change failure is ok if we started ctrl delete,
+		 * unless we're during creation of a new controller to
+		 * avoid races with teardown flow.
+		 */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+		WARN_ON_ONCE(new);
+		rc = -EINVAL;
+		goto destroy_io;
+	}
+
+	nvme_start_ctrl(nctrl);
+
+	return 0;
+
+destroy_io:
+	/* Placeholder - stop and destroy io queues*/
+destroy_admin:
+	/* Placeholder - stop and destroy admin queue*/
+out_release_ctrl:
+	ctrl->dev->ops->release_ctrl(ctrl);
+
+	return rc;
+}
+
+static int
+nvme_tcp_ofld_check_dev_opts(struct nvmf_ctrl_options *opts,
+			     struct nvme_tcp_ofld_ops *ofld_ops)
+{
+	unsigned int nvme_tcp_ofld_opt_mask = NVMF_ALLOWED_OPTS |
+			ofld_ops->allowed_opts | ofld_ops->required_opts;
+	struct nvmf_ctrl_options dev_opts_mask;
+
+	if (opts->mask & ~nvme_tcp_ofld_opt_mask) {
+		pr_warn("One or more nvmf options missing from ofld drvr %s.\n",
+			ofld_ops->name);
+
+		dev_opts_mask.mask = nvme_tcp_ofld_opt_mask;
+
+		return nvmf_check_required_opts(&dev_opts_mask, opts->mask);
+	}
+
+	return 0;
+}
+
+static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+
+	if (list_empty(&ctrl->list))
+		goto free_ctrl;
+
+	ctrl->dev->ops->release_ctrl(ctrl);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_del(&ctrl->list);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+
+	nvmf_free_options(nctrl->opts);
+free_ctrl:
+	module_put(dev->ops->module);
+	kfree(ctrl->queues);
+	kfree(ctrl);
+}
+
+static void
+nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove)
+{
+	/* Placeholder - teardown_admin_queue */
+}
+
+static void
+nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
+{
+	/* Placeholder - teardown_io_queues */
+}
+
+static void
+nvme_tcp_ofld_teardown_ctrl(struct nvme_ctrl *nctrl, bool shutdown)
+{
+	/* Placeholder - err_work and connect_work */
+	nvme_tcp_ofld_teardown_io_queues(nctrl, shutdown);
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	if (shutdown)
+		nvme_shutdown_ctrl(nctrl);
+	else
+		nvme_disable_ctrl(nctrl);
+	nvme_tcp_ofld_teardown_admin_queue(nctrl, shutdown);
+}
+
+static void nvme_tcp_ofld_delete_ctrl(struct nvme_ctrl *nctrl)
+{
+	nvme_tcp_ofld_teardown_ctrl(nctrl, true);
+}
+
+static int
+nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
+			   struct request *rq,
+			   unsigned int hctx_idx,
+			   unsigned int numa_node)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+
+	/* Placeholder - init request */
+
+	req->done = nvme_tcp_ofld_req_done;
+
+	return 0;
+}
+
+static blk_status_t
+nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
+		       const struct blk_mq_queue_data *bd)
+{
+	/* Call nvme_setup_cmd(...) */
+
+	/* Call ops->send_req(...) */
+
+	return BLK_STS_OK;
+}
+
+static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
+	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.init_request	= nvme_tcp_ofld_init_request,
+	/*
+	 * All additional ops will be also implemented and registered similar to
+	 * tcp.c
+	 */
+};
+
+static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
+	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.init_request	= nvme_tcp_ofld_init_request,
+	/*
+	 * All additional ops will be also implemented and registered similar to
+	 * tcp.c
+	 */
+};
+
+static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
+	.name			= "tcp_offload",
+	.module			= THIS_MODULE,
+	.flags			= NVME_F_FABRICS,
+	.reg_read32		= nvmf_reg_read32,
+	.reg_read64		= nvmf_reg_read64,
+	.reg_write32		= nvmf_reg_write32,
+	.free_ctrl		= nvme_tcp_ofld_free_ctrl,
+	.delete_ctrl		= nvme_tcp_ofld_delete_ctrl,
+	.get_address		= nvmf_get_address,
+};
+
+static bool
+nvme_tcp_ofld_existing_controller(struct nvmf_ctrl_options *opts)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	bool found = false;
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_for_each_entry(ctrl, &nvme_tcp_ofld_ctrl_list, list) {
+		found = nvmf_ip_options_match(&ctrl->nctrl, opts);
+		if (found)
+			break;
+	}
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+
+	return found;
+}
+
 static struct nvme_ctrl *
 nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 {
+	struct nvme_tcp_ofld_queue *queue;
 	struct nvme_tcp_ofld_ctrl *ctrl;
 	struct nvme_tcp_ofld_dev *dev;
 	struct nvme_ctrl *nctrl;
-	int rc = 0;
+	int i, rc = 0;
 
 	ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
 	if (!ctrl)
 		return ERR_PTR(-ENOMEM);
 
+	INIT_LIST_HEAD(&ctrl->list);
 	nctrl = &ctrl->nctrl;
+	nctrl->opts = opts;
+	nctrl->queue_count = opts->nr_io_queues + opts->nr_write_queues +
+			     opts->nr_poll_queues + 1;
+	nctrl->sqsize = opts->queue_size - 1;
+	nctrl->kato = opts->kato;
+	if (!(opts->mask & NVMF_OPT_TRSVCID)) {
+		opts->trsvcid =
+			kstrdup(__stringify(NVME_TCP_DISC_PORT), GFP_KERNEL);
+		if (!opts->trsvcid) {
+			rc = -ENOMEM;
+			goto out_free_ctrl;
+		}
+		opts->mask |= NVMF_OPT_TRSVCID;
+	}
+
+	rc = inet_pton_with_scope(&init_net, AF_UNSPEC, opts->traddr,
+				  opts->trsvcid,
+				  &ctrl->conn_params.remote_ip_addr);
+	if (rc) {
+		pr_err("malformed address passed: %s:%s\n",
+		       opts->traddr, opts->trsvcid);
+		goto out_free_ctrl;
+	}
+
+	if (opts->mask & NVMF_OPT_HOST_TRADDR) {
+		rc = inet_pton_with_scope(&init_net, AF_UNSPEC,
+					  opts->host_traddr, NULL,
+					  &ctrl->conn_params.local_ip_addr);
+		if (rc) {
+			pr_err("malformed src address passed: %s\n",
+			       opts->host_traddr);
+			goto out_free_ctrl;
+		}
+	}
 
-	/* Init nvme_tcp_ofld_ctrl and nvme_ctrl params based on received opts */
+	if (!opts->duplicate_connect &&
+	    nvme_tcp_ofld_existing_controller(opts)) {
+		rc = -EALREADY;
+		goto out_free_ctrl;
+	}
 
 	/* Find device that can reach the dest addr */
 	dev = nvme_tcp_ofld_lookup_dev(ctrl);
@@ -151,6 +573,10 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 		goto out_free_ctrl;
 	}
 
+	rc = nvme_tcp_ofld_check_dev_opts(opts, dev->ops);
+	if (rc)
+		goto out_module_put;
+
 	ctrl->dev = dev;
 
 	if (ctrl->dev->ops->max_hw_sectors)
@@ -158,14 +584,51 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 	if (ctrl->dev->ops->max_segments)
 		nctrl->max_segments = ctrl->dev->ops->max_segments;
 
-	/* Init queues */
+	ctrl->queues = kcalloc(nctrl->queue_count,
+			       sizeof(struct nvme_tcp_ofld_queue),
+			       GFP_KERNEL);
+	if (!ctrl->queues) {
+		rc = -ENOMEM;
+		goto out_module_put;
+	}
+
+	for (i = 0; i < nctrl->queue_count; ++i) {
+		queue = &ctrl->queues[i];
+		queue->ctrl = ctrl;
+		queue->dev = dev;
+		queue->report_err = nvme_tcp_ofld_report_queue_err;
+	}
+
+	rc = nvme_init_ctrl(nctrl, ndev, &nvme_tcp_ofld_ctrl_ops, 0);
+	if (rc)
+		goto out_free_queues;
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		WARN_ON_ONCE(1);
+		rc = -EINTR;
+		goto out_uninit_ctrl;
+	}
 
-	/* Call nvme_init_ctrl */
+	rc = nvme_tcp_ofld_setup_ctrl(nctrl, true);
+	if (rc)
+		goto out_uninit_ctrl;
 
-	/* Setup ctrl */
+	dev_info(nctrl->device, "new ctrl: NQN \"%s\", addr %pISp\n",
+		 opts->subsysnqn, &ctrl->conn_params.remote_ip_addr);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_add_tail(&ctrl->list, &nvme_tcp_ofld_ctrl_list);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
 
 	return nctrl;
 
+out_uninit_ctrl:
+	nvme_uninit_ctrl(nctrl);
+	nvme_put_ctrl(nctrl);
+out_free_queues:
+	kfree(ctrl->queues);
+out_module_put:
+	module_put(dev->ops->module);
 out_free_ctrl:
 	kfree(ctrl);
 
@@ -193,7 +656,15 @@ static int __init nvme_tcp_ofld_init_module(void)
 
 static void __exit nvme_tcp_ofld_cleanup_module(void)
 {
+	struct nvme_tcp_ofld_ctrl *ctrl;
+
 	nvmf_unregister_transport(&nvme_tcp_ofld_transport);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_for_each_entry(ctrl, &nvme_tcp_ofld_ctrl_list, list)
+		nvme_delete_ctrl(&ctrl->nctrl);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+	flush_workqueue(nvme_delete_wq);
 }
 
 module_init(nvme_tcp_ofld_init_module);
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 11:42:44 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 21:42:44 +0300
Subject: [PATCH 6/8] nvme-tcp-offload: Add controller level error recovery
 implementation
In-Reply-To: <20210602184246.14184-1-smalin@marvell.com>
References: <20210602184246.14184-1-smalin@marvell.com>
Message-ID: <20210602184246.14184-7-smalin@marvell.com>

From: Arie Gershberg <agershberg at marvell.com>

In this patch, we implement controller level error handling and recovery.
Upon an error discovered by the ULP or reset controller initiated by the
nvme-core (using reset_ctrl workqueue), the ULP will initiate a controller
recovery which includes teardown and re-connect of all queues.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 drivers/nvme/host/tcp-offload.c | 127 +++++++++++++++++++++++++++++++-
 drivers/nvme/host/tcp-offload.h |   1 +
 2 files changed, 126 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index d05dec9c0add..97c1fd33adb9 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -72,6 +72,23 @@ void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev)
 }
 EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
 
+/**
+ * nvme_tcp_ofld_error_recovery() - NVMeTCP Offload library error recovery.
+ * function.
+ * @nctrl:	NVMe controller instance to change to resetting.
+ *
+ * API function that change the controller state to resseting.
+ * Part of the overall controller reset sequence.
+ */
+void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl)
+{
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_RESETTING))
+		return;
+
+	queue_work(nvme_reset_wq, &to_tcp_ofld_ctrl(nctrl)->err_work);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_error_recovery);
+
 /**
  * nvme_tcp_ofld_report_queue_err() - NVMeTCP Offload report error event
  * callback function. Pointed to by nvme_tcp_ofld_queue->report_err.
@@ -82,7 +99,8 @@ EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
  */
 int nvme_tcp_ofld_report_queue_err(struct nvme_tcp_ofld_queue *queue)
 {
-	/* Placeholder - invoke error recovery flow */
+	pr_err("nvme-tcp-offload queue error\n");
+	nvme_tcp_ofld_error_recovery(&queue->ctrl->nctrl);
 
 	return 0;
 }
@@ -287,6 +305,28 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 	return rc;
 }
 
+static void nvme_tcp_ofld_reconnect_or_remove(struct nvme_ctrl *nctrl)
+{
+	/* If we are resetting/deleting then do nothing */
+	if (nctrl->state != NVME_CTRL_CONNECTING) {
+		WARN_ON_ONCE(nctrl->state == NVME_CTRL_NEW ||
+			     nctrl->state == NVME_CTRL_LIVE);
+
+		return;
+	}
+
+	if (nvmf_should_reconnect(nctrl)) {
+		dev_info(nctrl->device, "Reconnecting in %d seconds...\n",
+			 nctrl->opts->reconnect_delay);
+		queue_delayed_work(nvme_wq,
+				   &to_tcp_ofld_ctrl(nctrl)->connect_work,
+				   nctrl->opts->reconnect_delay * HZ);
+	} else {
+		dev_info(nctrl->device, "Removing controller...\n");
+		nvme_delete_ctrl(nctrl);
+	}
+}
+
 static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 {
 	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
@@ -410,10 +450,63 @@ nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
 	/* Placeholder - teardown_io_queues */
 }
 
+static void nvme_tcp_ofld_reconnect_ctrl_work(struct work_struct *work)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl =
+				container_of(to_delayed_work(work),
+					     struct nvme_tcp_ofld_ctrl,
+					     connect_work);
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+
+	++nctrl->nr_reconnects;
+
+	if (nvme_tcp_ofld_setup_ctrl(nctrl, false))
+		goto requeue;
+
+	dev_info(nctrl->device, "Successfully reconnected (%d attempt)\n",
+		 nctrl->nr_reconnects);
+
+	nctrl->nr_reconnects = 0;
+
+	return;
+
+requeue:
+	dev_info(nctrl->device, "Failed reconnect attempt %d\n",
+		 nctrl->nr_reconnects);
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
+static void nvme_tcp_ofld_error_recovery_work(struct work_struct *work)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl =
+		container_of(work, struct nvme_tcp_ofld_ctrl, err_work);
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+
+	nvme_stop_keep_alive(nctrl);
+	nvme_tcp_ofld_teardown_io_queues(nctrl, false);
+	/* unquiesce to fail fast pending requests */
+	nvme_start_queues(nctrl);
+	nvme_tcp_ofld_teardown_admin_queue(nctrl, false);
+	blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		/* state change failure is ok if we started nctrl delete */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+
+		return;
+	}
+
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
 static void
 nvme_tcp_ofld_teardown_ctrl(struct nvme_ctrl *nctrl, bool shutdown)
 {
-	/* Placeholder - err_work and connect_work */
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+
+	cancel_work_sync(&ctrl->err_work);
+	cancel_delayed_work_sync(&ctrl->connect_work);
 	nvme_tcp_ofld_teardown_io_queues(nctrl, shutdown);
 	blk_mq_quiesce_queue(nctrl->admin_q);
 	if (shutdown)
@@ -428,6 +521,32 @@ static void nvme_tcp_ofld_delete_ctrl(struct nvme_ctrl *nctrl)
 	nvme_tcp_ofld_teardown_ctrl(nctrl, true);
 }
 
+static void nvme_tcp_ofld_reset_ctrl_work(struct work_struct *work)
+{
+	struct nvme_ctrl *nctrl =
+		container_of(work, struct nvme_ctrl, reset_work);
+
+	nvme_stop_ctrl(nctrl);
+	nvme_tcp_ofld_teardown_ctrl(nctrl, false);
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		/* state change failure is ok if we started ctrl delete */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+
+		return;
+	}
+
+	if (nvme_tcp_ofld_setup_ctrl(nctrl, false))
+		goto out_fail;
+
+	return;
+
+out_fail:
+	++nctrl->nr_reconnects;
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
 static int
 nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 			   struct request *rq,
@@ -521,6 +640,10 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 			     opts->nr_poll_queues + 1;
 	nctrl->sqsize = opts->queue_size - 1;
 	nctrl->kato = opts->kato;
+	INIT_DELAYED_WORK(&ctrl->connect_work,
+			  nvme_tcp_ofld_reconnect_ctrl_work);
+	INIT_WORK(&ctrl->err_work, nvme_tcp_ofld_error_recovery_work);
+	INIT_WORK(&nctrl->reset_work, nvme_tcp_ofld_reset_ctrl_work);
 	if (!(opts->mask & NVMF_OPT_TRSVCID)) {
 		opts->trsvcid =
 			kstrdup(__stringify(NVME_TCP_DISC_PORT), GFP_KERNEL);
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index 520a0ea6f4b8..4281d1dacc94 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -197,3 +197,4 @@ struct nvme_tcp_ofld_ops {
 /* Exported functions for lower vendor specific offload drivers */
 int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
+void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl);
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 11:42:45 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 21:42:45 +0300
Subject: [PATCH 7/8] nvme-tcp-offload: Add queue level implementation
In-Reply-To: <20210602184246.14184-1-smalin@marvell.com>
References: <20210602184246.14184-1-smalin@marvell.com>
Message-ID: <20210602184246.14184-8-smalin@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

In this patch we implement queue level functionality.
The implementation is similar to the nvme-tcp module, the main
difference being that we call the vendor specific create_queue op which
creates the TCP connection, and NVMeTPC connection including
icreq+icresp negotiation.
Once create_queue returns successfully, we can move on to the fabrics
connect.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 drivers/nvme/host/tcp-offload.c | 417 +++++++++++++++++++++++++++++---
 drivers/nvme/host/tcp-offload.h |   4 +
 2 files changed, 393 insertions(+), 28 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 97c1fd33adb9..1700cdf42433 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -22,6 +22,11 @@ static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nctr
 	return container_of(nctrl, struct nvme_tcp_ofld_ctrl, nctrl);
 }
 
+static inline int nvme_tcp_ofld_qid(struct nvme_tcp_ofld_queue *queue)
+{
+	return queue - queue->ctrl->queues;
+}
+
 /**
  * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
  * function.
@@ -182,19 +187,124 @@ nvme_tcp_ofld_alloc_tagset(struct nvme_ctrl *nctrl, bool admin)
 	return set;
 }
 
+static void __nvme_tcp_ofld_stop_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	queue->dev->ops->drain_queue(queue);
+}
+
+static void nvme_tcp_ofld_stop_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+
+	mutex_lock(&queue->queue_lock);
+	if (test_and_clear_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags))
+		__nvme_tcp_ofld_stop_queue(queue);
+	mutex_unlock(&queue->queue_lock);
+}
+
+static void nvme_tcp_ofld_stop_io_queues(struct nvme_ctrl *ctrl)
+{
+	int i;
+
+	for (i = 1; i < ctrl->queue_count; i++)
+		nvme_tcp_ofld_stop_queue(ctrl, i);
+}
+
+static void __nvme_tcp_ofld_free_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	queue->dev->ops->destroy_queue(queue);
+}
+
+static void nvme_tcp_ofld_free_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+
+	if (test_and_clear_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags)) {
+		__nvme_tcp_ofld_free_queue(queue);
+		mutex_destroy(&queue->queue_lock);
+	}
+}
+
+static void
+nvme_tcp_ofld_free_io_queues(struct nvme_ctrl *nctrl)
+{
+	int i;
+
+	for (i = 1; i < nctrl->queue_count; i++)
+		nvme_tcp_ofld_free_queue(nctrl, i);
+}
+
+static void nvme_tcp_ofld_destroy_io_queues(struct nvme_ctrl *nctrl, bool remove)
+{
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+	if (remove) {
+		blk_cleanup_queue(nctrl->connect_q);
+		blk_mq_free_tag_set(nctrl->tagset);
+	}
+	nvme_tcp_ofld_free_io_queues(nctrl);
+}
+
+static void nvme_tcp_ofld_destroy_admin_queue(struct nvme_ctrl *nctrl, bool remove)
+{
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	if (remove) {
+		blk_cleanup_queue(nctrl->admin_q);
+		blk_cleanup_queue(nctrl->fabrics_q);
+		blk_mq_free_tag_set(nctrl->admin_tagset);
+	}
+	nvme_tcp_ofld_free_queue(nctrl, 0);
+}
+
+static int nvme_tcp_ofld_start_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+	int rc;
+
+	queue = &ctrl->queues[qid];
+	if (qid) {
+		queue->cmnd_capsule_len = nctrl->ioccsz * 16;
+		rc = nvmf_connect_io_queue(nctrl, qid, false);
+	} else {
+		queue->cmnd_capsule_len = sizeof(struct nvme_command) + NVME_TCP_ADMIN_CCSZ;
+		rc = nvmf_connect_admin_queue(nctrl);
+	}
+
+	if (!rc) {
+		set_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags);
+	} else {
+		if (test_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags))
+			__nvme_tcp_ofld_stop_queue(queue);
+		dev_err(nctrl->device,
+			"failed to connect queue: %d ret=%d\n", qid, rc);
+	}
+
+	return rc;
+}
+
 static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 					       bool new)
 {
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[0];
 	int rc;
 
-	/* Placeholder - alloc_admin_queue */
+	mutex_init(&queue->queue_lock);
+
+	rc = ctrl->dev->ops->create_queue(queue, 0, NVME_AQ_DEPTH);
+	if (rc)
+		return rc;
+
+	set_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags);
 	if (new) {
 		nctrl->admin_tagset =
 				nvme_tcp_ofld_alloc_tagset(nctrl, true);
 		if (IS_ERR(nctrl->admin_tagset)) {
 			rc = PTR_ERR(nctrl->admin_tagset);
 			nctrl->admin_tagset = NULL;
-			goto out_destroy_queue;
+			goto out_free_queue;
 		}
 
 		nctrl->fabrics_q = blk_mq_init_queue(nctrl->admin_tagset);
@@ -212,7 +322,9 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 		}
 	}
 
-	/* Placeholder - nvme_tcp_ofld_start_queue */
+	rc = nvme_tcp_ofld_start_queue(nctrl, 0);
+	if (rc)
+		goto out_cleanup_queue;
 
 	rc = nvme_enable_ctrl(nctrl);
 	if (rc)
@@ -229,19 +341,143 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 out_quiesce_queue:
 	blk_mq_quiesce_queue(nctrl->admin_q);
 	blk_sync_queue(nctrl->admin_q);
-
 out_stop_queue:
-	/* Placeholder - stop offload queue */
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
 	nvme_cancel_admin_tagset(nctrl);
-
+out_cleanup_queue:
+	if (new)
+		blk_cleanup_queue(nctrl->admin_q);
 out_cleanup_fabrics_q:
 	if (new)
 		blk_cleanup_queue(nctrl->fabrics_q);
 out_free_tagset:
 	if (new)
 		blk_mq_free_tag_set(nctrl->admin_tagset);
-out_destroy_queue:
-	/* Placeholder - free admin queue */
+out_free_queue:
+	nvme_tcp_ofld_free_queue(nctrl, 0);
+
+	return rc;
+}
+
+static unsigned int nvme_tcp_ofld_nr_io_queues(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+	u32 hw_vectors = dev->num_hw_vectors;
+	u32 nr_write_queues, nr_poll_queues;
+	u32 nr_io_queues, nr_total_queues;
+
+	nr_io_queues = min3(nctrl->opts->nr_io_queues, num_online_cpus(),
+			    hw_vectors);
+	nr_write_queues = min3(nctrl->opts->nr_write_queues, num_online_cpus(),
+			       hw_vectors);
+	nr_poll_queues = min3(nctrl->opts->nr_poll_queues, num_online_cpus(),
+			      hw_vectors);
+
+	nr_total_queues = nr_io_queues + nr_write_queues + nr_poll_queues;
+
+	return nr_total_queues;
+}
+
+static void
+nvme_tcp_ofld_set_io_queues(struct nvme_ctrl *nctrl, unsigned int nr_io_queues)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvmf_ctrl_options *opts = nctrl->opts;
+
+	if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) {
+		/*
+		 * separate read/write queues
+		 * hand out dedicated default queues only after we have
+		 * sufficient read queues.
+		 */
+		ctrl->io_queues[HCTX_TYPE_READ] = opts->nr_io_queues;
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_READ];
+		ctrl->io_queues[HCTX_TYPE_DEFAULT] =
+			min(opts->nr_write_queues, nr_io_queues);
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	} else {
+		/*
+		 * shared read/write queues
+		 * either no write queues were requested, or we don't have
+		 * sufficient queue count to have dedicated default queues.
+		 */
+		ctrl->io_queues[HCTX_TYPE_DEFAULT] =
+			min(opts->nr_io_queues, nr_io_queues);
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	}
+
+	if (opts->nr_poll_queues && nr_io_queues) {
+		/* map dedicated poll queues only if we have queues left */
+		ctrl->io_queues[HCTX_TYPE_POLL] =
+			min(opts->nr_poll_queues, nr_io_queues);
+	}
+}
+
+static int nvme_tcp_ofld_create_io_queues(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	int i, rc;
+
+	for (i = 1; i < nctrl->queue_count; i++) {
+		mutex_init(&ctrl->queues[i].queue_lock);
+
+		rc = ctrl->dev->ops->create_queue(&ctrl->queues[i],
+						  i, nctrl->sqsize + 1);
+		if (rc)
+			goto out_free_queues;
+
+		set_bit(NVME_TCP_OFLD_Q_ALLOCATED, &ctrl->queues[i].flags);
+	}
+
+	return 0;
+
+out_free_queues:
+	for (i--; i >= 1; i--)
+		nvme_tcp_ofld_free_queue(nctrl, i);
+
+	return rc;
+}
+
+static int nvme_tcp_ofld_alloc_io_queues(struct nvme_ctrl *nctrl)
+{
+	unsigned int nr_io_queues;
+	int rc;
+
+	nr_io_queues = nvme_tcp_ofld_nr_io_queues(nctrl);
+	rc = nvme_set_queue_count(nctrl, &nr_io_queues);
+	if (rc)
+		return rc;
+
+	nctrl->queue_count = nr_io_queues + 1;
+	if (nctrl->queue_count < 2) {
+		dev_err(nctrl->device,
+			"unable to set any I/O queues\n");
+
+		return -ENOMEM;
+	}
+
+	dev_info(nctrl->device, "creating %d I/O queues.\n", nr_io_queues);
+	nvme_tcp_ofld_set_io_queues(nctrl, nr_io_queues);
+
+	return nvme_tcp_ofld_create_io_queues(nctrl);
+}
+
+static int nvme_tcp_ofld_start_io_queues(struct nvme_ctrl *nctrl)
+{
+	int i, rc = 0;
+
+	for (i = 1; i < nctrl->queue_count; i++) {
+		rc = nvme_tcp_ofld_start_queue(nctrl, i);
+		if (rc)
+			goto out_stop_queues;
+	}
+
+	return 0;
+
+out_stop_queues:
+	for (i--; i >= 1; i--)
+		nvme_tcp_ofld_stop_queue(nctrl, i);
 
 	return rc;
 }
@@ -249,9 +485,10 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 static int
 nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 {
-	int rc;
+	int rc = nvme_tcp_ofld_alloc_io_queues(nctrl);
 
-	/* Placeholder - alloc_io_queues */
+	if (rc)
+		return rc;
 
 	if (new) {
 		nctrl->tagset = nvme_tcp_ofld_alloc_tagset(nctrl, false);
@@ -269,7 +506,9 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 		}
 	}
 
-	/* Placeholder - start_io_queues */
+	rc = nvme_tcp_ofld_start_io_queues(nctrl);
+	if (rc)
+		goto out_cleanup_connect_q;
 
 	if (!new) {
 		nvme_start_queues(nctrl);
@@ -291,16 +530,16 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 out_wait_freeze_timed_out:
 	nvme_stop_queues(nctrl);
 	nvme_sync_io_queues(nctrl);
-
-	/* Placeholder - Stop IO queues */
-
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+out_cleanup_connect_q:
+	nvme_cancel_tagset(nctrl);
 	if (new)
 		blk_cleanup_queue(nctrl->connect_q);
 out_free_tag_set:
 	if (new)
 		blk_mq_free_tag_set(nctrl->tagset);
 out_free_io_queues:
-	/* Placeholder - free_io_queues */
+	nvme_tcp_ofld_free_io_queues(nctrl);
 
 	return rc;
 }
@@ -327,6 +566,17 @@ static void nvme_tcp_ofld_reconnect_or_remove(struct nvme_ctrl *nctrl)
 	}
 }
 
+static int
+nvme_tcp_ofld_init_admin_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			      unsigned int hctx_idx)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = data;
+
+	hctx->driver_data = &ctrl->queues[0];
+
+	return 0;
+}
+
 static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 {
 	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
@@ -388,9 +638,19 @@ static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 	return 0;
 
 destroy_io:
-	/* Placeholder - stop and destroy io queues*/
+	if (nctrl->queue_count > 1) {
+		nvme_stop_queues(nctrl);
+		nvme_sync_io_queues(nctrl);
+		nvme_tcp_ofld_stop_io_queues(nctrl);
+		nvme_cancel_tagset(nctrl);
+		nvme_tcp_ofld_destroy_io_queues(nctrl, new);
+	}
 destroy_admin:
-	/* Placeholder - stop and destroy admin queue*/
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	nvme_cancel_admin_tagset(nctrl);
+	nvme_tcp_ofld_destroy_admin_queue(nctrl, new);
 out_release_ctrl:
 	ctrl->dev->ops->release_ctrl(ctrl);
 
@@ -439,15 +699,37 @@ static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
 }
 
 static void
-nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove)
+nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *nctrl, bool remove)
 {
-	/* Placeholder - teardown_admin_queue */
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	nvme_cancel_admin_tagset(nctrl);
+
+	if (remove)
+		blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	nvme_tcp_ofld_destroy_admin_queue(nctrl, remove);
 }
 
 static void
 nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
 {
-	/* Placeholder - teardown_io_queues */
+	if (nctrl->queue_count <= 1)
+		return;
+
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	nvme_start_freeze(nctrl);
+	nvme_stop_queues(nctrl);
+	nvme_sync_io_queues(nctrl);
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+	nvme_cancel_tagset(nctrl);
+
+	if (remove)
+		nvme_start_queues(nctrl);
+
+	nvme_tcp_ofld_destroy_io_queues(nctrl, remove);
 }
 
 static void nvme_tcp_ofld_reconnect_ctrl_work(struct work_struct *work)
@@ -562,6 +844,12 @@ nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 	return 0;
 }
 
+inline size_t nvme_tcp_ofld_inline_data_size(struct nvme_tcp_ofld_queue *queue)
+{
+	return queue->cmnd_capsule_len - sizeof(struct nvme_command);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_inline_data_size);
+
 static blk_status_t
 nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 		       const struct blk_mq_queue_data *bd)
@@ -573,22 +861,95 @@ nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 	return BLK_STS_OK;
 }
 
+static void
+nvme_tcp_ofld_exit_request(struct blk_mq_tag_set *set,
+			   struct request *rq, unsigned int hctx_idx)
+{
+	/*
+	 * Nothing is allocated in nvme_tcp_ofld_init_request,
+	 * hence empty.
+	 */
+}
+
+static int
+nvme_tcp_ofld_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			unsigned int hctx_idx)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = data;
+
+	hctx->driver_data = &ctrl->queues[hctx_idx + 1];
+
+	return 0;
+}
+
+static int nvme_tcp_ofld_map_queues(struct blk_mq_tag_set *set)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = set->driver_data;
+	struct nvmf_ctrl_options *opts = ctrl->nctrl.opts;
+
+	if (opts->nr_write_queues && ctrl->io_queues[HCTX_TYPE_READ]) {
+		/* separate read/write queues */
+		set->map[HCTX_TYPE_DEFAULT].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_DEFAULT].queue_offset = 0;
+		set->map[HCTX_TYPE_READ].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_READ];
+		set->map[HCTX_TYPE_READ].queue_offset =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	} else {
+		/* shared read/write queues */
+		set->map[HCTX_TYPE_DEFAULT].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_DEFAULT].queue_offset = 0;
+		set->map[HCTX_TYPE_READ].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_READ].queue_offset = 0;
+	}
+	blk_mq_map_queues(&set->map[HCTX_TYPE_DEFAULT]);
+	blk_mq_map_queues(&set->map[HCTX_TYPE_READ]);
+
+	if (opts->nr_poll_queues && ctrl->io_queues[HCTX_TYPE_POLL]) {
+		/* map dedicated poll queues only if we have queues left */
+		set->map[HCTX_TYPE_POLL].nr_queues =
+				ctrl->io_queues[HCTX_TYPE_POLL];
+		set->map[HCTX_TYPE_POLL].queue_offset =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT] +
+			ctrl->io_queues[HCTX_TYPE_READ];
+		blk_mq_map_queues(&set->map[HCTX_TYPE_POLL]);
+	}
+
+	dev_info(ctrl->nctrl.device,
+		 "mapped %d/%d/%d default/read/poll queues.\n",
+		 ctrl->io_queues[HCTX_TYPE_DEFAULT],
+		 ctrl->io_queues[HCTX_TYPE_READ],
+		 ctrl->io_queues[HCTX_TYPE_POLL]);
+
+	return 0;
+}
+
+static int nvme_tcp_ofld_poll(struct blk_mq_hw_ctx *hctx)
+{
+	/* Placeholder - Implement polling mechanism */
+
+	return 0;
+}
+
 static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
 	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.complete	= nvme_complete_rq,
 	.init_request	= nvme_tcp_ofld_init_request,
-	/*
-	 * All additional ops will be also implemented and registered similar to
-	 * tcp.c
-	 */
+	.exit_request	= nvme_tcp_ofld_exit_request,
+	.init_hctx	= nvme_tcp_ofld_init_hctx,
+	.map_queues	= nvme_tcp_ofld_map_queues,
+	.poll		= nvme_tcp_ofld_poll,
 };
 
 static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
 	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.complete	= nvme_complete_rq,
 	.init_request	= nvme_tcp_ofld_init_request,
-	/*
-	 * All additional ops will be also implemented and registered similar to
-	 * tcp.c
-	 */
+	.exit_request	= nvme_tcp_ofld_exit_request,
+	.init_hctx	= nvme_tcp_ofld_init_admin_hctx,
 };
 
 static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index 4281d1dacc94..875fcd3ec04a 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -65,6 +65,9 @@ struct nvme_tcp_ofld_queue {
 	unsigned long flags;
 	size_t cmnd_capsule_len;
 
+	/* mutex used during stop_queue */
+	struct mutex queue_lock;
+
 	u8 hdr_digest;
 	u8 data_digest;
 	u8 tos;
@@ -198,3 +201,4 @@ struct nvme_tcp_ofld_ops {
 int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl);
+inline size_t nvme_tcp_ofld_inline_data_size(struct nvme_tcp_ofld_queue *queue);
-- 
2.22.0



From smalin at marvell.com  Wed Jun  2 11:42:46 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 2 Jun 2021 21:42:46 +0300
Subject: [PATCH 8/8] nvme-tcp-offload: Add IO level implementation
In-Reply-To: <20210602184246.14184-1-smalin@marvell.com>
References: <20210602184246.14184-1-smalin@marvell.com>
Message-ID: <20210602184246.14184-9-smalin@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

In this patch, we present the IO level functionality.
The nvme-tcp-offload shall work on the IO-level, meaning the
nvme-tcp-offload ULP module shall pass the request to the nvme-tcp-offload
vendor driver and shall expect for the request completion.
No additional handling is needed in between, this design will reduce the
CPU utilization as we will describe below.

The nvme-tcp-offload vendor driver shall register to nvme-tcp-offload ULP
with the following IO-path ops:
 - send_req - in order to pass the request to the handling of the offload
   driver that shall pass it to the vendor specific device
 - poll_queue

The vendor driver will manage the context from which the request will be
executed and the request aggregations.
Once the IO completed, the nvme-tcp-offload vendor driver shall call
command.done() that shall invoke the nvme-tcp-offload ULP layer for
completing the request.

This patch also add support for the nvme-tcp-offload timeout and
nvme-tcp-offload ASYNC flow.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 drivers/nvme/host/tcp-offload.c | 176 ++++++++++++++++++++++++++++++--
 drivers/nvme/host/tcp-offload.h |   2 +
 2 files changed, 171 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 1700cdf42433..c76822e5ada7 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -125,7 +125,30 @@ void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
 			    union nvme_result *result,
 			    __le16 status)
 {
-	/* Placeholder - complete request with/without error */
+	struct request *rq = blk_mq_rq_from_pdu(req);
+
+	if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), *result))
+		nvme_complete_rq(rq);
+}
+
+/**
+ * nvme_tcp_ofld_async_req_done() - NVMeTCP Offload request done callback
+ * function for async request. Pointed to by nvme_tcp_ofld_req->done.
+ * Handles both NVME_TCP_F_DATA_SUCCESS flag and NVMe CQ.
+ * @req:	NVMeTCP offload request to complete.
+ * @result:     The nvme_result.
+ * @status:     The completion status.
+ *
+ * API function that allows the vendor specific offload driver to report request
+ * completions to the common offload layer.
+ */
+void nvme_tcp_ofld_async_req_done(struct nvme_tcp_ofld_req *req,
+				  union nvme_result *result, __le16 status)
+{
+	struct nvme_tcp_ofld_queue *queue = req->queue;
+	struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl;
+
+	nvme_complete_async_event(&ctrl->nctrl, status, result);
 }
 
 static struct nvme_tcp_ofld_dev *
@@ -698,6 +721,54 @@ static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
 	kfree(ctrl);
 }
 
+static void nvme_tcp_ofld_set_sg_null(struct nvme_command *c)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = 0;
+	sg->length = 0;
+	sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) | NVME_SGL_FMT_TRANSPORT_A;
+}
+
+inline void nvme_tcp_ofld_set_sg_inline(struct nvme_tcp_ofld_queue *queue,
+					struct nvme_command *c, u32 data_len)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = cpu_to_le64(queue->ctrl->nctrl.icdoff);
+	sg->length = cpu_to_le32(data_len);
+	sg->type = (NVME_SGL_FMT_DATA_DESC << 4) | NVME_SGL_FMT_OFFSET;
+}
+
+static void nvme_tcp_ofld_map_data(struct nvme_command *c, u32 data_len)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = 0;
+	sg->length = cpu_to_le32(data_len);
+	sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) | NVME_SGL_FMT_TRANSPORT_A;
+}
+
+static void nvme_tcp_ofld_submit_async_event(struct nvme_ctrl *arg)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(arg);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[0];
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+
+	ctrl->async_req.nvme_cmd.common.opcode = nvme_admin_async_event;
+	ctrl->async_req.nvme_cmd.common.command_id = NVME_AQ_BLK_MQ_DEPTH;
+	ctrl->async_req.nvme_cmd.common.flags |= NVME_CMD_SGL_METABUF;
+
+	nvme_tcp_ofld_set_sg_null(&ctrl->async_req.nvme_cmd);
+
+	ctrl->async_req.async = true;
+	ctrl->async_req.queue = queue;
+	ctrl->async_req.done = nvme_tcp_ofld_async_req_done;
+
+	ops->send_req(&ctrl->async_req);
+}
+
 static void
 nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *nctrl, bool remove)
 {
@@ -836,9 +907,13 @@ nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 			   unsigned int numa_node)
 {
 	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_ofld_ctrl *ctrl = set->driver_data;
+	int qid;
 
-	/* Placeholder - init request */
-
+	qid = (set == &ctrl->tag_set) ? hctx_idx + 1 : 0;
+	req->queue = &ctrl->queues[qid];
+	nvme_req(rq)->ctrl = &ctrl->nctrl;
+	nvme_req(rq)->cmd = &req->nvme_cmd;
 	req->done = nvme_tcp_ofld_req_done;
 
 	return 0;
@@ -854,9 +929,46 @@ static blk_status_t
 nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 		       const struct blk_mq_queue_data *bd)
 {
-	/* Call nvme_setup_cmd(...) */
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(bd->rq);
+	struct nvme_tcp_ofld_queue *queue = hctx->driver_data;
+	struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl;
+	struct nvme_ns *ns = hctx->queue->queuedata;
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+	struct nvme_command *nvme_cmd;
+	struct request *rq = bd->rq;
+	bool queue_ready;
+	u32 data_len;
+	int rc;
+
+	queue_ready = test_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags);
+
+	req->async = false;
+
+	if (!nvme_check_ready(&ctrl->nctrl, rq, queue_ready))
+		return nvme_fail_nonready_command(&ctrl->nctrl, rq);
+
+	rc = nvme_setup_cmd(ns, rq);
+	if (unlikely(rc))
+		return rc;
 
-	/* Call ops->send_req(...) */
+	blk_mq_start_request(rq);
+
+	nvme_cmd = &req->nvme_cmd;
+	nvme_cmd->common.flags |= NVME_CMD_SGL_METABUF;
+
+	data_len = blk_rq_nr_phys_segments(rq) ? blk_rq_payload_bytes(rq) : 0;
+	if (!data_len)
+		nvme_tcp_ofld_set_sg_null(&req->nvme_cmd);
+	else if ((rq_data_dir(rq) == WRITE) &&
+		 data_len <= nvme_tcp_ofld_inline_data_size(queue))
+		nvme_tcp_ofld_set_sg_inline(queue, nvme_cmd, data_len);
+	else
+		nvme_tcp_ofld_map_data(nvme_cmd, data_len);
+
+	rc = ops->send_req(req);
+	if (unlikely(rc))
+		return rc;
 
 	return BLK_STS_OK;
 }
@@ -929,9 +1041,56 @@ static int nvme_tcp_ofld_map_queues(struct blk_mq_tag_set *set)
 
 static int nvme_tcp_ofld_poll(struct blk_mq_hw_ctx *hctx)
 {
-	/* Placeholder - Implement polling mechanism */
+	struct nvme_tcp_ofld_queue *queue = hctx->driver_data;
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
 
-	return 0;
+	return ops->poll_queue(queue);
+}
+
+static void nvme_tcp_ofld_complete_timed_out(struct request *rq)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_ctrl *nctrl = &req->queue->ctrl->nctrl;
+
+	nvme_tcp_ofld_stop_queue(nctrl, nvme_tcp_ofld_qid(req->queue));
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq)) {
+		nvme_req(rq)->status = NVME_SC_HOST_ABORTED_CMD;
+		blk_mq_complete_request(rq);
+	}
+}
+
+static enum blk_eh_timer_return nvme_tcp_ofld_timeout(struct request *rq, bool reserved)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_ofld_ctrl *ctrl = req->queue->ctrl;
+
+	dev_warn(ctrl->nctrl.device,
+		 "queue %d: timeout request %#x type %d\n",
+		 nvme_tcp_ofld_qid(req->queue), rq->tag, req->nvme_cmd.common.opcode);
+
+	if (ctrl->nctrl.state != NVME_CTRL_LIVE) {
+		/*
+		 * If we are resetting, connecting or deleting we should
+		 * complete immediately because we may block controller
+		 * teardown or setup sequence
+		 * - ctrl disable/shutdown fabrics requests
+		 * - connect requests
+		 * - initialization admin requests
+		 * - I/O requests that entered after unquiescing and
+		 *   the controller stopped responding
+		 *
+		 * All other requests should be cancelled by the error
+		 * recovery work, so it's fine that we fail it here.
+		 */
+		nvme_tcp_ofld_complete_timed_out(rq);
+
+		return BLK_EH_DONE;
+	}
+
+	nvme_tcp_ofld_error_recovery(&ctrl->nctrl);
+
+	return BLK_EH_RESET_TIMER;
 }
 
 static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
@@ -940,6 +1099,7 @@ static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
 	.init_request	= nvme_tcp_ofld_init_request,
 	.exit_request	= nvme_tcp_ofld_exit_request,
 	.init_hctx	= nvme_tcp_ofld_init_hctx,
+	.timeout	= nvme_tcp_ofld_timeout,
 	.map_queues	= nvme_tcp_ofld_map_queues,
 	.poll		= nvme_tcp_ofld_poll,
 };
@@ -950,6 +1110,7 @@ static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
 	.init_request	= nvme_tcp_ofld_init_request,
 	.exit_request	= nvme_tcp_ofld_exit_request,
 	.init_hctx	= nvme_tcp_ofld_init_admin_hctx,
+	.timeout	= nvme_tcp_ofld_timeout,
 };
 
 static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
@@ -960,6 +1121,7 @@ static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
 	.reg_read64		= nvmf_reg_read64,
 	.reg_write32		= nvmf_reg_write32,
 	.free_ctrl		= nvme_tcp_ofld_free_ctrl,
+	.submit_async_event     = nvme_tcp_ofld_submit_async_event,
 	.delete_ctrl		= nvme_tcp_ofld_delete_ctrl,
 	.get_address		= nvmf_get_address,
 };
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index 875fcd3ec04a..2ac5b2428612 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -114,6 +114,8 @@ struct nvme_tcp_ofld_ctrl {
 	/* Connectivity params */
 	struct nvme_tcp_ofld_ctrl_con_params conn_params;
 
+	struct nvme_tcp_ofld_req async_req;
+
 	/* Vendor specific driver context */
 	void *private_data;
 };
-- 
2.22.0



From david.e.box at linux.intel.com  Wed Jun  2 15:49:38 2021
From: david.e.box at linux.intel.com (David E. Box)
Date: Wed, 02 Jun 2021 15:49:38 -0700
Subject: [PATCH v3 1/2] nvme: Look for StorageD3Enable on companion ACPI
 device instead
In-Reply-To: <474a58bf-dcc9-48a8-c5ce-fb79f2b76a79@amd.com>
References: <20210528160234.9402-1-mario.limonciello@amd.com>
 <20210531064655.GA1417@lst.de>
 <PH0PR11MB4853EBD21B8D89AC8138308F9E3E9@PH0PR11MB4853.namprd11.prod.outlook.com>
 <474a58bf-dcc9-48a8-c5ce-fb79f2b76a79@amd.com>
Message-ID: <7ff9423dc806f1354e51dbb69dea9b70dd8c6eea.camel@linux.intel.com>

Hi Mario,

The patch works for systems that worked on the current implementation.
Thanks.

Reviewed-by: David E. Box <david.e.box at linux.intel.com>

On Tue, 2021-06-01 at 10:15 -0500, Limonciello, Mario wrote:
> > 
> > I think we need to wait for the confirmation from David.? This looks
> > good, but I'd like to see testing.? I also wonder how many of the
> > simple suspend quirks we can drop with this.
> 
> David, do you think you can get this soonish or are you similarly 
> blocked like Shyjumon by remote work restrictions?? As this is causing 
> see this series land for 5.14.? If you can't get to it in time, I would
> prefer that we switch this patch back to how it was conservatively done
> in v1 for 5.14 (and redo 2/2 on top of that):
> https://lore.kernel.org/linux-nvme/20210527113751.GB17266 at lst.de/T/#t
> 
> and let you get back to the list when you can to optimize it "later".
> 
> > 
> > Shyjumon and Jon, can you retests the platforms quirked in
> > 1fae37accfc5 ("nvme/pci: Add sleep quirk for Samsung and Toshiba
> > drives") with this fix?
> > 
> > Shyjumon>> Yes, I do agree we need to test this also. However the
> > boards where which I had these issues are not in my remote access now
> > (as the work frequency on this boards are less and also due to Covid
> > situation),
> > ???????????????????????? It might? take some time for me to test. I
> > will update as soon as I can.
> > 




From chaitanya.kulkarni at wdc.com  Wed Jun  2 17:37:58 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  2 Jun 2021 17:37:58 -0700
Subject: [PATCH] nvmet: remove extra variable
Message-ID: <20210603003758.21134-1-chaitanya.kulkarni@wdc.com>

Remove extra variable bdev that is only used once in the
nvmet_bdev_alloc_bip() and use req->ns->bdev that is used everywhere
in the code to access the nvmet request's bdev.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/io-cmd-bdev.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index 429263ca9b97..f673679d258a 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -174,11 +174,10 @@ static int nvmet_bdev_alloc_bip(struct nvmet_req *req, struct bio *bio,
 {
 	struct blk_integrity *bi;
 	struct bio_integrity_payload *bip;
-	struct block_device *bdev = req->ns->bdev;
 	int rc;
 	size_t resid, len;
 
-	bi = bdev_get_integrity(bdev);
+	bi = bdev_get_integrity(req->ns->bdev);
 	if (unlikely(!bi)) {
 		pr_err("Unable to locate bio_integrity\n");
 		return -ENODEV;
-- 
2.22.1



From hch at infradead.org  Thu Jun  3 00:05:15 2021
From: hch at infradead.org (Christoph Hellwig)
Date: Thu, 3 Jun 2021 10:05:15 +0300
Subject: [GIT PULL] nvme fixes for Linux 5.13
Message-ID: <YLh/K7UbJFnA2HPa@infradead.org>

The following changes since commit a4b58f1721eb4d7d27e0fdcaba60d204248dcd25:

  Merge tag 'nvme-5.13-2021-05-27' of git://git.infradead.org/nvme into block-5.13 (2021-05-27 07:38:12 -0600)

are available in the Git repository at:

  git://git.infradead.org/nvme.git tags/nvme-5.13-2021-06-03

for you to fetch changes up to bcd9a0797d73eeff659582f23277e7ab6e5f18f3:

  nvmet: fix freeing unallocated p2pmem (2021-06-02 10:10:38 +0300)

----------------------------------------------------------------
nvme fixes for Linux 5.13:

 - fix corruption in RDMA in-capsule SGLs (Sagi Grimberg)
 - nvme-loop reset fixes (Hannes Reinecke)
 - nvmet fix for freeing unallocated p2pmem (Max Gurtovoy)

----------------------------------------------------------------
Hannes Reinecke (4):
      nvme-loop: reset queue count to 1 in nvme_loop_destroy_io_queues()
      nvme-loop: clear NVME_LOOP_Q_LIVE when nvme_loop_configure_admin_queue() fails
      nvme-loop: check for NVME_LOOP_Q_LIVE in nvme_loop_destroy_admin_queue()
      nvme-loop: do not warn for deleted controllers during reset

Max Gurtovoy (1):
      nvmet: fix freeing unallocated p2pmem

Sagi Grimberg (1):
      nvme-rdma: fix in-casule data send for chained sgls

 drivers/nvme/host/rdma.c   |  5 +++--
 drivers/nvme/target/core.c | 33 ++++++++++++++++-----------------
 drivers/nvme/target/loop.c | 11 ++++++++---
 3 files changed, 27 insertions(+), 22 deletions(-)


From hch at lst.de  Thu Jun  3 00:16:54 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 3 Jun 2021 09:16:54 +0200
Subject: [PATCH] nvme: verify MNAN value if ANA is enabled
In-Reply-To: <20210531103651.109426-1-dwagner@suse.de>
References: <20210531103651.109426-1-dwagner@suse.de>
Message-ID: <20210603071654.GA4536@lst.de>

> @@ -2972,6 +2972,16 @@ static int nvme_init_identify(struct nvme_ctrl *ctrl)
>  	if (ret < 0)
>  		goto out_free;
>  
> +#ifdef CONFIG_NVME_MULTIPATH
> +	if (ctrl->ana_log_buf && (!ctrl->max_namespaces ||
> +				  ctrl->max_namespaces > le32_to_cpu(id->nn))) {
> +		dev_err(ctrl->device,
> +			"Invalid MNAN value %u\n", ctrl->max_namespaces);
> +		ret = -EINVAL;
> +		goto out_free;
> +	}
> +#endif

Please move the check into nvme_mpath_init_identify, where we can avoid
the ifdef and the ana_log_buf check, which won't trigger for the first
initialization.


From hch at lst.de  Thu Jun  3 00:17:35 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 3 Jun 2021 09:17:35 +0200
Subject: [PATCH v3 1/2] nvme: Look for StorageD3Enable on companion
 ACPI device instead
In-Reply-To: <20210528160234.9402-1-mario.limonciello@amd.com>
References: <20210528160234.9402-1-mario.limonciello@amd.com>
Message-ID: <20210603071735.GA4607@lst.de>

Thanks,

applied to nvme-5.14.  I've also added a fixes tag.


From hch at lst.de  Thu Jun  3 00:20:57 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 3 Jun 2021 09:20:57 +0200
Subject: [PATCH] nvmet: remove extra variable
In-Reply-To: <20210603003758.21134-1-chaitanya.kulkarni@wdc.com>
References: <20210603003758.21134-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210603072057.GB4607@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Thu Jun  3 00:22:44 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 3 Jun 2021 09:22:44 +0200
Subject: [PATCH] nvme-multipath: call del_gendisk() in
 nvme_mpath_check_last_path()
In-Reply-To: <20210531070914.102439-1-hare@suse.de>
References: <20210531070914.102439-1-hare@suse.de>
Message-ID: <20210603072244.GA4687@lst.de>

> +void nvme_mpath_check_last_path(struct nvme_ns *ns)
>  {
> +	struct nvme_ns_head *head = ns->head;
> +
>  	if (!head->disk)
>  		return;
> -	if (head->disk->flags & GENHD_FL_UP) {
> +
> +	if (list_empty(&head->list) && head->disk->flags & GENHD_FL_UP) {

Please add the braces around the "&".

>  		nvme_cdev_del(&head->cdev, &head->cdev_device);
>  		del_gendisk(head->disk);
>  	}
> +}

Also it seems like we can just pass the head here, no need for ns.

> -	if (head->disk && list_empty(&head->list))
> -		kblockd_schedule_work(&head->requeue_work);

Did this get lost in the new version?


From hch at lst.de  Thu Jun  3 00:25:37 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 3 Jun 2021 09:25:37 +0200
Subject: [PATCH v2] nvme: reset disk to the mpath node also when requeuing
In-Reply-To: <20210602130039.122879-1-dwagner@suse.de>
References: <20210602130039.122879-1-dwagner@suse.de>
Message-ID: <20210603072537.GA4718@lst.de>

On Wed, Jun 02, 2021 at 03:00:39PM +0200, Daniel Wagner wrote:
> +	struct bio *b;
>  	blk_qc_t ret = BLK_QC_T_NONE;
>  	int srcu_idx;
>  
> @@ -324,6 +325,8 @@ static blk_qc_t nvme_ns_head_submit_bio(struct bio *bio)
>  		dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");
>  
>  		spin_lock_irq(&head->requeue_lock);
> +		for (b = bio; b; b = b->bi_next)
> +			bio_set_dev(b, head->disk->part0);
>  		bio_list_add(&head->requeue_list, bio);
>  		spin_unlock_irq(&head->requeue_lock);
>  	} else {

I can't see why we'd need this hunk.  bio->bi_bdev should have never been
changed to start with in this path.

> @@ -435,11 +438,6 @@ static void nvme_requeue_work(struct work_struct *work)
>  		next = bio->bi_next;
>  		bio->bi_next = NULL;
>  
> -		/*
> -		 * Reset disk to the mpath node and resubmit to select a new
> -		 * path.
> -		 */
> -		bio_set_dev(bio, head->disk->part0);
>  		submit_bio_noacct(bio);
>  	}

This hunk looks fine.


From hch at lst.de  Thu Jun  3 00:31:43 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 3 Jun 2021 09:31:43 +0200
Subject: [PATCH] nvme-multipath: call del_gendisk() in
 nvme_mpath_check_last_path()
In-Reply-To: <20210603072244.GA4687@lst.de>
References: <20210531070914.102439-1-hare@suse.de>
 <20210603072244.GA4687@lst.de>
Message-ID: <20210603073143.GA4829@lst.de>

As I had to rebase nvme-5.14 anyway, I've dropped the original patch to
keep the nshead around.  Can you resend it including the fixup in a single
patch?


From mgurtovoy at nvidia.com  Thu Jun  3 03:52:59 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Thu, 3 Jun 2021 13:52:59 +0300
Subject: [PATCH 1/3] mm,memory_hotplug: export mhp min alignment
In-Reply-To: <283740c3-db3f-3c9a-2954-f1c037a13e86@redhat.com>
References: <20210602111055.10480-1-mgurtovoy@nvidia.com>
 <20210602111055.10480-2-mgurtovoy@nvidia.com>
 <283740c3-db3f-3c9a-2954-f1c037a13e86@redhat.com>
Message-ID: <46524325-8e2a-9395-e0d0-7d559c753c67@nvidia.com>


On 6/2/2021 3:14 PM, David Hildenbrand wrote:
> On 02.06.21 13:10, Max Gurtovoy wrote:
>> Hotplugged memory has alignmet restrictions. E.g, it disallows all
>> operations smaller than a sub-section and only allow operations smaller
>> than a section for SPARSEMEM_VMEMMAP. Export the alignment restrictions
>> for mhp users.
>>
>> Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
>> ---
>> ? include/linux/memory_hotplug.h |? 5 +++++
>> ? mm/memory_hotplug.c??????????? | 33 +++++++++++++++++++--------------
>> ? 2 files changed, 24 insertions(+), 14 deletions(-)
>>
>> diff --git a/include/linux/memory_hotplug.h 
>> b/include/linux/memory_hotplug.h
>> index 28f32fd00fe9..c55a9049b11e 100644
>> --- a/include/linux/memory_hotplug.h
>> +++ b/include/linux/memory_hotplug.h
>> @@ -76,6 +76,7 @@ struct mhp_params {
>> ? ? bool mhp_range_allowed(u64 start, u64 size, bool need_mapping);
>> ? struct range mhp_get_pluggable_range(bool need_mapping);
>> +unsigned long mhp_get_min_align(void);
>> ? ? /*
>> ?? * Zone resizing functions
>> @@ -248,6 +249,10 @@ void mem_hotplug_done(void);
>> ????? ___page;??????????????? \
>> ?? })
>> ? +static inline unsigned long mhp_get_min_align(void)
>> +{
>> +??? return 0;
>> +}
>> ? static inline unsigned zone_span_seqbegin(struct zone *zone)
>> ? {
>> ????? return 0;
>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>> index 9e86e9ee0a10..161bb6704a9b 100644
>> --- a/mm/memory_hotplug.c
>> +++ b/mm/memory_hotplug.c
>> @@ -270,24 +270,29 @@ void __init 
>> register_page_bootmem_info_node(struct pglist_data *pgdat)
>> ? }
>> ? #endif /* CONFIG_HAVE_BOOTMEM_INFO_NODE */
>> ? +/*
>> + * Disallow all operations smaller than a sub-section and only
>> + * allow operations smaller than a section for
>> + * SPARSEMEM_VMEMMAP. Note that check_hotplug_memory_range()
>> + * enforces a larger memory_block_size_bytes() granularity for
>> + * memory that will be marked online, so this check should only
>> + * fire for direct arch_{add,remove}_memory() users outside of
>> + * add_memory_resource().
>> + */
>> +unsigned long mhp_get_min_align(void)
>> +{
>> +??? if (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP))
>> +??????? return PAGES_PER_SUBSECTION;
>> +??? return PAGES_PER_SECTION;
>> +}
>> +EXPORT_SYMBOL_GPL(mhp_get_min_align);
>
> We have to main interfaces to "hotplug" memory:
>
> a) add_memory() and friends for System RAM, which have memory block 
> alignment requirements.
>
> b) memremap_pages(), which has the alignemnt requirements you mention 
> here.
>
> I feel like what you need would better be exposed in mm/memremap.c, 
> for example, via "memremap_min_alignment" so it matches the 
> "memremap_pages" semantics.
>
> And then, memremap_pages() is only available with CONFIG_ZONE_DEVICE, 
> which depends on SPARSEMEM_VMEMMAP. So you'll always have 
> PAGES_PER_SUBSECTION.
>
> I can already spot "memremap_compat_align", maybe you can reuse that 
> or handle it accordingly in there?

Yes I think that since subsection is aligned to PAGE_SIZE I can do:

size_t pci_p2pdma_align_size(size_t size)
{
 ??????? unsigned long min_align;

 ??????? min_align = memremap_compat_align();
 ??????? if (!IS_ALIGNED(size, min_align))
 ??????????????? return ALIGN_DOWN(size, min_align);

 ??????? return size;
}


thoughts ?

>


From axboe at kernel.dk  Thu Jun  3 06:42:49 2021
From: axboe at kernel.dk (Jens Axboe)
Date: Thu, 3 Jun 2021 07:42:49 -0600
Subject: [GIT PULL] nvme fixes for Linux 5.13
In-Reply-To: <YLh/K7UbJFnA2HPa@infradead.org>
References: <YLh/K7UbJFnA2HPa@infradead.org>
Message-ID: <4e58ee70-d923-cb66-44de-5e9f82408189@kernel.dk>

On 6/3/21 1:05 AM, Christoph Hellwig wrote:
> The following changes since commit a4b58f1721eb4d7d27e0fdcaba60d204248dcd25:
> 
>   Merge tag 'nvme-5.13-2021-05-27' of git://git.infradead.org/nvme into block-5.13 (2021-05-27 07:38:12 -0600)
> 
> are available in the Git repository at:
> 
>   git://git.infradead.org/nvme.git tags/nvme-5.13-2021-06-03

Pulled, thanks.

-- 
Jens Axboe



From jonathan at knownhost.com  Thu Jun  3 09:57:02 2021
From: jonathan at knownhost.com (Jonathan Wright)
Date: Thu, 3 Jun 2021 11:57:02 -0500
Subject: nvmeof Issues with Zen 3/Ryzen 5000 Initiator
In-Reply-To: <39fabb5d-537c-febf-a538-ff6215777da5@grimberg.me>
References: <ffae89d5-05d8-e0a4-ca71-b3db18f70286@knownhost.com>
 <39fabb5d-537c-febf-a538-ff6215777da5@grimberg.me>
Message-ID: <69f0c4b6-c890-ef25-ddae-6d15d38881e0@knownhost.com>


>> I've been testing NVMe over Fabrics for the past few weeks and the 
>> performance has been nothing short of incredible, though I'm running 
>> into some major issues that seems to be specifically related to AMD 
>> Zen 3 Ryzen chips (in my case I'm testing with 5900x).
>>
>> Target:
>> Supermicro X10 board
>> Xeon E5-2620v4
>> Intel E810 NIC
>>
>> Problematic Client/initiator:
>> ASRock X570 board
>> Ryzen 9 5900x
>> Intel E810 NIC
>>
>> Stable Client/initiator:
>> Supermicro X10 board
>> Xeon E5-2620v4
>> Intel E810 NIC
>>
>> I'm using the same 2 E810 NICs and pair of 25G DACs in both cases.? 
>> The NICs are directly connected with the DACs and there is no switch 
>> in the equation.? To trigger the issue I'm simply using FIO similar 
>> to this:
>>
>> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 
>> --name=test --filename=/dev/nvme0n1 --bs=4k --iodepth=64 --size=10G 
>> --readwrite=randread --time_based --runtime=1200
>>
>> I'm primarily using RDMA/iWARP right now but I've also tested RoCE2 
>> which presents the same issues/symptoms.? Primary testing has been 
>> done with Ubuntu 20.04.2 with CentOS 8 in the mix as well just to try 
>> and rule out a weird distro-specific issue. All tests used the latest 
>> ice/irdma drivers from Intel (1.5.8 and 1.5.2 respectively)
>
> CCing Shiraz Saleem who maintains irdma.

Thanks.? I've done some testing now with Mellanox ConnectX-4 cards with 
Zen 3 and the issue does not exist.? This seems to point the finger to 
something specific between irdma and Zen 2/3 since irdma/E810 works fine 
on all-Intel hardware.? I tested the Mellanox on both Ubuntu 20.04 stock 
kernel (5.4) and CentOS 8.3 (stock kernel 4.18) as I tested the E810 on 
these combinations.

Further I tested an AMD target with an Intel initiator and the issue 
still exists so it doesn't seem to matter which end the Zen 3 (and/or 
Zen2) chip is on when paired with an E810/irdma.

The issue also exists with Zen 2 (Ryzen 3600).

@Shriaz since I guess this isn't a common setup right now let me know if 
I can be of any assistance with getting to the bottom of this seeming 
incompatibility.

>
>>
>> I've not yet tested a Ryzen 5900x target with an Intel initiator but 
>> i plan to to see if it exhibits the same instability.
>>
>> The issue presents itself as a connectivity loss between the two 
>> hosts - but there is no connectivity issue.? The issue is also 
>> somewhat inconsistent.? Sometimes it will show up after 1-2 minutes 
>> of testing, sometimes instantly, and sometimes close to 10 minutes in.
>>
>> Target dmesg sample:
>> [ 3867.598007] nvmet: ctrl 1 keep-alive timer (15 seconds) expired!
>> [ 3867.598384] nvmet: ctrl 1 fatal error occurred!
>>
>> Initiator dmesg sample:
>> <snip>
>> [? 348.122160] nvme nvme4: I/O 86 QID 17 timeout
>> [? 348.122224] nvme nvme4: I/O 87 QID 17 timeout
>> [? 348.122290] nvme nvme4: I/O 88 QID 17 timeout
>> [? 348.122354] nvme nvme4: I/O 89 QID 17 timeout
>> [? 348.122417] nvme nvme4: I/O 90 QID 17 timeout
>> [? 348.122480] nvme nvme4: I/O 91 QID 17 timeout
>> [? 348.122544] nvme nvme4: I/O 92 QID 17 timeout
>> [? 348.122607] nvme nvme4: I/O 93 QID 17 timeout
>> [? 348.122670] nvme nvme4: I/O 94 QID 17 timeout
>> [? 348.122733] nvme nvme4: I/O 95 QID 17 timeout
>> [? 348.122796] nvme nvme4: I/O 96 QID 17 timeout
>> <snip>
>> [? 380.387212] nvme nvme4: creating 24 I/O queues.
>> [? 380.573925] nvme nvme4: Successfully reconnected (1 attempts)
>>
>> All the while the underlying connectivity is working just fine. 
>> There's a long delay between the timeout and the successful 
>> reconnect.? I haven't timed it but it seems like about 5 minutes. 
>> This has luckily given me plenty of time to test connectivity which 
>> has consistently been just fine on all fronts.
>
> Seems like loss of connectivity from the driver perspective.
> While this is happening, can you try an rdma application like
> ib_send_bwib_send_lat or something?
>
> I'd also suggest to run both workloads concurrently and see if they
> both suffer from a connectivity issue, this will help rule out
> if this is something specific to the nvme-rdma driver.
>
>>
>> I'm testing with a single Micron 9300 Pro 7.68TB right now which can 
>> push about 850k read IOPs.? On the Intel target/initiator combo I can 
>> run it "balls to the walls" for hours on end with 0 issues.? On the 
>> AMD initiator I can trigger the disconnect/drop generally within 5 
>> minutes. Here's where things get weird - if I limit the test to 200K 
>> IOPs or less then it's relatively stable on the AMD and I've not seen 
>> any drops when this limitation is in place.
>>
>> Here are some things I've tried which make no difference (or make 
>> things worse):
>>
>> Ubuntu 20.04.2 kernel 5.4.
>> Ubuntu 20.04.2 kernel 5.8
>> Ubuntu 20.04.2 kernel 5.10
>> CentOS 8 kernel 4.18
>> CentOS 8 kernel 5.10 (from elrepo)
>> CentOS 8 kernel 5.12 (from elrepo) - whole system actually freezes 
>> upon "nvme connect" command on this one
>> With and without multipath (native)
>> With and without round-robin on multipath (native)
>> Different NVMe drive models
>> With and without PFC
>> 10G DAC
>> 25G DAC
>> 25G DAC negotiated at 10G
>> With and without a switch
>> iWARP and RoCE2
>
> Looks like this probably always existed...
>
>>
>> I did do some testing with TCP/IP but cannot reach the >200k IOPS 
>> threshold with it which seems to be important for triggering the 
>> issue. I did not experience the drops with TCP/IP.
>>
>> I can't seem to draw any conclusion other than this being something 
>> specific to Zen 3, but I'm not sure why.? Is there somewhere I should 
>> be looking aside from "dmesg" to get some useful debug info?? 
>> According to the irdma driver there are no rdma packets getting 
>> lost/dropped/erroring, etc.? Common things like rping and 
>> ib_read_bw/ib_write_bw tests all run indefinitely without error.
>
> Ah, that is an important detail.
>
> I think that packet sniffer can help here if this is the case, IIRC
> there should be way to sniff rdma traffic using tcpdump but I don't
> remember the details. Perhaps Intel folks can help you there...

-- 
Jonathan Wright
KnownHost, LLC
https://www.knownhost.com



From sashal at kernel.org  Thu Jun  3 10:07:25 2021
From: sashal at kernel.org (Sasha Levin)
Date: Thu,  3 Jun 2021 13:07:25 -0400
Subject: [PATCH AUTOSEL 5.12 35/43] nvme-fabrics: decode host pathing error
 for connect
In-Reply-To: <20210603170734.3168284-1-sashal@kernel.org>
References: <20210603170734.3168284-1-sashal@kernel.org>
Message-ID: <20210603170734.3168284-35-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 4d9442bf263ac45d495bb7ecf75009e59c0622b2 ]

Add an additional decoding for 'host pathing error' during connect.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/host/fabrics.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 604ab0e5a2ad..fb02ca2e3096 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -336,6 +336,11 @@ static void nvmf_log_connect_error(struct nvme_ctrl *ctrl,
 			cmd->connect.recfmt);
 		break;
 
+	case NVME_SC_HOST_PATH_ERROR:
+		dev_err(ctrl->device,
+			"Connect command failed: host path error\n");
+		break;
+
 	default:
 		dev_err(ctrl->device,
 			"Connect command failed, error wo/DNR bit: %d\n",
-- 
2.30.2



From sashal at kernel.org  Thu Jun  3 10:07:30 2021
From: sashal at kernel.org (Sasha Levin)
Date: Thu,  3 Jun 2021 13:07:30 -0400
Subject: [PATCH AUTOSEL 5.12 40/43] nvme-tcp: remove incorrect Kconfig dep in
 BLK_DEV_NVME
In-Reply-To: <20210603170734.3168284-1-sashal@kernel.org>
References: <20210603170734.3168284-1-sashal@kernel.org>
Message-ID: <20210603170734.3168284-40-sashal@kernel.org>

From: Sagi Grimberg <sagi at grimberg.me>

[ Upstream commit 042a3eaad6daeabcfaf163aa44da8ea3cf8b5496 ]

We need to select NVME_CORE.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Max Gurtovoy <mgurtovoy at nvidia.com>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/host/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index a44d49d63968..494675aeaaad 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -71,7 +71,8 @@ config NVME_FC
 config NVME_TCP
 	tristate "NVM Express over Fabrics TCP host driver"
 	depends on INET
-	depends on BLK_DEV_NVME
+	depends on BLOCK
+	select NVME_CORE
 	select NVME_FABRICS
 	select CRYPTO
 	select CRYPTO_CRC32C
-- 
2.30.2



From sashal at kernel.org  Thu Jun  3 10:07:31 2021
From: sashal at kernel.org (Sasha Levin)
Date: Thu,  3 Jun 2021 13:07:31 -0400
Subject: [PATCH AUTOSEL 5.12 41/43] nvmet: fix false keep-alive timeout when a
 controller is torn down
In-Reply-To: <20210603170734.3168284-1-sashal@kernel.org>
References: <20210603170734.3168284-1-sashal@kernel.org>
Message-ID: <20210603170734.3168284-41-sashal@kernel.org>

From: Sagi Grimberg <sagi at grimberg.me>

[ Upstream commit aaeadd7075dc9e184bc7876e9dd7b3bada771df2 ]

Controller teardown flow may take some time in case it has many I/O
queues, and the host may not send us keep-alive during this period.
Hence reset the traffic based keep-alive timer so we don't trigger
a controller teardown as a result of a keep-alive expiration.

Reported-by: Yi Zhang <yi.zhang at redhat.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Tested-by: Yi Zhang <yi.zhang at redhat.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/core.c  | 15 +++++++++++----
 drivers/nvme/target/nvmet.h |  2 +-
 2 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 348057fdc568..32880c38ec69 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -388,10 +388,10 @@ static void nvmet_keep_alive_timer(struct work_struct *work)
 {
 	struct nvmet_ctrl *ctrl = container_of(to_delayed_work(work),
 			struct nvmet_ctrl, ka_work);
-	bool cmd_seen = ctrl->cmd_seen;
+	bool reset_tbkas = ctrl->reset_tbkas;
 
-	ctrl->cmd_seen = false;
-	if (cmd_seen) {
+	ctrl->reset_tbkas = false;
+	if (reset_tbkas) {
 		pr_debug("ctrl %d reschedule traffic based keep-alive timer\n",
 			ctrl->cntlid);
 		schedule_delayed_work(&ctrl->ka_work, ctrl->kato * HZ);
@@ -804,6 +804,13 @@ void nvmet_sq_destroy(struct nvmet_sq *sq)
 	percpu_ref_exit(&sq->ref);
 
 	if (ctrl) {
+		/*
+		 * The teardown flow may take some time, and the host may not
+		 * send us keep-alive during this period, hence reset the
+		 * traffic based keep-alive timer so we don't trigger a
+		 * controller teardown as a result of a keep-alive expiration.
+		 */
+		ctrl->reset_tbkas = true;
 		nvmet_ctrl_put(ctrl);
 		sq->ctrl = NULL; /* allows reusing the queue later */
 	}
@@ -953,7 +960,7 @@ bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
 	}
 
 	if (sq->ctrl)
-		sq->ctrl->cmd_seen = true;
+		sq->ctrl->reset_tbkas = true;
 
 	return true;
 
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 5aad34b106dc..43a668dc8bc4 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -166,7 +166,7 @@ struct nvmet_ctrl {
 	struct nvmet_subsys	*subsys;
 	struct nvmet_sq		**sqs;
 
-	bool			cmd_seen;
+	bool			reset_tbkas;
 
 	struct mutex		lock;
 	u64			cap;
-- 
2.30.2



From sashal at kernel.org  Thu Jun  3 10:08:22 2021
From: sashal at kernel.org (Sasha Levin)
Date: Thu,  3 Jun 2021 13:08:22 -0400
Subject: [PATCH AUTOSEL 5.10 32/39] nvme-fabrics: decode host pathing error
 for connect
In-Reply-To: <20210603170829.3168708-1-sashal@kernel.org>
References: <20210603170829.3168708-1-sashal@kernel.org>
Message-ID: <20210603170829.3168708-32-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 4d9442bf263ac45d495bb7ecf75009e59c0622b2 ]

Add an additional decoding for 'host pathing error' during connect.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/host/fabrics.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 8575724734e0..7015fba2e512 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -336,6 +336,11 @@ static void nvmf_log_connect_error(struct nvme_ctrl *ctrl,
 			cmd->connect.recfmt);
 		break;
 
+	case NVME_SC_HOST_PATH_ERROR:
+		dev_err(ctrl->device,
+			"Connect command failed: host path error\n");
+		break;
+
 	default:
 		dev_err(ctrl->device,
 			"Connect command failed, error wo/DNR bit: %d\n",
-- 
2.30.2



From sashal at kernel.org  Thu Jun  3 10:08:26 2021
From: sashal at kernel.org (Sasha Levin)
Date: Thu,  3 Jun 2021 13:08:26 -0400
Subject: [PATCH AUTOSEL 5.10 36/39] nvme-tcp: remove incorrect Kconfig dep in
 BLK_DEV_NVME
In-Reply-To: <20210603170829.3168708-1-sashal@kernel.org>
References: <20210603170829.3168708-1-sashal@kernel.org>
Message-ID: <20210603170829.3168708-36-sashal@kernel.org>

From: Sagi Grimberg <sagi at grimberg.me>

[ Upstream commit 042a3eaad6daeabcfaf163aa44da8ea3cf8b5496 ]

We need to select NVME_CORE.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Max Gurtovoy <mgurtovoy at nvidia.com>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/host/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index a44d49d63968..494675aeaaad 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -71,7 +71,8 @@ config NVME_FC
 config NVME_TCP
 	tristate "NVM Express over Fabrics TCP host driver"
 	depends on INET
-	depends on BLK_DEV_NVME
+	depends on BLOCK
+	select NVME_CORE
 	select NVME_FABRICS
 	select CRYPTO
 	select CRYPTO_CRC32C
-- 
2.30.2



From sashal at kernel.org  Thu Jun  3 10:08:27 2021
From: sashal at kernel.org (Sasha Levin)
Date: Thu,  3 Jun 2021 13:08:27 -0400
Subject: [PATCH AUTOSEL 5.10 37/39] nvmet: fix false keep-alive timeout when a
 controller is torn down
In-Reply-To: <20210603170829.3168708-1-sashal@kernel.org>
References: <20210603170829.3168708-1-sashal@kernel.org>
Message-ID: <20210603170829.3168708-37-sashal@kernel.org>

From: Sagi Grimberg <sagi at grimberg.me>

[ Upstream commit aaeadd7075dc9e184bc7876e9dd7b3bada771df2 ]

Controller teardown flow may take some time in case it has many I/O
queues, and the host may not send us keep-alive during this period.
Hence reset the traffic based keep-alive timer so we don't trigger
a controller teardown as a result of a keep-alive expiration.

Reported-by: Yi Zhang <yi.zhang at redhat.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Tested-by: Yi Zhang <yi.zhang at redhat.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/core.c  | 15 +++++++++++----
 drivers/nvme/target/nvmet.h |  2 +-
 2 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 46e4f7ea34c8..a65a4a810654 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -379,10 +379,10 @@ static void nvmet_keep_alive_timer(struct work_struct *work)
 {
 	struct nvmet_ctrl *ctrl = container_of(to_delayed_work(work),
 			struct nvmet_ctrl, ka_work);
-	bool cmd_seen = ctrl->cmd_seen;
+	bool reset_tbkas = ctrl->reset_tbkas;
 
-	ctrl->cmd_seen = false;
-	if (cmd_seen) {
+	ctrl->reset_tbkas = false;
+	if (reset_tbkas) {
 		pr_debug("ctrl %d reschedule traffic based keep-alive timer\n",
 			ctrl->cntlid);
 		schedule_delayed_work(&ctrl->ka_work, ctrl->kato * HZ);
@@ -792,6 +792,13 @@ void nvmet_sq_destroy(struct nvmet_sq *sq)
 	percpu_ref_exit(&sq->ref);
 
 	if (ctrl) {
+		/*
+		 * The teardown flow may take some time, and the host may not
+		 * send us keep-alive during this period, hence reset the
+		 * traffic based keep-alive timer so we don't trigger a
+		 * controller teardown as a result of a keep-alive expiration.
+		 */
+		ctrl->reset_tbkas = true;
 		nvmet_ctrl_put(ctrl);
 		sq->ctrl = NULL; /* allows reusing the queue later */
 	}
@@ -942,7 +949,7 @@ bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
 	}
 
 	if (sq->ctrl)
-		sq->ctrl->cmd_seen = true;
+		sq->ctrl->reset_tbkas = true;
 
 	return true;
 
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index ea96487b5424..4bf6d21290c2 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -166,7 +166,7 @@ struct nvmet_ctrl {
 	struct nvmet_subsys	*subsys;
 	struct nvmet_sq		**sqs;
 
-	bool			cmd_seen;
+	bool			reset_tbkas;
 
 	struct mutex		lock;
 	u64			cap;
-- 
2.30.2



From sashal at kernel.org  Thu Jun  3 10:09:13 2021
From: sashal at kernel.org (Sasha Levin)
Date: Thu,  3 Jun 2021 13:09:13 -0400
Subject: [PATCH AUTOSEL 5.4 25/31] nvme-fabrics: decode host pathing error for
 connect
In-Reply-To: <20210603170919.3169112-1-sashal@kernel.org>
References: <20210603170919.3169112-1-sashal@kernel.org>
Message-ID: <20210603170919.3169112-25-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 4d9442bf263ac45d495bb7ecf75009e59c0622b2 ]

Add an additional decoding for 'host pathing error' during connect.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/host/fabrics.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 3bb71f177dfd..d884187d7706 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -336,6 +336,11 @@ static void nvmf_log_connect_error(struct nvme_ctrl *ctrl,
 			cmd->connect.recfmt);
 		break;
 
+	case NVME_SC_HOST_PATH_ERROR:
+		dev_err(ctrl->device,
+			"Connect command failed: host path error\n");
+		break;
+
 	default:
 		dev_err(ctrl->device,
 			"Connect command failed, error wo/DNR bit: %d\n",
-- 
2.30.2



From sashal at kernel.org  Thu Jun  3 10:09:17 2021
From: sashal at kernel.org (Sasha Levin)
Date: Thu,  3 Jun 2021 13:09:17 -0400
Subject: [PATCH AUTOSEL 5.4 29/31] nvme-tcp: remove incorrect Kconfig dep in
 BLK_DEV_NVME
In-Reply-To: <20210603170919.3169112-1-sashal@kernel.org>
References: <20210603170919.3169112-1-sashal@kernel.org>
Message-ID: <20210603170919.3169112-29-sashal@kernel.org>

From: Sagi Grimberg <sagi at grimberg.me>

[ Upstream commit 042a3eaad6daeabcfaf163aa44da8ea3cf8b5496 ]

We need to select NVME_CORE.

Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Max Gurtovoy <mgurtovoy at nvidia.com>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/host/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 7b3f6555e67b..cf0ae71c489e 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -62,7 +62,8 @@ config NVME_FC
 config NVME_TCP
 	tristate "NVM Express over Fabrics TCP host driver"
 	depends on INET
-	depends on BLK_DEV_NVME
+	depends on BLOCK
+	select NVME_CORE
 	select NVME_FABRICS
 	select CRYPTO
 	select CRYPTO_CRC32C
-- 
2.30.2



From sashal at kernel.org  Thu Jun  3 10:09:55 2021
From: sashal at kernel.org (Sasha Levin)
Date: Thu,  3 Jun 2021 13:09:55 -0400
Subject: [PATCH AUTOSEL 4.19 19/23] nvme-fabrics: decode host pathing error
 for connect
In-Reply-To: <20210603170959.3169420-1-sashal@kernel.org>
References: <20210603170959.3169420-1-sashal@kernel.org>
Message-ID: <20210603170959.3169420-19-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 4d9442bf263ac45d495bb7ecf75009e59c0622b2 ]

Add an additional decoding for 'host pathing error' during connect.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/host/fabrics.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 05dd46f98441..3ae800e87999 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -344,6 +344,11 @@ static void nvmf_log_connect_error(struct nvme_ctrl *ctrl,
 			cmd->connect.recfmt);
 		break;
 
+	case NVME_SC_HOST_PATH_ERROR:
+		dev_err(ctrl->device,
+			"Connect command failed: host path error\n");
+		break;
+
 	default:
 		dev_err(ctrl->device,
 			"Connect command failed, error wo/DNR bit: %d\n",
-- 
2.30.2



From tj at kernel.org  Thu Jun  3 10:13:29 2021
From: tj at kernel.org (Tejun Heo)
Date: Thu, 3 Jun 2021 13:13:29 -0400
Subject: [PATCH v10 01/13] cgroup: Added cgroup_get_from_id
In-Reply-To: <1619562897-14062-2-git-send-email-muneendra.kumar@broadcom.com>
References: <1619562897-14062-1-git-send-email-muneendra.kumar@broadcom.com>
 <1619562897-14062-2-git-send-email-muneendra.kumar@broadcom.com>
Message-ID: <YLkNufxDIv3Mlle6@slm.duckdns.org>

On Wed, Apr 28, 2021 at 04:04:45AM +0530, Muneendra wrote:
> Added a new function cgroup_get_from_id  to retrieve the cgroup
> associated with cgroup id.
> Exported the same as this can be used by blk-cgorup.c
> 
> Added function declaration of cgroup_get_from_id in cgorup.h
> 
> This patch also exported the function cgroup_get_e_css
> as this is getting used in blk-cgroup.h
> 
> Reviewed-by: Hannes Reinecke <hare at suse.de>
> Signed-off-by: Muneendra <muneendra.kumar at broadcom.com>

Acked-by: Tejun Heo <tj at kernel.org>

Thanks.

-- 
tejun


From tj at kernel.org  Thu Jun  3 10:15:34 2021
From: tj at kernel.org (Tejun Heo)
Date: Thu, 3 Jun 2021 13:15:34 -0400
Subject: [PATCH v10 02/13] blkcg: Added a app identifier support for blkcg
In-Reply-To: <1619562897-14062-3-git-send-email-muneendra.kumar@broadcom.com>
References: <1619562897-14062-1-git-send-email-muneendra.kumar@broadcom.com>
 <1619562897-14062-3-git-send-email-muneendra.kumar@broadcom.com>
Message-ID: <YLkONg69icwTD+bD@slm.duckdns.org>

Hello,

On Wed, Apr 28, 2021 at 04:04:46AM +0530, Muneendra wrote:
...
> +static inline int blkcg_set_fc_appid(char *app_id, u64 cgrp_id, size_t app_id_len)
> +{
...
> +	strlcpy(blkcg->fc_app_id, app_id, app_id_len);
> +}
...
> +static inline char *blkcg_get_fc_appid(struct bio *bio)
> +{
> +	if (bio && bio->bi_blkg &&
> +		(bio->bi_blkg->blkcg->fc_app_id[0] != '\0'))
> +		return bio->bi_blkg->blkcg->fc_app_id;
> +	return NULL;
> +}

So, there's reader-writer race here but if that's okay for the use case,
please note that with a comment. Otherwise:

Acked-by: Tejun Heo <tj at kernel.org>

Thanks.

-- 
tejun


From mario.limonciello at amd.com  Thu Jun  3 12:07:53 2021
From: mario.limonciello at amd.com (Mario Limonciello)
Date: Thu,  3 Jun 2021 14:07:53 -0500
Subject: [PATCH v4] acpi: Move check for _DSD StorageD3Enable property to acpi
Message-ID: <20210603190753.5084-1-mario.limonciello@amd.com>

Although first implemented for NVME, this check may be usable by
other drivers as well. Microsoft's specification explicitly mentions
that is may be usable by SATA and AHCI devices.  Google also indicates
that they have used this with SDHCI in a downstream kernel tree that
a user can plug a storage device into.

Link: Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
Suggested-by: Keith Busch <kbusch at kernel.org>
CC: rrangel at chromium.org
CC: david.e.box at linux.intel.com
CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
CC: Alexander Deucher <Alexander.Deucher at amd.com>
CC: Rafael J. Wysocki <rjw at rjwysocki.net>
CC: Prike Liang <prike.liang at amd.com>
Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
---
 drivers/acpi/device_pm.c | 25 +++++++++++++++++++++++++
 drivers/nvme/host/pci.c  | 28 +---------------------------
 include/linux/acpi.h     |  5 +++++
 3 files changed, 31 insertions(+), 27 deletions(-)

Changes from v3->v4
 * Rebase on nvme-5.14 (w/ patch 1/2 from v3 of series accepted)
 * Adjust commit message per Christoph's suggestions
 * Adjust documentation per Christoph's suggestions

diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index d260bc1f3e6e..1edb68d00b8e 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -1340,4 +1340,29 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on)
 	return 1;
 }
 EXPORT_SYMBOL_GPL(acpi_dev_pm_attach);
+
+/**
+ * acpi_storage_d3 - Check if a storage device should use D3.
+ * @dev: Device to check
+ *
+ * Returns %true if @dev should be put into D3 when the ->suspend method is
+ * called, else %false.  The name of this function is somewhat misleading
+ * as it has nothing to do with storage except for the name of the ACPI
+ * property.  On some platforms resume will not work if this hint is ignored.
+ *
+ */
+bool acpi_storage_d3(struct device *dev)
+{
+	struct acpi_device *adev = ACPI_COMPANION(dev);
+	u8 val;
+
+	if (!adev)
+		return false;
+	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
+			&val))
+		return false;
+	return val == 1;
+}
+EXPORT_SYMBOL_GPL(acpi_storage_d3);
+
 #endif /* CONFIG_PM */
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..8fbc4c87a0d8 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2828,32 +2828,6 @@ static unsigned long check_vendor_combination_bug(struct pci_dev *pdev)
 	return 0;
 }
 
-#ifdef CONFIG_ACPI
-static bool nvme_acpi_storage_d3(struct pci_dev *dev)
-{
-	struct acpi_device *adev = ACPI_COMPANION(&dev->dev);
-	u8 val;
-
-	/*
-	 * Look for _DSD property specifying that the storage device on the port
-	 * must use D3 to support deep platform power savings during
-	 * suspend-to-idle.
-	 */
-
-	if (!adev)
-		return false;
-	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
-			&val))
-		return false;
-	return val == 1;
-}
-#else
-static inline bool nvme_acpi_storage_d3(struct pci_dev *dev)
-{
-	return false;
-}
-#endif /* CONFIG_ACPI */
-
 static void nvme_async_probe(void *data, async_cookie_t cookie)
 {
 	struct nvme_dev *dev = data;
@@ -2903,7 +2877,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	quirks |= check_vendor_combination_bug(pdev);
 
-	if (!noacpi && nvme_acpi_storage_d3(pdev)) {
+	if (!noacpi && acpi_storage_d3(&pdev->dev)) {
 		/*
 		 * Some systems use a bios work around to ask for D3 on
 		 * platforms that support kernel managed suspend.
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index c60745f657e9..dd0dafd21e33 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -1004,6 +1004,7 @@ int acpi_dev_resume(struct device *dev);
 int acpi_subsys_runtime_suspend(struct device *dev);
 int acpi_subsys_runtime_resume(struct device *dev);
 int acpi_dev_pm_attach(struct device *dev, bool power_on);
+bool acpi_storage_d3(struct device *dev);
 #else
 static inline int acpi_subsys_runtime_suspend(struct device *dev) { return 0; }
 static inline int acpi_subsys_runtime_resume(struct device *dev) { return 0; }
@@ -1011,6 +1012,10 @@ static inline int acpi_dev_pm_attach(struct device *dev, bool power_on)
 {
 	return 0;
 }
+static inline bool acpi_storage_d3(struct device *dev)
+{
+	return false;
+}
 #endif
 
 #if defined(CONFIG_ACPI) && defined(CONFIG_PM_SLEEP)
-- 
2.25.1



From patchwork-bot+netdevbpf at kernel.org  Thu Jun  3 14:10:05 2021
From: patchwork-bot+netdevbpf at kernel.org (patchwork-bot+netdevbpf at kernel.org)
Date: Thu, 03 Jun 2021 21:10:05 +0000
Subject: [PATCH 0/7] QED NVMeTCP Offload
In-Reply-To: <20210602171655.23581-1-smalin@marvell.com>
References: <20210602171655.23581-1-smalin@marvell.com>
Message-ID: <162275460557.4513.1472731666027007171.git-patchwork-notify@kernel.org>

Hello:

This series was applied to netdev/net-next.git (refs/heads/master):

On Wed, 2 Jun 2021 20:16:48 +0300 you wrote:
> Hi Dave,
> 
> Intro:
> ======
> This is the qed part of Marvell?s NVMeTCP offload series, shared as
> RFC series "NVMeTCP Offload ULP and QEDN Device Drive".
> This part is a standalone series, and is not dependent on other parts
> of the RFC.
> The overall goal is to add qedn as the offload driver for NVMeTCP,
> alongside the existing offload drivers (qedr, qedi and qedf for rdma,
> iscsi and fcoe respectively).
> 
> [...]

Here is the summary with links:
  - [1/7] qed: Add TCP_ULP FW resource layout
    https://git.kernel.org/netdev/net-next/c/1bd4f5716fc3
  - [2/7] qed: Add NVMeTCP Offload PF Level FW and HW HSI
    https://git.kernel.org/netdev/net-next/c/897e87a10c35
  - [3/7] qed: Add NVMeTCP Offload Connection Level FW and HW HSI
    https://git.kernel.org/netdev/net-next/c/76684ab8f4f9
  - [4/7] qed: Add support of HW filter block
    https://git.kernel.org/netdev/net-next/c/203d136e8958
  - [5/7] qed: Add NVMeTCP Offload IO Level FW and HW HSI
    https://git.kernel.org/netdev/net-next/c/ab47bdfd2e2e
  - [6/7] qed: Add NVMeTCP Offload IO Level FW Initializations
    https://git.kernel.org/netdev/net-next/c/826da4861430
  - [7/7] qed: Add IP services APIs support
    https://git.kernel.org/netdev/net-next/c/806ee7f81a2b

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html




From patchwork-bot+netdevbpf at kernel.org  Thu Jun  3 14:20:05 2021
From: patchwork-bot+netdevbpf at kernel.org (patchwork-bot+netdevbpf at kernel.org)
Date: Thu, 03 Jun 2021 21:20:05 +0000
Subject: [PATCH 0/8] NVMeTCP Offload ULP
In-Reply-To: <20210602184246.14184-1-smalin@marvell.com>
References: <20210602184246.14184-1-smalin@marvell.com>
Message-ID: <162275520500.10237.9696423451786528380.git-patchwork-notify@kernel.org>

Hello:

This series was applied to netdev/net-next.git (refs/heads/master):

On Wed, 2 Jun 2021 21:42:38 +0300 you wrote:
> With the goal of enabling a generic infrastructure that allows NVMe/TCP
> offload devices like NICs to seamlessly plug into the NVMe-oF stack, this
> patch series introduces the nvme-tcp-offload ULP host layer, which will
> be a new transport type called "tcp-offload" and will serve as an
> abstraction layer to work with vendor specific nvme-tcp offload drivers.
> 
> NVMeTCP offload is a full offload of the NVMeTCP protocol, this includes
> both the TCP level and the NVMeTCP level.
> 
> [...]

Here is the summary with links:
  - [1/8] nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW offload ULP
    https://git.kernel.org/netdev/net-next/c/f0e8cb6106da
  - [2/8] nvme-fabrics: Move NVMF_ALLOWED_OPTS and NVMF_REQUIRED_OPTS definitions
    https://git.kernel.org/netdev/net-next/c/98a5097d1e08
  - [3/8] nvme-fabrics: Expose nvmf_check_required_opts() globally
    https://git.kernel.org/netdev/net-next/c/af527935bd5a
  - [4/8] nvme-tcp-offload: Add device scan implementation
    https://git.kernel.org/netdev/net-next/c/4b8178ec5794
  - [5/8] nvme-tcp-offload: Add controller level implementation
    https://git.kernel.org/netdev/net-next/c/5aadd5f9311e
  - [6/8] nvme-tcp-offload: Add controller level error recovery implementation
    https://git.kernel.org/netdev/net-next/c/5faf6d685548
  - [7/8] nvme-tcp-offload: Add queue level implementation
    https://git.kernel.org/netdev/net-next/c/e4ba452ded39
  - [8/8] nvme-tcp-offload: Add IO level implementation
    https://git.kernel.org/netdev/net-next/c/35155e2626dc

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html




From rrangel at chromium.org  Thu Jun  3 15:27:49 2021
From: rrangel at chromium.org (Raul E Rangel)
Date: Thu, 3 Jun 2021 16:27:49 -0600
Subject: [PATCH v4] acpi: Move check for _DSD StorageD3Enable property to
 acpi
In-Reply-To: <20210603190753.5084-1-mario.limonciello@amd.com>
References: <20210603190753.5084-1-mario.limonciello@amd.com>
Message-ID: <YLlXZcp6ne7hcXNM@google.com>

On Thu, Jun 03, 2021 at 02:07:53PM -0500, Mario Limonciello wrote:
> Although first implemented for NVME, this check may be usable by
> other drivers as well. Microsoft's specification explicitly mentions
> that is may be usable by SATA and AHCI devices.  Google also indicates
> that they have used this with SDHCI in a downstream kernel tree that
> a user can plug a storage device into.
> 
> Link: Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Suggested-by: Keith Busch <kbusch at kernel.org>
> CC: rrangel at chromium.org
> CC: david.e.box at linux.intel.com
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> ---
>  drivers/acpi/device_pm.c | 25 +++++++++++++++++++++++++
>  drivers/nvme/host/pci.c  | 28 +---------------------------
>  include/linux/acpi.h     |  5 +++++
>  3 files changed, 31 insertions(+), 27 deletions(-)
> 
> Changes from v3->v4
>  * Rebase on nvme-5.14 (w/ patch 1/2 from v3 of series accepted)
>  * Adjust commit message per Christoph's suggestions
>  * Adjust documentation per Christoph's suggestions
> 
> diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
> index d260bc1f3e6e..1edb68d00b8e 100644
> --- a/drivers/acpi/device_pm.c
> +++ b/drivers/acpi/device_pm.c
> @@ -1340,4 +1340,29 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on)
>  	return 1;
>  }
>  EXPORT_SYMBOL_GPL(acpi_dev_pm_attach);
> +
> +/**
> + * acpi_storage_d3 - Check if a storage device should use D3.
> + * @dev: Device to check
> + *
> + * Returns %true if @dev should be put into D3 when the ->suspend method is
> + * called, else %false.  The name of this function is somewhat misleading
> + * as it has nothing to do with storage except for the name of the ACPI
> + * property.  On some platforms resume will not work if this hint is ignored.
> + *
> + */
> +bool acpi_storage_d3(struct device *dev)
> +{
> +	struct acpi_device *adev = ACPI_COMPANION(dev);
> +	u8 val;
> +
> +	if (!adev)
> +		return false;
> +	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> +			&val))
> +		return false;
> +	return val == 1;
> +}
> +EXPORT_SYMBOL_GPL(acpi_storage_d3);
> +
>  #endif /* CONFIG_PM */
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 3aa7245a505f..8fbc4c87a0d8 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -2828,32 +2828,6 @@ static unsigned long check_vendor_combination_bug(struct pci_dev *pdev)
>  	return 0;
>  }
>  
> -#ifdef CONFIG_ACPI
> -static bool nvme_acpi_storage_d3(struct pci_dev *dev)
> -{
> -	struct acpi_device *adev = ACPI_COMPANION(&dev->dev);
> -	u8 val;
> -
> -	/*
> -	 * Look for _DSD property specifying that the storage device on the port
> -	 * must use D3 to support deep platform power savings during
> -	 * suspend-to-idle.
> -	 */
> -
> -	if (!adev)
> -		return false;
> -	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> -			&val))
> -		return false;
> -	return val == 1;
> -}
> -#else
> -static inline bool nvme_acpi_storage_d3(struct pci_dev *dev)
> -{
> -	return false;
> -}
> -#endif /* CONFIG_ACPI */
> -
>  static void nvme_async_probe(void *data, async_cookie_t cookie)
>  {
>  	struct nvme_dev *dev = data;
> @@ -2903,7 +2877,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>  
>  	quirks |= check_vendor_combination_bug(pdev);
>  
> -	if (!noacpi && nvme_acpi_storage_d3(pdev)) {
> +	if (!noacpi && acpi_storage_d3(&pdev->dev)) {
>  		/*
>  		 * Some systems use a bios work around to ask for D3 on
>  		 * platforms that support kernel managed suspend.
> diff --git a/include/linux/acpi.h b/include/linux/acpi.h
> index c60745f657e9..dd0dafd21e33 100644
> --- a/include/linux/acpi.h
> +++ b/include/linux/acpi.h
> @@ -1004,6 +1004,7 @@ int acpi_dev_resume(struct device *dev);
>  int acpi_subsys_runtime_suspend(struct device *dev);
>  int acpi_subsys_runtime_resume(struct device *dev);
>  int acpi_dev_pm_attach(struct device *dev, bool power_on);
> +bool acpi_storage_d3(struct device *dev);
>  #else
>  static inline int acpi_subsys_runtime_suspend(struct device *dev) { return 0; }
>  static inline int acpi_subsys_runtime_resume(struct device *dev) { return 0; }
> @@ -1011,6 +1012,10 @@ static inline int acpi_dev_pm_attach(struct device *dev, bool power_on)
>  {
>  	return 0;
>  }
> +static inline bool acpi_storage_d3(struct device *dev)
> +{
> +	return false;
> +}
>  #endif
>  
>  #if defined(CONFIG_ACPI) && defined(CONFIG_PM_SLEEP)

Tested on an AMD Chromebook and it still works.

Reviewed-by: Raul E Rangel <rrangel at chromium.org>


From lkp at intel.com  Thu Jun  3 16:47:55 2021
From: lkp at intel.com (kernel test robot)
Date: Fri, 04 Jun 2021 07:47:55 +0800
Subject: [linux-nvme:nvme-5.13] BUILD SUCCESS
 bcd9a0797d73eeff659582f23277e7ab6e5f18f3
Message-ID: <60b96a2b.h8AOvvrJ2MJMHBC4%lkp@intel.com>

tree/branch: git://git.infradead.org/nvme.git nvme-5.13
branch HEAD: bcd9a0797d73eeff659582f23277e7ab6e5f18f3  nvmet: fix freeing unallocated p2pmem

elapsed time: 2345m

configs tested: 352
configs skipped: 3

The following configs have been built successfully.
More configs may be tested in the coming days.

gcc tested configs:
arm                                 defconfig
arm64                            allyesconfig
arm64                               defconfig
arm                              allyesconfig
arm                              allmodconfig
arm                       omap2plus_defconfig
m68k                          amiga_defconfig
arm                           sunxi_defconfig
riscv                             allnoconfig
riscv                               defconfig
arm                         socfpga_defconfig
arm                            qcom_defconfig
sh                          sdk7786_defconfig
mips                       rbtx49xx_defconfig
parisc                           allyesconfig
mips                        vocore2_defconfig
mips                           ip27_defconfig
powerpc                      pasemi_defconfig
m68k                       m5249evb_defconfig
arm                         lpc32xx_defconfig
mips                        qi_lb60_defconfig
arm                      footbridge_defconfig
mips                  decstation_64_defconfig
mips                           ip22_defconfig
mips                      maltaaprp_defconfig
mips                        nlm_xlr_defconfig
powerpc                    adder875_defconfig
m68k                         apollo_defconfig
powerpc               mpc834x_itxgp_defconfig
powerpc                      arches_defconfig
powerpc                           allnoconfig
powerpc                    mvme5100_defconfig
arm                      pxa255-idp_defconfig
arm                         s3c2410_defconfig
microblaze                          defconfig
arc                     nsimosci_hs_defconfig
h8300                     edosk2674_defconfig
arm                             mxs_defconfig
parisc                generic-64bit_defconfig
arm                           h5000_defconfig
ia64                      gensparse_defconfig
arm                          pxa168_defconfig
mips                          rm200_defconfig
sh                            titan_defconfig
powerpc                     tqm8548_defconfig
powerpc                 xes_mpc85xx_defconfig
sh                         apsh4a3a_defconfig
arc                         haps_hs_defconfig
arm                         mv78xx0_defconfig
sh                         ecovec24_defconfig
mips                        workpad_defconfig
arc                     haps_hs_smp_defconfig
arm                           sama5_defconfig
sh                            migor_defconfig
powerpc                     tqm8560_defconfig
sh                           se7722_defconfig
m68k                         amcore_defconfig
mips                    maltaup_xpa_defconfig
arm                        mini2440_defconfig
parisc                              defconfig
sh                               j2_defconfig
powerpc                 mpc837x_mds_defconfig
mips                       lemote2f_defconfig
arm                              alldefconfig
arc                        vdk_hs38_defconfig
powerpc                     powernv_defconfig
arm                       cns3420vb_defconfig
mips                        jmr3927_defconfig
sh                           sh2007_defconfig
arm                            zeus_defconfig
m68k                        m5407c3_defconfig
x86_64                           alldefconfig
powerpc                      pmac32_defconfig
sh                          lboxre2_defconfig
sh                          polaris_defconfig
sh                          landisk_defconfig
um                           x86_64_defconfig
powerpc                 mpc832x_mds_defconfig
mips                        bcm47xx_defconfig
s390                       zfcpdump_defconfig
arm                         bcm2835_defconfig
mips                            ar7_defconfig
powerpc                  storcenter_defconfig
arm                         s3c6400_defconfig
sparc                       sparc32_defconfig
powerpc                     tqm8540_defconfig
sh                               alldefconfig
powerpc                     sbc8548_defconfig
arm                       netwinder_defconfig
mips                           rs90_defconfig
arm                        cerfcube_defconfig
powerpc                 canyonlands_defconfig
powerpc64                        alldefconfig
powerpc64                           defconfig
arc                           tb10x_defconfig
arm                             ezx_defconfig
sh                           se7343_defconfig
sh                  sh7785lcr_32bit_defconfig
mips                   sb1250_swarm_defconfig
arm                    vt8500_v6_v7_defconfig
xtensa                generic_kc705_defconfig
mips                            gpr_defconfig
h8300                       h8s-sim_defconfig
sh                         microdev_defconfig
powerpc                 mpc8315_rdb_defconfig
powerpc                 mpc85xx_cds_defconfig
powerpc                     redwood_defconfig
powerpc                    gamecube_defconfig
mips                      loongson3_defconfig
powerpc                     tqm5200_defconfig
sh                             shx3_defconfig
arm                           viper_defconfig
nios2                         3c120_defconfig
powerpc                      pcm030_defconfig
nds32                             allnoconfig
powerpc                   lite5200b_defconfig
arm                           u8500_defconfig
um                                  defconfig
alpha                               defconfig
x86_64                            allnoconfig
arc                              alldefconfig
arm                        neponset_defconfig
m68k                       m5475evb_defconfig
arm                          pcm027_defconfig
arm                         lubbock_defconfig
sh                        sh7785lcr_defconfig
sh                          kfr2r09_defconfig
arm                  colibri_pxa270_defconfig
mips                            e55_defconfig
h8300                    h8300h-sim_defconfig
m68k                       m5208evb_defconfig
sh                          r7785rp_defconfig
mips                       capcella_defconfig
mips                        nlm_xlp_defconfig
sh                           se7721_defconfig
powerpc                 mpc8272_ads_defconfig
powerpc                        cell_defconfig
mips                        bcm63xx_defconfig
mips                        maltaup_defconfig
powerpc                      ppc64e_defconfig
s390                             alldefconfig
arm                       imx_v4_v5_defconfig
sparc                            allyesconfig
powerpc                      katmai_defconfig
xtensa                              defconfig
nios2                               defconfig
mips                         db1xxx_defconfig
powerpc                     pseries_defconfig
powerpc                      tqm8xx_defconfig
sh                          urquell_defconfig
powerpc                      cm5200_defconfig
sh                            hp6xx_defconfig
um                            kunit_defconfig
i386                             alldefconfig
m68k                                defconfig
arm                     davinci_all_defconfig
sh                      rts7751r2d1_defconfig
arc                            hsdk_defconfig
powerpc                   currituck_defconfig
mips                       bmips_be_defconfig
powerpc                     mpc83xx_defconfig
parisc                generic-32bit_defconfig
openrisc                 simple_smp_defconfig
arc                        nsimosci_defconfig
xtensa                    xip_kc705_defconfig
arm                      tct_hammer_defconfig
powerpc                 mpc8560_ads_defconfig
mips                           mtx1_defconfig
sh                           se7712_defconfig
powerpc                     rainier_defconfig
arm                         orion5x_defconfig
arm                         axm55xx_defconfig
sh                             espt_defconfig
sh                        dreamcast_defconfig
mips                  cavium_octeon_defconfig
arm                         lpc18xx_defconfig
arm                         palmz72_defconfig
mips                          rb532_defconfig
powerpc                         wii_defconfig
powerpc                  mpc866_ads_defconfig
arm                           omap1_defconfig
arm                            mps2_defconfig
powerpc                         ps3_defconfig
sh                           se7206_defconfig
powerpc                 linkstation_defconfig
arm                         vf610m4_defconfig
riscv             nommu_k210_sdcard_defconfig
sh                     sh7710voipgw_defconfig
sh                        sh7763rdp_defconfig
powerpc                 mpc836x_rdk_defconfig
mips                         tb0219_defconfig
m68k                        m5307c3_defconfig
arm                       multi_v4t_defconfig
arm                       spear13xx_defconfig
powerpc                    ge_imp3a_defconfig
openrisc                  or1klitex_defconfig
arc                          axs101_defconfig
powerpc                       maple_defconfig
sparc64                             defconfig
m68k                            q40_defconfig
h8300                            alldefconfig
mips                           ci20_defconfig
arm                           tegra_defconfig
powerpc                          allmodconfig
sh                              ul2_defconfig
mips                     loongson2k_defconfig
mips                      pistachio_defconfig
sh                        sh7757lcr_defconfig
arm                       aspeed_g4_defconfig
xtensa                  cadence_csp_defconfig
arm                         shannon_defconfig
powerpc                     tqm8541_defconfig
sh                         ap325rxa_defconfig
arm                          simpad_defconfig
mips                      malta_kvm_defconfig
arm                       versatile_defconfig
mips                          ath25_defconfig
arm                          gemini_defconfig
arm                           stm32_defconfig
arm                          badge4_defconfig
mips                         cobalt_defconfig
sparc                       sparc64_defconfig
m68k                            mac_defconfig
m68k                        mvme147_defconfig
parisc                           alldefconfig
powerpc                 mpc834x_mds_defconfig
powerpc                      bamboo_defconfig
powerpc                       ppc64_defconfig
powerpc                      chrp32_defconfig
arm                        shmobile_defconfig
sh                          sdk7780_defconfig
riscv                    nommu_virt_defconfig
powerpc                     asp8347_defconfig
powerpc                        warp_defconfig
ia64                         bigsur_defconfig
sh                     magicpanelr2_defconfig
arm                         at91_dt_defconfig
openrisc                            defconfig
arm                            mmp2_defconfig
arm                        magician_defconfig
powerpc                        icon_defconfig
arm                       mainstone_defconfig
powerpc                     skiroot_defconfig
ia64                             allmodconfig
ia64                                defconfig
ia64                             allyesconfig
m68k                             allmodconfig
m68k                             allyesconfig
arc                              allyesconfig
nds32                               defconfig
nios2                            allyesconfig
csky                                defconfig
alpha                            allyesconfig
xtensa                           allyesconfig
h8300                            allyesconfig
arc                                 defconfig
sh                               allmodconfig
s390                             allyesconfig
s390                             allmodconfig
s390                                defconfig
i386                             allyesconfig
sparc                               defconfig
i386                                defconfig
mips                             allyesconfig
mips                             allmodconfig
powerpc                          allyesconfig
x86_64               randconfig-a002-20210602
x86_64               randconfig-a004-20210602
x86_64               randconfig-a003-20210602
x86_64               randconfig-a006-20210602
x86_64               randconfig-a005-20210602
x86_64               randconfig-a001-20210602
i386                 randconfig-a003-20210602
i386                 randconfig-a006-20210602
i386                 randconfig-a004-20210602
i386                 randconfig-a001-20210602
i386                 randconfig-a005-20210602
i386                 randconfig-a002-20210602
i386                 randconfig-a003-20210603
i386                 randconfig-a006-20210603
i386                 randconfig-a004-20210603
i386                 randconfig-a001-20210603
i386                 randconfig-a002-20210603
i386                 randconfig-a005-20210603
i386                 randconfig-a003-20210601
i386                 randconfig-a006-20210601
i386                 randconfig-a004-20210601
i386                 randconfig-a001-20210601
i386                 randconfig-a002-20210601
i386                 randconfig-a005-20210601
x86_64               randconfig-a015-20210603
x86_64               randconfig-a011-20210603
x86_64               randconfig-a012-20210603
x86_64               randconfig-a014-20210603
x86_64               randconfig-a016-20210603
x86_64               randconfig-a013-20210603
x86_64               randconfig-a015-20210601
x86_64               randconfig-a011-20210601
x86_64               randconfig-a012-20210601
x86_64               randconfig-a014-20210601
x86_64               randconfig-a016-20210601
x86_64               randconfig-a013-20210601
i386                 randconfig-a015-20210603
i386                 randconfig-a011-20210603
i386                 randconfig-a014-20210603
i386                 randconfig-a012-20210603
i386                 randconfig-a015-20210602
i386                 randconfig-a013-20210602
i386                 randconfig-a016-20210602
i386                 randconfig-a011-20210602
i386                 randconfig-a014-20210602
i386                 randconfig-a012-20210602
i386                 randconfig-a015-20210601
i386                 randconfig-a013-20210601
i386                 randconfig-a011-20210601
i386                 randconfig-a016-20210601
i386                 randconfig-a014-20210601
i386                 randconfig-a012-20210601
i386                 randconfig-a013-20210603
i386                 randconfig-a016-20210603
riscv                    nommu_k210_defconfig
riscv                            allyesconfig
riscv                          rv32_defconfig
riscv                            allmodconfig
um                             i386_defconfig
x86_64                           allyesconfig
x86_64                    rhel-8.3-kselftests
x86_64                              defconfig
x86_64                               rhel-8.3
x86_64                      rhel-8.3-kbuiltin
x86_64                                  kexec

clang tested configs:
x86_64               randconfig-b001-20210601
x86_64               randconfig-b001-20210603
x86_64               randconfig-b001-20210602
x86_64               randconfig-a002-20210603
x86_64               randconfig-a004-20210603
x86_64               randconfig-a003-20210603
x86_64               randconfig-a006-20210603
x86_64               randconfig-a005-20210603
x86_64               randconfig-a001-20210603
x86_64               randconfig-a002-20210601
x86_64               randconfig-a004-20210601
x86_64               randconfig-a003-20210601
x86_64               randconfig-a006-20210601
x86_64               randconfig-a005-20210601
x86_64               randconfig-a001-20210601
x86_64               randconfig-a015-20210602
x86_64               randconfig-a011-20210602
x86_64               randconfig-a012-20210602
x86_64               randconfig-a014-20210602
x86_64               randconfig-a016-20210602
x86_64               randconfig-a013-20210602

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org


From jsmart2021 at gmail.com  Thu Jun  3 16:52:54 2021
From: jsmart2021 at gmail.com (James Smart)
Date: Thu, 3 Jun 2021 16:52:54 -0700
Subject: [PATCH] nvmet/fc: do not check for invalid target port in
 nvmet_fc_handle_fcp_rqst()
In-Reply-To: <20210525125414.68118-1-hare@suse.de>
References: <20210525125414.68118-1-hare@suse.de>
Message-ID: <9e76d0b2-fa41-03d0-2230-f7ecc2444485@gmail.com>

On 5/25/2021 5:54 AM, Hannes Reinecke wrote:
> When parsing a request in nvmet_fc_handle_fcp_rqst() we should not
> check for invalid target ports; if we do the command is aborted
> from the fcp layer, causing the host to assume a transport error.
> Rather we should still forward this request to the nvmet layer, which
> will then correctly fail the command with an appropriate error status.
> 
> Signed-off-by: Hannes Reinecke <hare at suse.de>

Well, correct behavior either way as I to would think it is a transport 
error if the port hasn't been tied to any subsystem yet.  But, I guess 
it's fine to let nvmet_find_get_subsys() see the null port pointer and 
fail Connect with INVALID_PARAM|DNR as well.

so - ok

Reviewed-by: James Smart <jsmart2021 at gmail.com>

-- james



From lkp at intel.com  Fri Jun  4 07:12:12 2021
From: lkp at intel.com (kernel test robot)
Date: Fri, 04 Jun 2021 22:12:12 +0800
Subject: [linux-nvme:nvme-5.14] BUILD SUCCESS
 346ac785badf66120d8b4c7b48f87b0a536f691e
Message-ID: <60ba34bc.IAep/NPSY0rhBMGB%lkp@intel.com>

tree/branch: git://git.infradead.org/nvme.git nvme-5.14
branch HEAD: 346ac785badf66120d8b4c7b48f87b0a536f691e  nvmet: remove a superfluous variable

elapsed time: 1802m

configs tested: 109
configs skipped: 2

The following configs have been built successfully.
More configs may be tested in the coming days.

gcc tested configs:
arm                                 defconfig
arm64                               defconfig
arm                              allyesconfig
arm                              allmodconfig
arm64                            allyesconfig
arm                         socfpga_defconfig
arm                            qcom_defconfig
sh                          sdk7786_defconfig
mips                       rbtx49xx_defconfig
mips                        vocore2_defconfig
powerpc                 xes_mpc85xx_defconfig
powerpc                      obs600_defconfig
mips                           jazz_defconfig
arm                          collie_defconfig
sh                           se7722_defconfig
m68k                         amcore_defconfig
mips                    maltaup_xpa_defconfig
arm                          pxa168_defconfig
powerpc                 mpc832x_rdb_defconfig
powerpc                        warp_defconfig
arm                        neponset_defconfig
powerpc                 mpc8272_ads_defconfig
mips                            ar7_defconfig
powerpc                        cell_defconfig
mips                        bcm63xx_defconfig
arm                         axm55xx_defconfig
sh                             espt_defconfig
sh                        dreamcast_defconfig
arm                         shannon_defconfig
powerpc                     tqm8541_defconfig
sh                        sh7785lcr_defconfig
xtensa                    smp_lx200_defconfig
mips                        jmr3927_defconfig
sh                        apsh4ad0a_defconfig
ia64                             alldefconfig
s390                                defconfig
x86_64                            allnoconfig
ia64                             allmodconfig
ia64                                defconfig
ia64                             allyesconfig
m68k                             allmodconfig
m68k                                defconfig
m68k                             allyesconfig
nios2                               defconfig
arc                              allyesconfig
nds32                             allnoconfig
nds32                               defconfig
nios2                            allyesconfig
csky                                defconfig
alpha                               defconfig
alpha                            allyesconfig
xtensa                           allyesconfig
h8300                            allyesconfig
arc                                 defconfig
sh                               allmodconfig
parisc                              defconfig
s390                             allyesconfig
s390                             allmodconfig
parisc                           allyesconfig
i386                             allyesconfig
sparc                            allyesconfig
sparc                               defconfig
i386                                defconfig
mips                             allyesconfig
mips                             allmodconfig
powerpc                          allyesconfig
powerpc                          allmodconfig
powerpc                           allnoconfig
i386                 randconfig-a003-20210603
i386                 randconfig-a006-20210603
i386                 randconfig-a004-20210603
i386                 randconfig-a001-20210603
i386                 randconfig-a002-20210603
i386                 randconfig-a005-20210603
x86_64               randconfig-a015-20210603
x86_64               randconfig-a011-20210603
x86_64               randconfig-a012-20210603
x86_64               randconfig-a014-20210603
x86_64               randconfig-a016-20210603
x86_64               randconfig-a013-20210603
i386                 randconfig-a015-20210603
i386                 randconfig-a013-20210603
i386                 randconfig-a011-20210603
i386                 randconfig-a016-20210603
i386                 randconfig-a014-20210603
i386                 randconfig-a012-20210603
riscv                    nommu_k210_defconfig
riscv                            allyesconfig
riscv                    nommu_virt_defconfig
riscv                             allnoconfig
riscv                               defconfig
riscv                          rv32_defconfig
riscv                            allmodconfig
um                           x86_64_defconfig
um                             i386_defconfig
um                            kunit_defconfig
x86_64                           allyesconfig
x86_64                    rhel-8.3-kselftests
x86_64                              defconfig
x86_64                               rhel-8.3
x86_64                      rhel-8.3-kbuiltin
x86_64                                  kexec

clang tested configs:
x86_64               randconfig-b001-20210603
x86_64               randconfig-a002-20210603
x86_64               randconfig-a004-20210603
x86_64               randconfig-a003-20210603
x86_64               randconfig-a006-20210603
x86_64               randconfig-a005-20210603
x86_64               randconfig-a001-20210603

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org


From mario.limonciello at amd.com  Fri Jun  4 09:54:02 2021
From: mario.limonciello at amd.com (Mario Limonciello)
Date: Fri,  4 Jun 2021 11:54:02 -0500
Subject: [PATCH v5 1/2] acpi: PM: Move check for _DSD StorageD3Enable property
 to acpi
Message-ID: <20210604165403.2317-1-mario.limonciello@amd.com>

Although first implemented for NVME, this check may be usable by
other drivers as well. Microsoft's specification explicitly mentions
that is may be usable by SATA and AHCI devices.  Google also indicates
that they have used this with SDHCI in a downstream kernel tree that
a user can plug a storage device into.

Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
Suggested-by: Keith Busch <kbusch at kernel.org>
CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
CC: Alexander Deucher <Alexander.Deucher at amd.com>
CC: Rafael J. Wysocki <rjw at rjwysocki.net>
CC: Prike Liang <prike.liang at amd.com>
Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
---
 drivers/acpi/device_pm.c | 25 +++++++++++++++++++++++++
 drivers/nvme/host/pci.c  | 28 +---------------------------
 include/linux/acpi.h     |  5 +++++
 3 files changed, 31 insertions(+), 27 deletions(-)

Changes from v4->v5:
 * Correct extra "Link:" word in commit message

diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index d260bc1f3e6e..1edb68d00b8e 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -1340,4 +1340,29 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on)
 	return 1;
 }
 EXPORT_SYMBOL_GPL(acpi_dev_pm_attach);
+
+/**
+ * acpi_storage_d3 - Check if a storage device should use D3.
+ * @dev: Device to check
+ *
+ * Returns %true if @dev should be put into D3 when the ->suspend method is
+ * called, else %false.  The name of this function is somewhat misleading
+ * as it has nothing to do with storage except for the name of the ACPI
+ * property.  On some platforms resume will not work if this hint is ignored.
+ *
+ */
+bool acpi_storage_d3(struct device *dev)
+{
+	struct acpi_device *adev = ACPI_COMPANION(dev);
+	u8 val;
+
+	if (!adev)
+		return false;
+	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
+			&val))
+		return false;
+	return val == 1;
+}
+EXPORT_SYMBOL_GPL(acpi_storage_d3);
+
 #endif /* CONFIG_PM */
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..8fbc4c87a0d8 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2828,32 +2828,6 @@ static unsigned long check_vendor_combination_bug(struct pci_dev *pdev)
 	return 0;
 }
 
-#ifdef CONFIG_ACPI
-static bool nvme_acpi_storage_d3(struct pci_dev *dev)
-{
-	struct acpi_device *adev = ACPI_COMPANION(&dev->dev);
-	u8 val;
-
-	/*
-	 * Look for _DSD property specifying that the storage device on the port
-	 * must use D3 to support deep platform power savings during
-	 * suspend-to-idle.
-	 */
-
-	if (!adev)
-		return false;
-	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
-			&val))
-		return false;
-	return val == 1;
-}
-#else
-static inline bool nvme_acpi_storage_d3(struct pci_dev *dev)
-{
-	return false;
-}
-#endif /* CONFIG_ACPI */
-
 static void nvme_async_probe(void *data, async_cookie_t cookie)
 {
 	struct nvme_dev *dev = data;
@@ -2903,7 +2877,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	quirks |= check_vendor_combination_bug(pdev);
 
-	if (!noacpi && nvme_acpi_storage_d3(pdev)) {
+	if (!noacpi && acpi_storage_d3(&pdev->dev)) {
 		/*
 		 * Some systems use a bios work around to ask for D3 on
 		 * platforms that support kernel managed suspend.
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index c60745f657e9..dd0dafd21e33 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -1004,6 +1004,7 @@ int acpi_dev_resume(struct device *dev);
 int acpi_subsys_runtime_suspend(struct device *dev);
 int acpi_subsys_runtime_resume(struct device *dev);
 int acpi_dev_pm_attach(struct device *dev, bool power_on);
+bool acpi_storage_d3(struct device *dev);
 #else
 static inline int acpi_subsys_runtime_suspend(struct device *dev) { return 0; }
 static inline int acpi_subsys_runtime_resume(struct device *dev) { return 0; }
@@ -1011,6 +1012,10 @@ static inline int acpi_dev_pm_attach(struct device *dev, bool power_on)
 {
 	return 0;
 }
+static inline bool acpi_storage_d3(struct device *dev)
+{
+	return false;
+}
 #endif
 
 #if defined(CONFIG_ACPI) && defined(CONFIG_PM_SLEEP)
-- 
2.25.1



From mario.limonciello at amd.com  Fri Jun  4 09:54:03 2021
From: mario.limonciello at amd.com (Mario Limonciello)
Date: Fri,  4 Jun 2021 11:54:03 -0500
Subject: [PATCH v5 2/2] acpi: PM: Add quirks for AMD Renoir/Lucienne CPUs to
 force the D3 hint
In-Reply-To: <20210604165403.2317-1-mario.limonciello@amd.com>
References: <20210604165403.2317-1-mario.limonciello@amd.com>
Message-ID: <20210604165403.2317-2-mario.limonciello@amd.com>

AMD systems from Renoir and Lucienne require that the NVME controller
is put into D3 over a Modern Standby / suspend-to-idle
cycle.  This is "typically" accomplished using the `StorageD3Enable`
property in the _DSD, but this property was introduced after many
of these systems launched and most OEM systems don't have it in
their BIOS.

On AMD Renoir without these drives going into D3 over suspend-to-idle
the resume will fail with the NVME controller being reset and a trace
like this in the kernel logs:
```
[   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
[   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
[   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
[   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
[   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
[   95.332843] nvme nvme0: Abort status: 0x371
[   95.332852] nvme nvme0: Abort status: 0x371
[   95.332856] nvme nvme0: Abort status: 0x371
[   95.332859] nvme nvme0: Abort status: 0x371
[   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
[   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
```

The Microsoft documentation for StorageD3Enable mentioned that Windows has
a hardcoded allowlist for D3 support, which was used for these platforms.
Introduce quirks to hardcode them for Linux as well.

As this property is now "standardized", OEM systems using AMD Cezanne and
newer APU's have adopted this property, and quirks like this should not be
necessary.

CC: Julian Sikorski <belegdol at gmail.com>
CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
CC: Alexander Deucher <Alexander.Deucher at amd.com>
CC: Rafael J. Wysocki <rjw at rjwysocki.net>
CC: Prike Liang <prike.liang at amd.com>
Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
---
 drivers/acpi/device_pm.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

Changes from v4->v5:
 * Add this patch back in as it's been made apparent that the
   system needs to be hardcoded for these.
   Changes:
   - Drop Cezanne - it's now covered by StorageD3Enable
   - Rebase ontop of acpi_storage_d3 outside of NVME

diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index 1edb68d00b8e..8fd2a15bf478 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -20,6 +20,10 @@
 #include <linux/pm_runtime.h>
 #include <linux/suspend.h>
 
+#ifdef CONFIG_X86
+#include <asm/cpu_device_id.h>
+#endif
+
 #include "internal.h"
 
 /**
@@ -1341,6 +1345,15 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on)
 }
 EXPORT_SYMBOL_GPL(acpi_dev_pm_attach);
 
+
+#ifdef CONFIG_X86
+static const struct x86_cpu_id storage_d3_cpu_ids[] = {
+	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),	/* Renoir */
+	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL),	/* Lucienne */
+	{}
+};
+#endif
+
 /**
  * acpi_storage_d3 - Check if a storage device should use D3.
  * @dev: Device to check
@@ -1356,6 +1369,12 @@ bool acpi_storage_d3(struct device *dev)
 	struct acpi_device *adev = ACPI_COMPANION(dev);
 	u8 val;
 
+#ifdef CONFIG_X86
+	/* Devices requiring D3, but from before StorageD3Enable was "standardized" */
+	if (x86_match_cpu(storage_d3_cpu_ids))
+		return true;
+#endif
+
 	if (!adev)
 		return false;
 	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
-- 
2.25.1



From rrangel at chromium.org  Fri Jun  4 10:03:07 2021
From: rrangel at chromium.org (Raul Rangel)
Date: Fri, 4 Jun 2021 11:03:07 -0600
Subject: [PATCH v5 1/2] acpi: PM: Move check for _DSD StorageD3Enable
 property to acpi
In-Reply-To: <20210604165403.2317-1-mario.limonciello@amd.com>
References: <20210604165403.2317-1-mario.limonciello@amd.com>
Message-ID: <CAHQZ30B9niNnqPEi5XMjjiVv+N-GAeBOC0jk20owz3UZRyT4Gg@mail.gmail.com>

On Fri, Jun 4, 2021 at 10:54 AM Mario Limonciello
<mario.limonciello at amd.com> wrote:
>
> Although first implemented for NVME, this check may be usable by
> other drivers as well. Microsoft's specification explicitly mentions
> that is may be usable by SATA and AHCI devices.  Google also indicates
> that they have used this with SDHCI in a downstream kernel tree that
> a user can plug a storage device into.
>
> Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Suggested-by: Keith Busch <kbusch at kernel.org>
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> ---
>  drivers/acpi/device_pm.c | 25 +++++++++++++++++++++++++
>  drivers/nvme/host/pci.c  | 28 +---------------------------
>  include/linux/acpi.h     |  5 +++++
>  3 files changed, 31 insertions(+), 27 deletions(-)
>
> Changes from v4->v5:
>  * Correct extra "Link:" word in commit message
>

Acked-by: Raul E Rangel <rrangel at chromium.org>


From rrangel at chromium.org  Fri Jun  4 10:43:08 2021
From: rrangel at chromium.org (Raul Rangel)
Date: Fri, 4 Jun 2021 11:43:08 -0600
Subject: [PATCH v5 2/2] acpi: PM: Add quirks for AMD Renoir/Lucienne CPUs
 to force the D3 hint
In-Reply-To: <20210604165403.2317-2-mario.limonciello@amd.com>
References: <20210604165403.2317-1-mario.limonciello@amd.com>
 <20210604165403.2317-2-mario.limonciello@amd.com>
Message-ID: <CAHQZ30BWzY=oLvOtj==uinW2Fu__skuoLNYhz5NmDg=oRwq4Sw@mail.gmail.com>

On Fri, Jun 4, 2021 at 10:54 AM Mario Limonciello
<mario.limonciello at amd.com> wrote:

> +
> +#ifdef CONFIG_X86
> +static const struct x86_cpu_id storage_d3_cpu_ids[] = {
> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),  /* Renoir */
> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL), /* Lucienne */
> +       {}
> +};
> +#endif
> +

Is this the same matching logic that Windows is using?


From mario.limonciello at amd.com  Fri Jun  4 10:57:55 2021
From: mario.limonciello at amd.com (Limonciello, Mario)
Date: Fri, 4 Jun 2021 12:57:55 -0500
Subject: [PATCH v5 2/2] acpi: PM: Add quirks for AMD Renoir/Lucienne CPUs
 to force the D3 hint
In-Reply-To: <CAHQZ30BWzY=oLvOtj==uinW2Fu__skuoLNYhz5NmDg=oRwq4Sw@mail.gmail.com>
References: <20210604165403.2317-1-mario.limonciello@amd.com>
 <20210604165403.2317-2-mario.limonciello@amd.com>
 <CAHQZ30BWzY=oLvOtj==uinW2Fu__skuoLNYhz5NmDg=oRwq4Sw@mail.gmail.com>
Message-ID: <acd70140-aac1-bde2-225c-1a8c1c272753@amd.com>

On 6/4/2021 12:43, Raul Rangel wrote:
> On Fri, Jun 4, 2021 at 10:54 AM Mario Limonciello
> <mario.limonciello at amd.com> wrote:
> 
>> +
>> +#ifdef CONFIG_X86
>> +static const struct x86_cpu_id storage_d3_cpu_ids[] = {
>> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),  /* Renoir */
>> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL), /* Lucienne */
>> +       {}
>> +};
>> +#endif
>> +
> 
> Is this the same matching logic that Windows is using?
> 

I don't have access to confirm their logic for it - but they do have an 
allowlist that was used for systems before StorageD3Enable was 
introduced as well as a registry key to override it in Windows.

In Linux we can do it a number of ways:

* Detect the CPU in RN/LCN platforms
   - Like I did in this patch

* Detect some other PCI device only present in RN/LCN platforms and set 
this hint
   - Like some earlier versions of this patch series from Prike did

* Introduce a tri-state module parameter like d3=auto,off,on
   - Set up logic behind auto to use acpi_storage_d3 primarily and look 
at a quirk list as a fallback if that was false.

* Add a compile time option to include these quirks in either acpi or 
nvme.ko only if a user selected them.

* Enumerate all the field systems SMBIOS we can find with these CPUs
   - Expect this to be a large quirk list.

I don't have a strong opinion between those two first options, but 
suspect the 3rd through 5th aren't really acceptable or scalable.

I'm open to other suggestions but testers of the patches thus far have 
made it clear that /something/ needs to be done to avoid the problems on 
RN with Linux though.


From Alexander.Deucher at amd.com  Fri Jun  4 11:48:41 2021
From: Alexander.Deucher at amd.com (Deucher, Alexander)
Date: Fri, 4 Jun 2021 18:48:41 +0000
Subject: [PATCH v5 2/2] acpi: PM: Add quirks for AMD Renoir/Lucienne CPUs
 to force the D3 hint
In-Reply-To: <acd70140-aac1-bde2-225c-1a8c1c272753@amd.com>
References: <20210604165403.2317-1-mario.limonciello@amd.com>
 <20210604165403.2317-2-mario.limonciello@amd.com>
 <CAHQZ30BWzY=oLvOtj==uinW2Fu__skuoLNYhz5NmDg=oRwq4Sw@mail.gmail.com>
 <acd70140-aac1-bde2-225c-1a8c1c272753@amd.com>
Message-ID: <MN2PR12MB4488E83A1117FBD5DDA51FF4F73B9@MN2PR12MB4488.namprd12.prod.outlook.com>

[AMD Public Use]

> -----Original Message-----
> From: Limonciello, Mario <Mario.Limonciello at amd.com>
> Sent: Friday, June 4, 2021 1:58 PM
> To: Raul Rangel <rrangel at chromium.org>
> Cc: Keith Busch <kbusch at kernel.org>; Jens Axboe <axboe at fb.com>;
> Christoph Hellwig <hch at lst.de>; Sagi Grimberg <sagi at grimberg.me>; Rafael
> J . Wysocki <rjw at rjwysocki.net>; open list:NVM EXPRESS DRIVER <linux-
> nvme at lists.infradead.org>; linux-acpi at vger.kernel.org;
> david.e.box at linux.intel.com; S-k, Shyam-sundar <Shyam-sundar.S-
> k at amd.com>; Deucher, Alexander <Alexander.Deucher at amd.com>; Liang,
> Prike <Prike.Liang at amd.com>; Julian Sikorski <belegdol at gmail.com>
> Subject: Re: [PATCH v5 2/2] acpi: PM: Add quirks for AMD Renoir/Lucienne
> CPUs to force the D3 hint
> 
> On 6/4/2021 12:43, Raul Rangel wrote:
> > On Fri, Jun 4, 2021 at 10:54 AM Mario Limonciello
> > <mario.limonciello at amd.com> wrote:
> >
> >> +
> >> +#ifdef CONFIG_X86
> >> +static const struct x86_cpu_id storage_d3_cpu_ids[] = {
> >> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),  /* Renoir
> */
> >> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL), /*
> Lucienne */
> >> +       {}
> >> +};
> >> +#endif
> >> +
> >
> > Is this the same matching logic that Windows is using?
> >
> 
> I don't have access to confirm their logic for it - but they do have an allowlist
> that was used for systems before StorageD3Enable was introduced as well as
> a registry key to override it in Windows.


My understanding from the windows team is that these AMD platforms use the allow list.

Alex

> 
> In Linux we can do it a number of ways:
> 
> * Detect the CPU in RN/LCN platforms
>    - Like I did in this patch
> 
> * Detect some other PCI device only present in RN/LCN platforms and set
> this hint
>    - Like some earlier versions of this patch series from Prike did
> 
> * Introduce a tri-state module parameter like d3=auto,off,on
>    - Set up logic behind auto to use acpi_storage_d3 primarily and look at a
> quirk list as a fallback if that was false.
> 
> * Add a compile time option to include these quirks in either acpi or nvme.ko
> only if a user selected them.
> 
> * Enumerate all the field systems SMBIOS we can find with these CPUs
>    - Expect this to be a large quirk list.
> 
> I don't have a strong opinion between those two first options, but suspect
> the 3rd through 5th aren't really acceptable or scalable.
> 
> I'm open to other suggestions but testers of the patches thus far have made
> it clear that /something/ needs to be done to avoid the problems on RN with
> Linux though.

From belegdol at gmail.com  Fri Jun  4 12:59:53 2021
From: belegdol at gmail.com (Julian Sikorski)
Date: Fri, 4 Jun 2021 21:59:53 +0200
Subject: [PATCH v5 2/2] acpi: PM: Add quirks for AMD Renoir/Lucienne CPUs
 to force the D3 hint
In-Reply-To: <20210604165403.2317-2-mario.limonciello@amd.com>
References: <20210604165403.2317-1-mario.limonciello@amd.com>
 <20210604165403.2317-2-mario.limonciello@amd.com>
Message-ID: <efa36cd9-c125-e660-048d-86b7178fbdbd@gmail.com>

Am 04.06.21 um 18:54 schrieb Mario Limonciello:
> AMD systems from Renoir and Lucienne require that the NVME controller
> is put into D3 over a Modern Standby / suspend-to-idle
> cycle.  This is "typically" accomplished using the `StorageD3Enable`
> property in the _DSD, but this property was introduced after many
> of these systems launched and most OEM systems don't have it in
> their BIOS.
> 
> On AMD Renoir without these drives going into D3 over suspend-to-idle
> the resume will fail with the NVME controller being reset and a trace
> like this in the kernel logs:
> ```
> [   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
> [   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
> [   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
> [   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
> [   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
> [   95.332843] nvme nvme0: Abort status: 0x371
> [   95.332852] nvme nvme0: Abort status: 0x371
> [   95.332856] nvme nvme0: Abort status: 0x371
> [   95.332859] nvme nvme0: Abort status: 0x371
> [   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
> [   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
> ```
> 
> The Microsoft documentation for StorageD3Enable mentioned that Windows has
> a hardcoded allowlist for D3 support, which was used for these platforms.
> Introduce quirks to hardcode them for Linux as well.
> 
> As this property is now "standardized", OEM systems using AMD Cezanne and
> newer APU's have adopted this property, and quirks like this should not be
> necessary.
> 
> CC: Julian Sikorski <belegdol at gmail.com>
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> ---
>   drivers/acpi/device_pm.c | 19 +++++++++++++++++++
>   1 file changed, 19 insertions(+)
> 
> Changes from v4->v5:
>   * Add this patch back in as it's been made apparent that the
>     system needs to be hardcoded for these.
>     Changes:
>     - Drop Cezanne - it's now covered by StorageD3Enable
>     - Rebase ontop of acpi_storage_d3 outside of NVME

Tested-by: Julian Sikorski <belegdol at gmail.com>


From dan.carpenter at oracle.com  Sat Jun  5 05:48:16 2021
From: dan.carpenter at oracle.com (Dan Carpenter)
Date: Sat, 5 Jun 2021 15:48:16 +0300
Subject: [PATCH] nvme-tcp: fix error codes in nvme_tcp_setup_ctrl()
Message-ID: <YLtykIn+Or0l08oV@mwanda>

These error paths currently return success but they should return
-EOPNOTSUPP.

Fixes: 73ffcefcfca0 ("nvme-tcp: check sgl supported by target")
Fixes: 3f2304f8c6d6 ("nvme-tcp: add NVMe over TCP host driver")
Signed-off-by: Dan Carpenter <dan.carpenter at oracle.com>
---
 drivers/nvme/host/tcp.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 34f4b3402f7c..72e9d34c3092 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1973,11 +1973,13 @@ static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new)
 		return ret;
 
 	if (ctrl->icdoff) {
+		ret = -EOPNOTSUPP;
 		dev_err(ctrl->device, "icdoff is not supported!\n");
 		goto destroy_admin;
 	}
 
 	if (!(ctrl->sgls & ((1 << 0) | (1 << 1)))) {
+		ret = -EOPNOTSUPP;
 		dev_err(ctrl->device, "Mandatory sgls are not supported!\n");
 		goto destroy_admin;
 	}
-- 
2.30.2



From Chaitanya.Kulkarni at wdc.com  Sat Jun  5 11:38:16 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sat, 5 Jun 2021 18:38:16 +0000
Subject: [PATCH] nvme-tcp: fix error codes in nvme_tcp_setup_ctrl()
References: <YLtykIn+Or0l08oV@mwanda>
Message-ID: <BYAPR04MB4965D9BCCF0AF010D283986E863A9@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/5/21 05:49, Dan Carpenter wrote:
> These error paths currently return success but they should return
> -EOPNOTSUPP.
>
> Fixes: 73ffcefcfca0 ("nvme-tcp: check sgl supported by target")
> Fixes: 3f2304f8c6d6 ("nvme-tcp: add NVMe over TCP host driver")
> Signed-off-by: Dan Carpenter <dan.carpenter at oracle.com>

Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>





From chaitanya.kulkarni at wdc.com  Sun Jun  6 17:37:00 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun,  6 Jun 2021 17:37:00 -0700
Subject: [PATCH 0/5] nvme: small cleanups
Message-ID: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>

Hi,

This adds a couple of helpers to remove the duplicate code. The last
patch removes the exta white lines from nvme-pci.

If this makes sense, please consider this series for 5.14.

-ck

Chaitanya Kulkarni (5):
  nvme: use helper to remove duplicate code
  nvme-fc: add a helper to check ctrl sgl support
  nvme-tcp: use helper for ctrl sgl check
  nvme-pci: use helper for ctrl sgl check
  nvme-pci: remove trailing lines for helpers

 drivers/nvme/host/fc.c    |  2 +-
 drivers/nvme/host/ioctl.c | 26 ++++++++++++++++----------
 drivers/nvme/host/nvme.h  |  8 ++++++++
 drivers/nvme/host/pci.c   |  4 +---
 drivers/nvme/host/tcp.c   |  2 +-
 5 files changed, 27 insertions(+), 15 deletions(-)

-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun  6 17:37:01 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun,  6 Jun 2021 17:37:01 -0700
Subject: [PATCH 1/5] nvme: use helper to remove duplicate code
In-Reply-To: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>
References: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210607003705.21784-2-chaitanya.kulkarni@wdc.com>

Add a helper nvme_validate_passthru_nsid() to validate the nsid that
removes the nsid validation and error message print code from
nvme_user_cmd() and nvme_user_cmd64().

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/ioctl.c | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)

diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c
index 2e7780ea0354..d93928d1e5bd 100644
--- a/drivers/nvme/host/ioctl.c
+++ b/drivers/nvme/host/ioctl.c
@@ -177,6 +177,20 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
 			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
 }
 
+static bool nvme_validate_passthru_nsid(struct nvme_ctrl *ctrl,
+					struct nvme_ns *ns, __u32 nsid)
+{
+	if (ns && nsid != ns->head->ns_id) {
+		dev_err(ctrl->device,
+			"%s: nsid (%u) in cmd does not match nsid (%u)"
+			"of namespace\n",
+			current->comm, nsid, ns->head->ns_id);
+		return false;
+	}
+
+	return true;
+}
+
 static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			struct nvme_passthru_cmd __user *ucmd)
 {
@@ -192,12 +206,8 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 		return -EFAULT;
 	if (cmd.flags)
 		return -EINVAL;
-	if (ns && cmd.nsid != ns->head->ns_id) {
-		dev_err(ctrl->device,
-			"%s: nsid (%u) in cmd does not match nsid (%u) of namespace\n",
-			current->comm, cmd.nsid, ns->head->ns_id);
+	if (!nvme_validate_passthru_nsid(ctrl, ns, cmd.nsid))
 		return -EINVAL;
-	}
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = cmd.opcode;
@@ -242,12 +252,8 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 		return -EFAULT;
 	if (cmd.flags)
 		return -EINVAL;
-	if (ns && cmd.nsid != ns->head->ns_id) {
-		dev_err(ctrl->device,
-			"%s: nsid (%u) in cmd does not match nsid (%u) of namespace\n",
-			current->comm, cmd.nsid, ns->head->ns_id);
+	if (!nvme_validate_passthru_nsid(ctrl, ns, cmd.nsid))
 		return -EINVAL;
-	}
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = cmd.opcode;
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun  6 17:37:02 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun,  6 Jun 2021 17:37:02 -0700
Subject: [PATCH 2/5] nvme-fc: add a helper to check ctrl sgl support
In-Reply-To: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>
References: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210607003705.21784-3-chaitanya.kulkarni@wdc.com>

For transports it is common to check if NVMe SGLs are supported or not
by the controller.

Add a helper instead of open coding controller SGL support and use it
in fc transport.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/fc.c   | 2 +-
 drivers/nvme/host/nvme.h | 8 ++++++++
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index f183f9fa03d0..7600863f7752 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -3112,7 +3112,7 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	}
 
 	/* FC-NVME supports normal SGL Data Block Descriptors */
-	if (!(ctrl->ctrl.sgls & ((1 << 0) | (1 << 1)))) {
+	if (!nvme_ctrl_sgl_supported(&ctrl->ctrl)) {
 		dev_err(ctrl->ctrl.device,
 			"Mandatory sgls are not supported!\n");
 		ret = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1f397ecba16c..1aab74128d40 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -869,6 +869,14 @@ static inline void nvme_hwmon_exit(struct nvme_ctrl *ctrl)
 }
 #endif
 
+static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
+{
+	if (!(ctrl->sgls & ((1 << 0) | (1 << 1))))
+		return true;
+
+	return false;
+}
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 void nvme_execute_passthru_rq(struct request *rq);
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun  6 17:37:03 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun,  6 Jun 2021 17:37:03 -0700
Subject: [PATCH 3/5] nvme-tcp: use helper for ctrl sgl check
In-Reply-To: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>
References: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210607003705.21784-4-chaitanya.kulkarni@wdc.com>

Use helper to check the controller's SGL support instead of open coding.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/tcp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 5fc6c568c626..8d279aacb027 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1992,7 +1992,7 @@ static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new)
 		goto destroy_admin;
 	}
 
-	if (!(ctrl->sgls & ((1 << 0) | (1 << 1)))) {
+	if (!nvme_ctrl_sgl_supported(ctrl)) {
 		dev_err(ctrl->device, "Mandatory sgls are not supported!\n");
 		goto destroy_admin;
 	}
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun  6 17:37:04 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun,  6 Jun 2021 17:37:04 -0700
Subject: [PATCH 4/5] nvme-pci: use helper for ctrl sgl check
In-Reply-To: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>
References: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210607003705.21784-5-chaitanya.kulkarni@wdc.com>

Use helper to check the controller's SGL support instead of open coding.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..a7e7afc609b4 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -536,7 +536,7 @@ static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
 
 	avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req), nseg);
 
-	if (!(dev->ctrl.sgls & ((1 << 0) | (1 << 1))))
+	if (!nvme_ctrl_sgl_supported(&dev->ctrl))
 		return false;
 	if (!iod->nvmeq->qid)
 		return false;
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun  6 17:37:05 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun,  6 Jun 2021 17:37:05 -0700
Subject: [PATCH 5/5] nvme-pci: remove trailing lines for helpers
In-Reply-To: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>
References: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210607003705.21784-6-chaitanya.kulkarni@wdc.com>

Remove the extra white lines at the end of the functions.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/pci.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index a7e7afc609b4..07d2bc317fba 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -559,7 +559,6 @@ static void nvme_free_prps(struct nvme_dev *dev, struct request *req)
 		dma_pool_free(dev->prp_page_pool, prp_list, dma_addr);
 		dma_addr = next_dma_addr;
 	}
-
 }
 
 static void nvme_free_sgls(struct nvme_dev *dev, struct request *req)
@@ -576,7 +575,6 @@ static void nvme_free_sgls(struct nvme_dev *dev, struct request *req)
 		dma_pool_free(dev->prp_page_pool, sg_list, dma_addr);
 		dma_addr = next_dma_addr;
 	}
-
 }
 
 static void nvme_unmap_sg(struct nvme_dev *dev, struct request *req)
-- 
2.22.1



From Niklas.Cassel at wdc.com  Mon Jun  7 01:02:12 2021
From: Niklas.Cassel at wdc.com (Niklas Cassel)
Date: Mon, 7 Jun 2021 08:02:12 +0000
Subject: [PATCH 4/5] nvme-pci: use helper for ctrl sgl check
In-Reply-To: <20210607003705.21784-5-chaitanya.kulkarni@wdc.com>
References: <20210607003705.21784-1-chaitanya.kulkarni@wdc.com>
 <20210607003705.21784-5-chaitanya.kulkarni@wdc.com>
Message-ID: <YL3Sg2xgnKN+/a4y@x1-carbon>

On Sun, Jun 06, 2021 at 05:37:04PM -0700, Chaitanya Kulkarni wrote:
> Use helper to check the controller's SGL support instead of open coding.
> 
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> ---
>  drivers/nvme/host/pci.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 3aa7245a505f..a7e7afc609b4 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -536,7 +536,7 @@ static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
>  
>  	avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req), nseg);
>  
> -	if (!(dev->ctrl.sgls & ((1 << 0) | (1 << 1))))
> +	if (!nvme_ctrl_sgl_supported(&dev->ctrl))
>  		return false;
>  	if (!iod->nvmeq->qid)
>  		return false;
> -- 
> 2.22.1

Hello Chaitanya,

I believe that you can apply the same cleanup for pci.c:nvme_map_data()


Kind regards,
Niklas

From dwagner at suse.de  Mon Jun  7 01:46:51 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Mon,  7 Jun 2021 10:46:51 +0200
Subject: [PATCH v3] nvme: remove superfluous bio_set_dev in nvme_requeue_work
Message-ID: <20210607084651.99122-1-dwagner@suse.de>

Commit ce86dad222e9 ("nvme-multipath: reset bdev to ns head when
failover") moved the reset code where the bio is added to the
requeue_list for the failover path. But it left the original
bio_set_dev in nvme_requeue_work.

There is a second path to nvme_requee_work. It is via
nvme_ns_head_submit_bio. Though we don't have to set bio->bi_bdev for
this path either, as it points to the correct bdev already.

Let's remove the bio_set_dev. It's updating the bio->bi_bdev with the
same pointer and thus it's unnecessary.

Signed-off-by: Daniel Wagner <dwagner at suse.de>
---

v3: - dropped bio_set_dev from nvme_ns_head_submit_bio
    - renamed patch
v2: https://lore.kernel.org/linux-nvme/20210603072537.GA4718 at lst.de/

 drivers/nvme/host/multipath.c | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 127a17b4c13d..a501917d4c6e 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -435,11 +435,6 @@ static void nvme_requeue_work(struct work_struct *work)
 		next = bio->bi_next;
 		bio->bi_next = NULL;
 
-		/*
-		 * Reset disk to the mpath node and resubmit to select a new
-		 * path.
-		 */
-		bio_set_dev(bio, head->disk->part0);
 		submit_bio_noacct(bio);
 	}
 }
-- 
2.29.2



From dwagner at suse.de  Mon Jun  7 01:56:56 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Mon,  7 Jun 2021 10:56:56 +0200
Subject: [PATCH v2] nvme: verify MNAN value if ANA is enabled
Message-ID: <20210607085656.101772-1-dwagner@suse.de>

The controller is required to have a non-zero MNAN value if it supports
ANA:

   If the controller supports Asymmetric Namespace Access Reporting, then
   this field shall be set to a non-zero value that is less than or equal
   to the NN value.

Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Daniel Wagner <dwagner at suse.de>
---

v2: - moved the check to nvme_mpath_init_identify
    - added reviewed tags

 drivers/nvme/host/multipath.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index a501917d4c6e..23573fe3fc7d 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -813,6 +813,13 @@ int nvme_mpath_init_identify(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 	    !(ctrl->subsys->cmic & NVME_CTRL_CMIC_ANA))
 		return 0;
 
+	if (!ctrl->max_namespaces ||
+	    ctrl->max_namespaces > le32_to_cpu(id->nn)) {
+		dev_err(ctrl->device,
+			"Invalid MNAN value %u\n", ctrl->max_namespaces);
+		return -EINVAL;
+	}
+
 	ctrl->anacap = id->anacap;
 	ctrl->anatt = id->anatt;
 	ctrl->nanagrpid = le32_to_cpu(id->nanagrpid);
-- 
2.29.2



From mgurtovoy at nvidia.com  Mon Jun  7 02:23:21 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Mon, 7 Jun 2021 12:23:21 +0300
Subject: [PATCH v2 1/4] nvmet: change sn size and check validity
Message-ID: <20210607092324.1031046-1-mgurtovoy@nvidia.com>

From: Noam Gottlieb <ngottlieb at nvidia.com>

According to the NVM specification, the serial_number should be 20 bytes
(bytes 23:04 of the Identify Controller data structure), and should
contain only ASCII characters.

In accordance, the serial_number size is changed to 20 bytes and before
any attempt to store a new value in serial_number we check that the
input is valid - i.e. contains only ASCII characters, is not empty and
does not exceed 20 bytes.

Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
Signed-off-by: Noam Gottlieb <ngottlieb at nvidia.com>
---

changes from v1: minor nits from Chaitanya

---
 drivers/nvme/target/admin-cmd.c |  4 +---
 drivers/nvme/target/configfs.c  | 33 +++++++++++++++++++++++----------
 drivers/nvme/target/core.c      |  4 +++-
 drivers/nvme/target/discovery.c |  4 +---
 drivers/nvme/target/nvmet.h     |  3 ++-
 5 files changed, 30 insertions(+), 18 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index dcd49a72f2f3..9c73dbfb8228 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -357,9 +357,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	id->vid = 0;
 	id->ssvid = 0;
 
-	memset(id->sn, ' ', sizeof(id->sn));
-	bin2hex(id->sn, &ctrl->subsys->serial,
-		min(sizeof(ctrl->subsys->serial), sizeof(id->sn) / 2));
+	memcpy(id->sn, ctrl->subsys->serial, NVMET_SN_MAX_SIZE);
 	memcpy_and_pad(id->mn, sizeof(id->mn), subsys->model_number,
 		       strlen(subsys->model_number), ' ');
 	memcpy_and_pad(id->fr, sizeof(id->fr),
diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index 65a0cf99f557..027b28aaf7cd 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -1030,24 +1030,43 @@ static ssize_t nvmet_subsys_attr_version_store(struct config_item *item,
 }
 CONFIGFS_ATTR(nvmet_subsys_, attr_version);
 
+/* See Section 1.5 of NVMe 1.4 */
+static bool nvmet_is_ascii(const char c)
+{
+	return c >= 0x20 && c <= 0x7e;
+}
+
 static ssize_t nvmet_subsys_attr_serial_show(struct config_item *item,
 					     char *page)
 {
 	struct nvmet_subsys *subsys = to_subsys(item);
 
-	return snprintf(page, PAGE_SIZE, "%llx\n", subsys->serial);
+	return snprintf(page, PAGE_SIZE, "%s\n", subsys->serial);
 }
 
 static ssize_t nvmet_subsys_attr_serial_store(struct config_item *item,
 					      const char *page, size_t count)
 {
-	u64 serial;
+	struct nvmet_subsys *subsys = to_subsys(item);
+	int pos, len = strcspn(page, "\n");
 
-	if (sscanf(page, "%llx\n", &serial) != 1)
+	if (!len || len > NVMET_SN_MAX_SIZE) {
+		pr_err("Serial Number can not be empty or exceed %d Bytes\n",
+		       NVMET_SN_MAX_SIZE);
 		return -EINVAL;
+	}
+
+	for (pos = 0; pos < len; pos++) {
+		if (!nvmet_is_ascii(page[pos])) {
+			pr_err("Serial Number must contain only ASCII strings\n");
+			return -EINVAL;
+		}
+	}
 
 	down_write(&nvmet_config_sem);
-	to_subsys(item)->serial = serial;
+	mutex_lock(&subsys->lock);
+	memcpy_and_pad(subsys->serial, NVMET_SN_MAX_SIZE, page, len, ' ');
+	mutex_unlock(&subsys->lock);
 	up_write(&nvmet_config_sem);
 
 	return count;
@@ -1128,12 +1147,6 @@ static ssize_t nvmet_subsys_attr_model_show(struct config_item *item,
 	return ret;
 }
 
-/* See Section 1.5 of NVMe 1.4 */
-static bool nvmet_is_ascii(const char c)
-{
-	return c >= 0x20 && c <= 0x7e;
-}
-
 static ssize_t nvmet_subsys_attr_model_store_locked(struct nvmet_subsys *subsys,
 		const char *page, size_t count)
 {
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 4ae4bea6625d..213a0c2af4f7 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1493,6 +1493,7 @@ struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
 		enum nvme_subsys_type type)
 {
 	struct nvmet_subsys *subsys;
+	char serial[NVMET_SN_MAX_SIZE / 2];
 
 	subsys = kzalloc(sizeof(*subsys), GFP_KERNEL);
 	if (!subsys)
@@ -1500,7 +1501,8 @@ struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
 
 	subsys->ver = NVMET_DEFAULT_VS;
 	/* generate a random serial number as our controllers are ephemeral: */
-	get_random_bytes(&subsys->serial, sizeof(subsys->serial));
+	get_random_bytes(&serial, sizeof(serial));
+	bin2hex(subsys->serial, &serial, sizeof(serial));
 
 	switch (type) {
 	case NVME_NQN_NVME:
diff --git a/drivers/nvme/target/discovery.c b/drivers/nvme/target/discovery.c
index fc3645fc2c24..b7fdad13094a 100644
--- a/drivers/nvme/target/discovery.c
+++ b/drivers/nvme/target/discovery.c
@@ -262,9 +262,7 @@ static void nvmet_execute_disc_identify(struct nvmet_req *req)
 		goto out;
 	}
 
-	memset(id->sn, ' ', sizeof(id->sn));
-	bin2hex(id->sn, &ctrl->subsys->serial,
-		min(sizeof(ctrl->subsys->serial), sizeof(id->sn) / 2));
+	memcpy(id->sn, ctrl->subsys->serial, NVMET_SN_MAX_SIZE);
 	memset(id->fr, ' ', sizeof(id->fr));
 	memcpy_and_pad(id->mn, sizeof(id->mn), model, sizeof(model) - 1, ' ');
 	memcpy_and_pad(id->fr, sizeof(id->fr),
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index d69a409515d6..0ae809ca428c 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -28,6 +28,7 @@
 #define NVMET_NO_ERROR_LOC		((u16)-1)
 #define NVMET_DEFAULT_CTRL_MODEL	"Linux"
 #define NVMET_MN_MAX_SIZE		40
+#define NVMET_SN_MAX_SIZE		20
 
 /*
  * Supported optional AENs:
@@ -229,7 +230,7 @@ struct nvmet_subsys {
 	u16			max_qid;
 
 	u64			ver;
-	u64			serial;
+	char			serial[NVMET_SN_MAX_SIZE];
 	char			*subsysnqn;
 	bool			pi_support;
 
-- 
2.25.1



From mgurtovoy at nvidia.com  Mon Jun  7 02:23:22 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Mon, 7 Jun 2021 12:23:22 +0300
Subject: [PATCH v2 2/4] nvmet: make sn stable once connection was established
In-Reply-To: <20210607092324.1031046-1-mgurtovoy@nvidia.com>
References: <20210607092324.1031046-1-mgurtovoy@nvidia.com>
Message-ID: <20210607092324.1031046-2-mgurtovoy@nvidia.com>

From: Noam Gottlieb <ngottlieb at nvidia.com>

Once some host has connected to the target, make sure that the serial
number is stable and cannot be changed.

Reviewed-by: Max Gurtovoy <mgurtovoy at nvidia.com>
Signed-off-by: Noam Gottlieb <ngottlieb at nvidia.com>
---
 drivers/nvme/target/admin-cmd.c |  6 ++++++
 drivers/nvme/target/configfs.c  | 27 ++++++++++++++++++++++-----
 drivers/nvme/target/nvmet.h     |  1 +
 3 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 9c73dbfb8228..5f15c3285a5b 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -337,6 +337,12 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	u32 cmd_capsule_size;
 	u16 status = 0;
 
+	if (!subsys->subsys_discovered) {
+		mutex_lock(&subsys->lock);
+		subsys->subsys_discovered = true;
+		mutex_unlock(&subsys->lock);
+	}
+
 	/*
 	 * If there is no model number yet, set it now.  It will then remain
 	 * stable for the life time of the subsystem.
diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index 027b28aaf7cd..a13da86fb374 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -1044,12 +1044,18 @@ static ssize_t nvmet_subsys_attr_serial_show(struct config_item *item,
 	return snprintf(page, PAGE_SIZE, "%s\n", subsys->serial);
 }
 
-static ssize_t nvmet_subsys_attr_serial_store(struct config_item *item,
-					      const char *page, size_t count)
+static ssize_t
+nvmet_subsys_attr_serial_store_locked(struct nvmet_subsys *subsys,
+		const char *page, size_t count)
 {
-	struct nvmet_subsys *subsys = to_subsys(item);
 	int pos, len = strcspn(page, "\n");
 
+	if (subsys->subsys_discovered) {
+		pr_err("Can't set serial number. %s is already assigned\n",
+		       subsys->serial);
+		return -EINVAL;
+	}
+
 	if (!len || len > NVMET_SN_MAX_SIZE) {
 		pr_err("Serial Number can not be empty or exceed %d Bytes\n",
 		       NVMET_SN_MAX_SIZE);
@@ -1063,13 +1069,24 @@ static ssize_t nvmet_subsys_attr_serial_store(struct config_item *item,
 		}
 	}
 
+	memcpy_and_pad(subsys->serial, NVMET_SN_MAX_SIZE, page, len, ' ');
+
+	return count;
+}
+
+static ssize_t nvmet_subsys_attr_serial_store(struct config_item *item,
+					      const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item);
+	ssize_t ret;
+
 	down_write(&nvmet_config_sem);
 	mutex_lock(&subsys->lock);
-	memcpy_and_pad(subsys->serial, NVMET_SN_MAX_SIZE, page, len, ' ');
+	ret = nvmet_subsys_attr_serial_store_locked(subsys, page, count);
 	mutex_unlock(&subsys->lock);
 	up_write(&nvmet_config_sem);
 
-	return count;
+	return ret;
 }
 CONFIGFS_ATTR(nvmet_subsys_, attr_serial);
 
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 0ae809ca428c..bd0a0b91d843 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -231,6 +231,7 @@ struct nvmet_subsys {
 
 	u64			ver;
 	char			serial[NVMET_SN_MAX_SIZE];
+	bool			subsys_discovered;
 	char			*subsysnqn;
 	bool			pi_support;
 
-- 
2.25.1



From mgurtovoy at nvidia.com  Mon Jun  7 02:23:23 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Mon, 7 Jun 2021 12:23:23 +0300
Subject: [PATCH v2 3/4] nvmet: allow mn change if subsys not discovered
In-Reply-To: <20210607092324.1031046-1-mgurtovoy@nvidia.com>
References: <20210607092324.1031046-1-mgurtovoy@nvidia.com>
Message-ID: <20210607092324.1031046-3-mgurtovoy@nvidia.com>

From: Noam Gottlieb <ngottlieb at nvidia.com>

Currently, once the subsystem's model_number is set for the first time
there is no way to change it. However, as long as no connection was
established to nvmf target, there is no reason for such restriction and
we should allow to change the subsystem's model_number as many times as
needed.

In addition, in order to simplfy the changes and make the model number
flow more similar to the rest of the attributes in the Identify
Controller data structure, we set a default value for the model number
at the initiation of the subsystem.

Reviewed-by: Max Gurtovoy <mgurtovoy at nvidia.com>
Signed-off-by: Noam Gottlieb <ngottlieb at nvidia.com>
---
 drivers/nvme/target/admin-cmd.c | 26 --------------------------
 drivers/nvme/target/configfs.c  | 10 ++--------
 drivers/nvme/target/core.c      | 21 +++++++++++++++++----
 drivers/nvme/target/discovery.c |  4 ++--
 4 files changed, 21 insertions(+), 40 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 5f15c3285a5b..cd60a8184d04 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -313,22 +313,6 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
 }
 
-static u16 nvmet_set_model_number(struct nvmet_subsys *subsys)
-{
-	u16 status = 0;
-
-	mutex_lock(&subsys->lock);
-	if (!subsys->model_number) {
-		subsys->model_number =
-			kstrdup(NVMET_DEFAULT_CTRL_MODEL, GFP_KERNEL);
-		if (!subsys->model_number)
-			status = NVME_SC_INTERNAL;
-	}
-	mutex_unlock(&subsys->lock);
-
-	return status;
-}
-
 static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 {
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
@@ -343,16 +327,6 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 		mutex_unlock(&subsys->lock);
 	}
 
-	/*
-	 * If there is no model number yet, set it now.  It will then remain
-	 * stable for the life time of the subsystem.
-	 */
-	if (!subsys->model_number) {
-		status = nvmet_set_model_number(subsys);
-		if (status)
-			goto out;
-	}
-
 	id = kzalloc(sizeof(*id), GFP_KERNEL);
 	if (!id) {
 		status = NVME_SC_INTERNAL;
diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index a13da86fb374..9ef8708b92c6 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -1154,14 +1154,8 @@ static ssize_t nvmet_subsys_attr_model_show(struct config_item *item,
 					    char *page)
 {
 	struct nvmet_subsys *subsys = to_subsys(item);
-	int ret;
-
-	mutex_lock(&subsys->lock);
-	ret = snprintf(page, PAGE_SIZE, "%s\n", subsys->model_number ?
-			subsys->model_number : NVMET_DEFAULT_CTRL_MODEL);
-	mutex_unlock(&subsys->lock);
 
-	return ret;
+	return snprintf(page, PAGE_SIZE, "%s\n", subsys->model_number);
 }
 
 static ssize_t nvmet_subsys_attr_model_store_locked(struct nvmet_subsys *subsys,
@@ -1169,7 +1163,7 @@ static ssize_t nvmet_subsys_attr_model_store_locked(struct nvmet_subsys *subsys,
 {
 	int pos = 0, len;
 
-	if (subsys->model_number) {
+	if (subsys->subsys_discovered) {
 		pr_err("Can't set model number. %s is already assigned\n",
 		       subsys->model_number);
 		return -EINVAL;
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 213a0c2af4f7..146909486b8f 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1494,6 +1494,7 @@ struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
 {
 	struct nvmet_subsys *subsys;
 	char serial[NVMET_SN_MAX_SIZE / 2];
+	int ret;
 
 	subsys = kzalloc(sizeof(*subsys), GFP_KERNEL);
 	if (!subsys)
@@ -1504,6 +1505,12 @@ struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
 	get_random_bytes(&serial, sizeof(serial));
 	bin2hex(subsys->serial, &serial, sizeof(serial));
 
+	subsys->model_number = kstrdup(NVMET_DEFAULT_CTRL_MODEL, GFP_KERNEL);
+	if (!subsys->model_number) {
+		ret = -ENOMEM;
+		goto free_subsys;
+	}
+
 	switch (type) {
 	case NVME_NQN_NVME:
 		subsys->max_qid = NVMET_NR_QUEUES;
@@ -1513,15 +1520,15 @@ struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
 		break;
 	default:
 		pr_err("%s: Unknown Subsystem type - %d\n", __func__, type);
-		kfree(subsys);
-		return ERR_PTR(-EINVAL);
+		ret = -EINVAL;
+		goto free_mn;
 	}
 	subsys->type = type;
 	subsys->subsysnqn = kstrndup(subsysnqn, NVMF_NQN_SIZE,
 			GFP_KERNEL);
 	if (!subsys->subsysnqn) {
-		kfree(subsys);
-		return ERR_PTR(-ENOMEM);
+		ret = -ENOMEM;
+		goto free_mn;
 	}
 	subsys->cntlid_min = NVME_CNTLID_MIN;
 	subsys->cntlid_max = NVME_CNTLID_MAX;
@@ -1533,6 +1540,12 @@ struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
 	INIT_LIST_HEAD(&subsys->hosts);
 
 	return subsys;
+
+free_mn:
+	kfree(subsys->model_number);
+free_subsys:
+	kfree(subsys);
+	return ERR_PTR(ret);
 }
 
 static void nvmet_subsys_free(struct kref *ref)
diff --git a/drivers/nvme/target/discovery.c b/drivers/nvme/target/discovery.c
index b7fdad13094a..7aa62bc6ae84 100644
--- a/drivers/nvme/target/discovery.c
+++ b/drivers/nvme/target/discovery.c
@@ -244,7 +244,6 @@ static void nvmet_execute_disc_identify(struct nvmet_req *req)
 {
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	struct nvme_id_ctrl *id;
-	const char model[] = "Linux";
 	u16 status = 0;
 
 	if (!nvmet_check_transfer_len(req, NVME_IDENTIFY_DATA_SIZE))
@@ -264,7 +263,8 @@ static void nvmet_execute_disc_identify(struct nvmet_req *req)
 
 	memcpy(id->sn, ctrl->subsys->serial, NVMET_SN_MAX_SIZE);
 	memset(id->fr, ' ', sizeof(id->fr));
-	memcpy_and_pad(id->mn, sizeof(id->mn), model, sizeof(model) - 1, ' ');
+	memcpy_and_pad(id->mn, sizeof(id->mn), ctrl->subsys->model_number,
+		       strlen(ctrl->subsys->model_number), ' ');
 	memcpy_and_pad(id->fr, sizeof(id->fr),
 		       UTS_RELEASE, strlen(UTS_RELEASE), ' ');
 
-- 
2.25.1



From mgurtovoy at nvidia.com  Mon Jun  7 02:23:24 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Mon, 7 Jun 2021 12:23:24 +0300
Subject: [PATCH 4/4] nvmet: make ver stable once connection established
In-Reply-To: <20210607092324.1031046-1-mgurtovoy@nvidia.com>
References: <20210607092324.1031046-1-mgurtovoy@nvidia.com>
Message-ID: <20210607092324.1031046-4-mgurtovoy@nvidia.com>

From: Noam Gottlieb <ngottlieb at nvidia.com>

Once some host has connected to the nvmf target, make sure that the
version number is stable and cannot be changed.

Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
Signed-off-by: Noam Gottlieb <ngottlieb at nvidia.com>
---

changes from v1: changing "ret" to ssize_t (from Chaitanya)

---
 drivers/nvme/target/configfs.c | 36 +++++++++++++++++++++++++++++-----
 1 file changed, 31 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index 9ef8708b92c6..273555127188 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -1007,13 +1007,26 @@ static ssize_t nvmet_subsys_attr_version_show(struct config_item *item,
 			NVME_MINOR(subsys->ver));
 }
 
-static ssize_t nvmet_subsys_attr_version_store(struct config_item *item,
-					       const char *page, size_t count)
+static ssize_t
+nvmet_subsys_attr_version_store_locked(struct nvmet_subsys *subsys,
+		const char *page, size_t count)
 {
-	struct nvmet_subsys *subsys = to_subsys(item);
 	int major, minor, tertiary = 0;
 	int ret;
 
+	if (subsys->subsys_discovered) {
+		if (NVME_TERTIARY(subsys->ver))
+			pr_err("Can't set version number. %llu.%llu.%llu is already assigned\n",
+			       NVME_MAJOR(subsys->ver),
+			       NVME_MINOR(subsys->ver),
+			       NVME_TERTIARY(subsys->ver));
+		else
+			pr_err("Can't set version number. %llu.%llu is already assigned\n",
+			       NVME_MAJOR(subsys->ver),
+			       NVME_MINOR(subsys->ver));
+		return -EINVAL;
+	}
+
 	/* passthru subsystems use the underlying controller's version */
 	if (nvmet_passthru_ctrl(subsys))
 		return -EINVAL;
@@ -1022,12 +1035,25 @@ static ssize_t nvmet_subsys_attr_version_store(struct config_item *item,
 	if (ret != 2 && ret != 3)
 		return -EINVAL;
 
-	down_write(&nvmet_config_sem);
 	subsys->ver = NVME_VS(major, minor, tertiary);
-	up_write(&nvmet_config_sem);
 
 	return count;
 }
+
+static ssize_t nvmet_subsys_attr_version_store(struct config_item *item,
+					       const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item);
+	ssize_t ret;
+
+	down_write(&nvmet_config_sem);
+	mutex_lock(&subsys->lock);
+	ret = nvmet_subsys_attr_version_store_locked(subsys, page, count);
+	mutex_unlock(&subsys->lock);
+	up_write(&nvmet_config_sem);
+
+	return ret;
+}
 CONFIGFS_ATTR(nvmet_subsys_, attr_version);
 
 /* See Section 1.5 of NVMe 1.4 */
-- 
2.25.1



From hare at suse.de  Mon Jun  7 02:46:14 2021
From: hare at suse.de (Hannes Reinecke)
Date: Mon, 7 Jun 2021 11:46:14 +0200
Subject: [PATCH] nvme-multipath: call del_gendisk() in
 nvme_mpath_check_last_path()
In-Reply-To: <20210603072244.GA4687@lst.de>
References: <20210531070914.102439-1-hare@suse.de>
 <20210603072244.GA4687@lst.de>
Message-ID: <afb8bb95-0b53-0dcf-89cb-7d96c91c2189@suse.de>

On 6/3/21 9:22 AM, Christoph Hellwig wrote:
>> +void nvme_mpath_check_last_path(struct nvme_ns *ns)
>>  {
>> +	struct nvme_ns_head *head = ns->head;
>> +
>>  	if (!head->disk)
>>  		return;
>> -	if (head->disk->flags & GENHD_FL_UP) {
>> +
>> +	if (list_empty(&head->list) && head->disk->flags & GENHD_FL_UP) {
> 
> Please add the braces around the "&".
> 
Ok.

>>  		nvme_cdev_del(&head->cdev, &head->cdev_device);
>>  		del_gendisk(head->disk);
>>  	}
>> +}
> 
> Also it seems like we can just pass the head here, no need for ns.
> 
Ok.

>> -	if (head->disk && list_empty(&head->list))
>> -		kblockd_schedule_work(&head->requeue_work);
> 
> Did this get lost in the new version?
> 
I had removed it as it'll be called from nvme_mpath_remove_disk()
anyway, but seeing that the refcount might induce a different lifetime
I'll be re-adding it.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		        Kernel Storage Architect
hare at suse.de			               +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 N?rnberg
GF: F. Imend?rffer, HRB 36809 (AG N?rnberg)


From hare at suse.de  Mon Jun  7 02:53:10 2021
From: hare at suse.de (Hannes Reinecke)
Date: Mon,  7 Jun 2021 11:53:10 +0200
Subject: [PATCHv2] nvme-multipath: call del_gendisk() in
 nvme_mpath_check_last_path()
Message-ID: <20210607095310.131310-1-hare@suse.de>

We cannot call del_gendisk() during the final nvme_put_ns_head(), as
this might be called from blkdev_put(). Doing so will deadlock on
bd_mutex:

[ 1108.529767]  __mutex_lock.constprop.0+0x2a4/0x470
[ 1108.534472]  ? __kernfs_remove.part.0+0x174/0x1f0
[ 1108.539178]  ? kernfs_remove_by_name_ns+0x5c/0x90
[ 1108.543885]  del_gendisk+0x99/0x230
[ 1108.547378]  nvme_mpath_remove_disk+0x97/0xb0 [nvme_core]
[ 1108.552787]  nvme_put_ns_head+0x2a/0xb0 [nvme_core]
[ 1108.557664]  __blkdev_put+0x115/0x160
[ 1108.561339]  blkdev_put+0x4c/0x130
[ 1108.564745]  blkdev_close+0x22/0x30
[ 1108.568238]  __fput+0x94/0x240

So call 'del_gendisk()' when checking if the last path has been removed
to avoid this deadlock.

Signed-off-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/core.c      |  2 +-
 drivers/nvme/host/multipath.c | 12 ++++++++++--
 drivers/nvme/host/nvme.h      | 11 ++---------
 3 files changed, 13 insertions(+), 12 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index cb1a76a3d42a..41a8c1de7e15 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3856,7 +3856,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	list_del_init(&ns->list);
 	up_write(&ns->ctrl->namespaces_rwsem);
 
-	nvme_mpath_check_last_path(ns);
+	nvme_mpath_check_last_path(ns->head);
 	nvme_put_ns(ns);
 }
 
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 127a17b4c13d..de74650cdb2d 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -776,14 +776,22 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id)
 #endif
 }
 
-void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+void nvme_mpath_check_last_path(struct nvme_ns_head *head)
 {
 	if (!head->disk)
 		return;
-	if (head->disk->flags & GENHD_FL_UP) {
+
+	if (list_empty(&head->list) && (head->disk->flags & GENHD_FL_UP)) {
+		kblockd_schedule_work(&head->requeue_work);
 		nvme_cdev_del(&head->cdev, &head->cdev_device);
 		del_gendisk(head->disk);
 	}
+}
+
+void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+{
+	if (!head->disk)
+		return;
 	blk_set_queue_dying(head->disk->queue);
 	/* make sure all pending bios are cleaned up */
 	kblockd_schedule_work(&head->requeue_work);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1f397ecba16c..812fc1d273e3 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -716,14 +716,7 @@ void nvme_mpath_uninit(struct nvme_ctrl *ctrl);
 void nvme_mpath_stop(struct nvme_ctrl *ctrl);
 bool nvme_mpath_clear_current_path(struct nvme_ns *ns);
 void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl);
-
-static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)
-{
-	struct nvme_ns_head *head = ns->head;
-
-	if (head->disk && list_empty(&head->list))
-		kblockd_schedule_work(&head->requeue_work);
-}
+void nvme_mpath_check_last_path(struct nvme_ns_head *head);
 
 static inline void nvme_trace_bio_complete(struct request *req)
 {
@@ -772,7 +765,7 @@ static inline bool nvme_mpath_clear_current_path(struct nvme_ns *ns)
 static inline void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl)
 {
 }
-static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)
+static inline void nvme_mpath_check_last_path(struct nvme_ns_head *head)
 {
 }
 static inline void nvme_trace_bio_complete(struct request *req)
-- 
2.29.2



From rafael at kernel.org  Mon Jun  7 05:53:24 2021
From: rafael at kernel.org (Rafael J. Wysocki)
Date: Mon, 7 Jun 2021 14:53:24 +0200
Subject: [PATCH v4] acpi: Move check for _DSD StorageD3Enable property to
 acpi
In-Reply-To: <20210603190753.5084-1-mario.limonciello@amd.com>
References: <20210603190753.5084-1-mario.limonciello@amd.com>
Message-ID: <CAJZ5v0hH_XuMucyYef=Cq-WNWt2aiVWmV5v-tm77AZ+R7HZFxg@mail.gmail.com>

On Thu, Jun 3, 2021 at 9:08 PM Mario Limonciello
<mario.limonciello at amd.com> wrote:
>
> Although first implemented for NVME, this check may be usable by
> other drivers as well. Microsoft's specification explicitly mentions
> that is may be usable by SATA and AHCI devices.  Google also indicates
> that they have used this with SDHCI in a downstream kernel tree that
> a user can plug a storage device into.

Please spell ACPI in capitals in the subject.

Otherwise

Acked-by: Rafael J. Wysocki <rafael.j.wysocki at intel.com>

or please let me know if you want me to take this patch (in which case
I'll need ACKs from the PCI and NVMe side).

> Link: Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Suggested-by: Keith Busch <kbusch at kernel.org>
> CC: rrangel at chromium.org
> CC: david.e.box at linux.intel.com
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> ---
>  drivers/acpi/device_pm.c | 25 +++++++++++++++++++++++++
>  drivers/nvme/host/pci.c  | 28 +---------------------------
>  include/linux/acpi.h     |  5 +++++
>  3 files changed, 31 insertions(+), 27 deletions(-)
>
> Changes from v3->v4
>  * Rebase on nvme-5.14 (w/ patch 1/2 from v3 of series accepted)
>  * Adjust commit message per Christoph's suggestions
>  * Adjust documentation per Christoph's suggestions
>
> diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
> index d260bc1f3e6e..1edb68d00b8e 100644
> --- a/drivers/acpi/device_pm.c
> +++ b/drivers/acpi/device_pm.c
> @@ -1340,4 +1340,29 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on)
>         return 1;
>  }
>  EXPORT_SYMBOL_GPL(acpi_dev_pm_attach);
> +
> +/**
> + * acpi_storage_d3 - Check if a storage device should use D3.
> + * @dev: Device to check
> + *
> + * Returns %true if @dev should be put into D3 when the ->suspend method is
> + * called, else %false.  The name of this function is somewhat misleading
> + * as it has nothing to do with storage except for the name of the ACPI
> + * property.  On some platforms resume will not work if this hint is ignored.
> + *
> + */
> +bool acpi_storage_d3(struct device *dev)
> +{
> +       struct acpi_device *adev = ACPI_COMPANION(dev);
> +       u8 val;
> +
> +       if (!adev)
> +               return false;
> +       if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> +                       &val))
> +               return false;
> +       return val == 1;
> +}
> +EXPORT_SYMBOL_GPL(acpi_storage_d3);
> +
>  #endif /* CONFIG_PM */
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 3aa7245a505f..8fbc4c87a0d8 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -2828,32 +2828,6 @@ static unsigned long check_vendor_combination_bug(struct pci_dev *pdev)
>         return 0;
>  }
>
> -#ifdef CONFIG_ACPI
> -static bool nvme_acpi_storage_d3(struct pci_dev *dev)
> -{
> -       struct acpi_device *adev = ACPI_COMPANION(&dev->dev);
> -       u8 val;
> -
> -       /*
> -        * Look for _DSD property specifying that the storage device on the port
> -        * must use D3 to support deep platform power savings during
> -        * suspend-to-idle.
> -        */
> -
> -       if (!adev)
> -               return false;
> -       if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> -                       &val))
> -               return false;
> -       return val == 1;
> -}
> -#else
> -static inline bool nvme_acpi_storage_d3(struct pci_dev *dev)
> -{
> -       return false;
> -}
> -#endif /* CONFIG_ACPI */
> -
>  static void nvme_async_probe(void *data, async_cookie_t cookie)
>  {
>         struct nvme_dev *dev = data;
> @@ -2903,7 +2877,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>
>         quirks |= check_vendor_combination_bug(pdev);
>
> -       if (!noacpi && nvme_acpi_storage_d3(pdev)) {
> +       if (!noacpi && acpi_storage_d3(&pdev->dev)) {
>                 /*
>                  * Some systems use a bios work around to ask for D3 on
>                  * platforms that support kernel managed suspend.
> diff --git a/include/linux/acpi.h b/include/linux/acpi.h
> index c60745f657e9..dd0dafd21e33 100644
> --- a/include/linux/acpi.h
> +++ b/include/linux/acpi.h
> @@ -1004,6 +1004,7 @@ int acpi_dev_resume(struct device *dev);
>  int acpi_subsys_runtime_suspend(struct device *dev);
>  int acpi_subsys_runtime_resume(struct device *dev);
>  int acpi_dev_pm_attach(struct device *dev, bool power_on);
> +bool acpi_storage_d3(struct device *dev);
>  #else
>  static inline int acpi_subsys_runtime_suspend(struct device *dev) { return 0; }
>  static inline int acpi_subsys_runtime_resume(struct device *dev) { return 0; }
> @@ -1011,6 +1012,10 @@ static inline int acpi_dev_pm_attach(struct device *dev, bool power_on)
>  {
>         return 0;
>  }
> +static inline bool acpi_storage_d3(struct device *dev)
> +{
> +       return false;
> +}
>  #endif
>
>  #if defined(CONFIG_ACPI) && defined(CONFIG_PM_SLEEP)
> --
> 2.25.1
>


From hare at suse.de  Mon Jun  7 06:34:35 2021
From: hare at suse.de (Hannes Reinecke)
Date: Mon,  7 Jun 2021 15:34:35 +0200
Subject: [PATCHv5] nvme: allow to re-attach namespaces after all paths are down
Message-ID: <20210607133435.138525-1-hare@suse.de>

We should only remove the ns head from the list of heads per
subsystem if the reference count drops to zero. Removing it
at the start of nvme_ns_remove() will prevent us from reattaching
the namespace to the correct ns head once a path becomes available
again.

Changes to v4:
- Call del_gendisk() in nvme_mpath_check_last_path() to avoid deadlock
Changes to v3:
- Simplify if() clause to detect duplicate namespaces
Changes to v2:
- Drop memcpy() statement
Changes to v1:
- Always check NSIDs after reattach

Signed-off-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/core.c      | 13 +++++++++----
 drivers/nvme/host/multipath.c | 12 ++++++++++--
 drivers/nvme/host/nvme.h      | 11 ++---------
 3 files changed, 21 insertions(+), 15 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 177cae44b612..065edf20bef4 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -566,6 +566,9 @@ static void nvme_free_ns_head(struct kref *ref)
 	struct nvme_ns_head *head =
 		container_of(ref, struct nvme_ns_head, ref);
 
+	mutex_lock(&head->subsys->lock);
+	list_del_init(&head->entry);
+	mutex_unlock(&head->subsys->lock);
 	nvme_mpath_remove_disk(head);
 	ida_simple_remove(&head->subsys->ns_ida, head->instance);
 	cleanup_srcu_struct(&head->srcu);
@@ -3669,7 +3672,11 @@ static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid,
 		head->shared = is_shared;
 	} else {
 		ret = -EINVAL;
-		if (!is_shared || !head->shared) {
+		/*
+		 * If multipath is enabled we might hit an ns head with no
+		 * paths, but that doesn't indicate it's a shared namespace.
+		 */
+		if (!list_empty(&head->list) && (!is_shared || !head->shared)) {
 			dev_err(ctrl->device,
 				"Duplicate unshared namespace %d\n", nsid);
 			goto out_put_ns_head;
@@ -3828,8 +3835,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	mutex_lock(&ns->ctrl->subsys->lock);
 	list_del_rcu(&ns->siblings);
-	if (list_empty(&ns->head->list))
-		list_del_init(&ns->head->entry);
 	mutex_unlock(&ns->ctrl->subsys->lock);
 
 	synchronize_rcu(); /* guarantee not available in head->list */
@@ -3849,7 +3854,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	list_del_init(&ns->list);
 	up_write(&ns->ctrl->namespaces_rwsem);
 
-	nvme_mpath_check_last_path(ns);
+	nvme_mpath_check_last_path(ns->head);
 	nvme_put_ns(ns);
 }
 
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 127a17b4c13d..de74650cdb2d 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -776,14 +776,22 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id)
 #endif
 }
 
-void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+void nvme_mpath_check_last_path(struct nvme_ns_head *head)
 {
 	if (!head->disk)
 		return;
-	if (head->disk->flags & GENHD_FL_UP) {
+
+	if (list_empty(&head->list) && (head->disk->flags & GENHD_FL_UP)) {
+		kblockd_schedule_work(&head->requeue_work);
 		nvme_cdev_del(&head->cdev, &head->cdev_device);
 		del_gendisk(head->disk);
 	}
+}
+
+void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+{
+	if (!head->disk)
+		return;
 	blk_set_queue_dying(head->disk->queue);
 	/* make sure all pending bios are cleaned up */
 	kblockd_schedule_work(&head->requeue_work);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1f397ecba16c..812fc1d273e3 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -716,14 +716,7 @@ void nvme_mpath_uninit(struct nvme_ctrl *ctrl);
 void nvme_mpath_stop(struct nvme_ctrl *ctrl);
 bool nvme_mpath_clear_current_path(struct nvme_ns *ns);
 void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl);
-
-static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)
-{
-	struct nvme_ns_head *head = ns->head;
-
-	if (head->disk && list_empty(&head->list))
-		kblockd_schedule_work(&head->requeue_work);
-}
+void nvme_mpath_check_last_path(struct nvme_ns_head *head);
 
 static inline void nvme_trace_bio_complete(struct request *req)
 {
@@ -772,7 +765,7 @@ static inline bool nvme_mpath_clear_current_path(struct nvme_ns *ns)
 static inline void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl)
 {
 }
-static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)
+static inline void nvme_mpath_check_last_path(struct nvme_ns_head *head)
 {
 }
 static inline void nvme_trace_bio_complete(struct request *req)
-- 
2.29.2



From kbusch at kernel.org  Mon Jun  7 07:17:47 2021
From: kbusch at kernel.org (Keith Busch)
Date: Mon, 7 Jun 2021 23:17:47 +0900
Subject: [PATCH v4] acpi: Move check for _DSD StorageD3Enable property to
 acpi
In-Reply-To: <20210603190753.5084-1-mario.limonciello@amd.com>
References: <20210603190753.5084-1-mario.limonciello@amd.com>
Message-ID: <20210607141747.GA21631@redsun51.ssa.fujisawa.hgst.com>

On Thu, Jun 03, 2021 at 02:07:53PM -0500, Mario Limonciello wrote:
> Although first implemented for NVME, this check may be usable by
> other drivers as well. Microsoft's specification explicitly mentions
> that is may be usable by SATA and AHCI devices.  Google also indicates
> that they have used this with SDHCI in a downstream kernel tree that
> a user can plug a storage device into.
> 
> Link: Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Suggested-by: Keith Busch <kbusch at kernel.org>
> CC: rrangel at chromium.org
> CC: david.e.box at linux.intel.com
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>

Looks good.

Reviewed-by: Keith Busch <kbusch at kernel.org>


From rafael at kernel.org  Mon Jun  7 07:36:35 2021
From: rafael at kernel.org (Rafael J. Wysocki)
Date: Mon, 7 Jun 2021 16:36:35 +0200
Subject: [PATCH v5 1/2] acpi: PM: Move check for _DSD StorageD3Enable
 property to acpi
In-Reply-To: <20210604165403.2317-1-mario.limonciello@amd.com>
References: <20210604165403.2317-1-mario.limonciello@amd.com>
Message-ID: <CAJZ5v0iKDwttHxFSB5tTMgyaOcvM40oq474BzwE9mfX28n+BJQ@mail.gmail.com>

On Fri, Jun 4, 2021 at 6:54 PM Mario Limonciello
<mario.limonciello at amd.com> wrote:
>
> Although first implemented for NVME, this check may be usable by
> other drivers as well. Microsoft's specification explicitly mentions
> that is may be usable by SATA and AHCI devices.  Google also indicates
> that they have used this with SDHCI in a downstream kernel tree that
> a user can plug a storage device into.
>
> Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Suggested-by: Keith Busch <kbusch at kernel.org>
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>

You've received some tags on the v4, can you please resend it with the
tags added?

> ---
>  drivers/acpi/device_pm.c | 25 +++++++++++++++++++++++++
>  drivers/nvme/host/pci.c  | 28 +---------------------------
>  include/linux/acpi.h     |  5 +++++
>  3 files changed, 31 insertions(+), 27 deletions(-)
>
> Changes from v4->v5:
>  * Correct extra "Link:" word in commit message
>
> diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
> index d260bc1f3e6e..1edb68d00b8e 100644
> --- a/drivers/acpi/device_pm.c
> +++ b/drivers/acpi/device_pm.c
> @@ -1340,4 +1340,29 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on)
>         return 1;
>  }
>  EXPORT_SYMBOL_GPL(acpi_dev_pm_attach);
> +
> +/**
> + * acpi_storage_d3 - Check if a storage device should use D3.
> + * @dev: Device to check
> + *
> + * Returns %true if @dev should be put into D3 when the ->suspend method is
> + * called, else %false.  The name of this function is somewhat misleading
> + * as it has nothing to do with storage except for the name of the ACPI
> + * property.  On some platforms resume will not work if this hint is ignored.
> + *
> + */
> +bool acpi_storage_d3(struct device *dev)
> +{
> +       struct acpi_device *adev = ACPI_COMPANION(dev);
> +       u8 val;
> +
> +       if (!adev)
> +               return false;
> +       if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> +                       &val))
> +               return false;
> +       return val == 1;
> +}
> +EXPORT_SYMBOL_GPL(acpi_storage_d3);
> +
>  #endif /* CONFIG_PM */
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 3aa7245a505f..8fbc4c87a0d8 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -2828,32 +2828,6 @@ static unsigned long check_vendor_combination_bug(struct pci_dev *pdev)
>         return 0;
>  }
>
> -#ifdef CONFIG_ACPI
> -static bool nvme_acpi_storage_d3(struct pci_dev *dev)
> -{
> -       struct acpi_device *adev = ACPI_COMPANION(&dev->dev);
> -       u8 val;
> -
> -       /*
> -        * Look for _DSD property specifying that the storage device on the port
> -        * must use D3 to support deep platform power savings during
> -        * suspend-to-idle.
> -        */
> -
> -       if (!adev)
> -               return false;
> -       if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> -                       &val))
> -               return false;
> -       return val == 1;
> -}
> -#else
> -static inline bool nvme_acpi_storage_d3(struct pci_dev *dev)
> -{
> -       return false;
> -}
> -#endif /* CONFIG_ACPI */
> -
>  static void nvme_async_probe(void *data, async_cookie_t cookie)
>  {
>         struct nvme_dev *dev = data;
> @@ -2903,7 +2877,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>
>         quirks |= check_vendor_combination_bug(pdev);
>
> -       if (!noacpi && nvme_acpi_storage_d3(pdev)) {
> +       if (!noacpi && acpi_storage_d3(&pdev->dev)) {
>                 /*
>                  * Some systems use a bios work around to ask for D3 on
>                  * platforms that support kernel managed suspend.
> diff --git a/include/linux/acpi.h b/include/linux/acpi.h
> index c60745f657e9..dd0dafd21e33 100644
> --- a/include/linux/acpi.h
> +++ b/include/linux/acpi.h
> @@ -1004,6 +1004,7 @@ int acpi_dev_resume(struct device *dev);
>  int acpi_subsys_runtime_suspend(struct device *dev);
>  int acpi_subsys_runtime_resume(struct device *dev);
>  int acpi_dev_pm_attach(struct device *dev, bool power_on);
> +bool acpi_storage_d3(struct device *dev);
>  #else
>  static inline int acpi_subsys_runtime_suspend(struct device *dev) { return 0; }
>  static inline int acpi_subsys_runtime_resume(struct device *dev) { return 0; }
> @@ -1011,6 +1012,10 @@ static inline int acpi_dev_pm_attach(struct device *dev, bool power_on)
>  {
>         return 0;
>  }
> +static inline bool acpi_storage_d3(struct device *dev)
> +{
> +       return false;
> +}
>  #endif
>
>  #if defined(CONFIG_ACPI) && defined(CONFIG_PM_SLEEP)
> --
> 2.25.1
>


From rafael at kernel.org  Mon Jun  7 07:39:26 2021
From: rafael at kernel.org (Rafael J. Wysocki)
Date: Mon, 7 Jun 2021 16:39:26 +0200
Subject: [PATCH v5 2/2] acpi: PM: Add quirks for AMD Renoir/Lucienne CPUs
 to force the D3 hint
In-Reply-To: <20210604165403.2317-2-mario.limonciello@amd.com>
References: <20210604165403.2317-1-mario.limonciello@amd.com>
 <20210604165403.2317-2-mario.limonciello@amd.com>
Message-ID: <CAJZ5v0g5KCkE4eRSjKZgX14Nm3FNKQmfzsCYzfigD-M-uT5QJA@mail.gmail.com>

On Fri, Jun 4, 2021 at 6:54 PM Mario Limonciello
<mario.limonciello at amd.com> wrote:
>
> AMD systems from Renoir and Lucienne require that the NVME controller
> is put into D3 over a Modern Standby / suspend-to-idle
> cycle.  This is "typically" accomplished using the `StorageD3Enable`
> property in the _DSD, but this property was introduced after many
> of these systems launched and most OEM systems don't have it in
> their BIOS.
>
> On AMD Renoir without these drives going into D3 over suspend-to-idle
> the resume will fail with the NVME controller being reset and a trace
> like this in the kernel logs:
> ```
> [   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
> [   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
> [   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
> [   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
> [   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
> [   95.332843] nvme nvme0: Abort status: 0x371
> [   95.332852] nvme nvme0: Abort status: 0x371
> [   95.332856] nvme nvme0: Abort status: 0x371
> [   95.332859] nvme nvme0: Abort status: 0x371
> [   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
> [   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
> ```
>
> The Microsoft documentation for StorageD3Enable mentioned that Windows has
> a hardcoded allowlist for D3 support, which was used for these platforms.
> Introduce quirks to hardcode them for Linux as well.
>
> As this property is now "standardized", OEM systems using AMD Cezanne and
> newer APU's have adopted this property, and quirks like this should not be
> necessary.
>
> CC: Julian Sikorski <belegdol at gmail.com>
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> ---
>  drivers/acpi/device_pm.c | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
>
> Changes from v4->v5:
>  * Add this patch back in as it's been made apparent that the
>    system needs to be hardcoded for these.
>    Changes:
>    - Drop Cezanne - it's now covered by StorageD3Enable
>    - Rebase ontop of acpi_storage_d3 outside of NVME
>
> diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
> index 1edb68d00b8e..8fd2a15bf478 100644
> --- a/drivers/acpi/device_pm.c
> +++ b/drivers/acpi/device_pm.c
> @@ -20,6 +20,10 @@
>  #include <linux/pm_runtime.h>
>  #include <linux/suspend.h>
>
> +#ifdef CONFIG_X86
> +#include <asm/cpu_device_id.h>
> +#endif

This is a generic file, not x86 (or any other arch-specific)
#ifdeffery in it, please.

There is the x86/ subdir under drivers/acpi/ for x86-specific stuff.

> +
>  #include "internal.h"
>
>  /**
> @@ -1341,6 +1345,15 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on)
>  }
>  EXPORT_SYMBOL_GPL(acpi_dev_pm_attach);
>
> +
> +#ifdef CONFIG_X86
> +static const struct x86_cpu_id storage_d3_cpu_ids[] = {
> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),  /* Renoir */
> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL), /* Lucienne */
> +       {}
> +};
> +#endif
> +
>  /**
>   * acpi_storage_d3 - Check if a storage device should use D3.
>   * @dev: Device to check
> @@ -1356,6 +1369,12 @@ bool acpi_storage_d3(struct device *dev)
>         struct acpi_device *adev = ACPI_COMPANION(dev);
>         u8 val;
>
> +#ifdef CONFIG_X86
> +       /* Devices requiring D3, but from before StorageD3Enable was "standardized" */
> +       if (x86_match_cpu(storage_d3_cpu_ids))
> +               return true;
> +#endif
> +
>         if (!adev)
>                 return false;
>         if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> --
> 2.25.1
>


From hare at suse.de  Mon Jun  7 08:46:09 2021
From: hare at suse.de (Hannes Reinecke)
Date: Mon, 7 Jun 2021 17:46:09 +0200
Subject: [PATCH 5/8] nvme-tcp-offload: Add controller level implementation
In-Reply-To: <20210602184246.14184-6-smalin@marvell.com>
References: <20210602184246.14184-1-smalin@marvell.com>
 <20210602184246.14184-6-smalin@marvell.com>
Message-ID: <87a2822c-c5ac-13d1-336a-b96045c3de5b@suse.de>

On 6/2/21 8:42 PM, Shai Malin wrote:
> From: Arie Gershberg <agershberg at marvell.com>
> 
> In this patch we implement controller level functionality including:
> - create_ctrl.
> - delete_ctrl.
> - free_ctrl.
> 
> The implementation is similar to other nvme fabrics modules, the main
> difference being that the nvme-tcp-offload ULP calls the vendor specific
> claim_dev() op with the given TCP/IP parameters to determine which device
> will be used for this controller.
> Once found, the vendor specific device and controller will be paired and
> kept in a controller list managed by the ULP.
> 
> Acked-by: Igor Russkikh <irusskikh at marvell.com>
> Signed-off-by: Arie Gershberg <agershberg at marvell.com>
> Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
> Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
> Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
> Signed-off-by: Ariel Elior <aelior at marvell.com>
> Signed-off-by: Shai Malin <smalin at marvell.com>
> Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
> ---
>  drivers/nvme/host/tcp-offload.c | 481 +++++++++++++++++++++++++++++++-
>  1 file changed, 476 insertions(+), 5 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		        Kernel Storage Architect
hare at suse.de			               +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 N?rnberg
GF: F. Imend?rffer, HRB 36809 (AG N?rnberg)


From hare at suse.de  Mon Jun  7 08:49:46 2021
From: hare at suse.de (Hannes Reinecke)
Date: Mon, 7 Jun 2021 17:49:46 +0200
Subject: [PATCH 7/8] nvme-tcp-offload: Add queue level implementation
In-Reply-To: <20210602184246.14184-8-smalin@marvell.com>
References: <20210602184246.14184-1-smalin@marvell.com>
 <20210602184246.14184-8-smalin@marvell.com>
Message-ID: <68d83513-aaf1-4956-367e-ab69ca1d37ce@suse.de>

On 6/2/21 8:42 PM, Shai Malin wrote:
> From: Dean Balandin <dbalandin at marvell.com>
> 
> In this patch we implement queue level functionality.
> The implementation is similar to the nvme-tcp module, the main
> difference being that we call the vendor specific create_queue op which
> creates the TCP connection, and NVMeTPC connection including
> icreq+icresp negotiation.
> Once create_queue returns successfully, we can move on to the fabrics
> connect.
> 
> Acked-by: Igor Russkikh <irusskikh at marvell.com>
> Signed-off-by: Dean Balandin <dbalandin at marvell.com>
> Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
> Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
> Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
> Signed-off-by: Ariel Elior <aelior at marvell.com>
> Signed-off-by: Shai Malin <smalin at marvell.com>
> Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
> ---
>  drivers/nvme/host/tcp-offload.c | 417 +++++++++++++++++++++++++++++---
>  drivers/nvme/host/tcp-offload.h |   4 +
>  2 files changed, 393 insertions(+), 28 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		        Kernel Storage Architect
hare at suse.de			               +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 N?rnberg
GF: F. Imend?rffer, HRB 36809 (AG N?rnberg)


From sashal at kernel.org  Mon Jun  7 09:12:03 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:12:03 -0400
Subject: [PATCH AUTOSEL 5.12 37/49] nvme-loop: clear NVME_LOOP_Q_LIVE when
 nvme_loop_configure_admin_queue() fails
In-Reply-To: <20210607161215.3583176-1-sashal@kernel.org>
References: <20210607161215.3583176-1-sashal@kernel.org>
Message-ID: <20210607161215.3583176-37-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 1c5f8e882a05de5c011e8c3fbeceb0d1c590eb53 ]

When the call to nvme_enable_ctrl() in nvme_loop_configure_admin_queue()
fails the NVME_LOOP_Q_LIVE flag is not cleared.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 4b2a6330feb5..c34f785e699d 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -404,6 +404,7 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
 	return 0;
 
 out_cleanup_queue:
+	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 out_cleanup_fabrics_q:
 	blk_cleanup_queue(ctrl->ctrl.fabrics_q);
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:12:02 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:12:02 -0400
Subject: [PATCH AUTOSEL 5.12 36/49] nvme-loop: reset queue count to 1 in
 nvme_loop_destroy_io_queues()
In-Reply-To: <20210607161215.3583176-1-sashal@kernel.org>
References: <20210607161215.3583176-1-sashal@kernel.org>
Message-ID: <20210607161215.3583176-36-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit a6c144f3d2e230f2b3ac5ed8c51e0f0391556197 ]

The queue count is increased in nvme_loop_init_io_queues(), so we
need to reset it to 1 at the end of nvme_loop_destroy_io_queues().
Otherwise the function is not re-entrant safe, and crash will happen
during concurrent reset and remove calls.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 14913a4588ec..4b2a6330feb5 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -297,6 +297,7 @@ static void nvme_loop_destroy_io_queues(struct nvme_loop_ctrl *ctrl)
 		clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[i].flags);
 		nvmet_sq_destroy(&ctrl->queues[i].nvme_sq);
 	}
+	ctrl->ctrl.queue_count = 1;
 }
 
 static int nvme_loop_init_io_queues(struct nvme_loop_ctrl *ctrl)
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:12:04 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:12:04 -0400
Subject: [PATCH AUTOSEL 5.12 38/49] nvme-loop: check for NVME_LOOP_Q_LIVE in
 nvme_loop_destroy_admin_queue()
In-Reply-To: <20210607161215.3583176-1-sashal@kernel.org>
References: <20210607161215.3583176-1-sashal@kernel.org>
Message-ID: <20210607161215.3583176-38-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 4237de2f73a669e4f89ac0aa2b44fb1a1d9ec583 ]

We need to check the NVME_LOOP_Q_LIVE flag in
nvme_loop_destroy_admin_queue() to protect against duplicate
invocations eg during concurrent reset and remove calls.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index c34f785e699d..fe14609d2254 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -261,7 +261,8 @@ static const struct blk_mq_ops nvme_loop_admin_mq_ops = {
 
 static void nvme_loop_destroy_admin_queue(struct nvme_loop_ctrl *ctrl)
 {
-	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
+	if (!test_and_clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags))
+		return;
 	nvmet_sq_destroy(&ctrl->queues[0].nvme_sq);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 	blk_cleanup_queue(ctrl->ctrl.fabrics_q);
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:12:05 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:12:05 -0400
Subject: [PATCH AUTOSEL 5.12 39/49] nvme-loop: do not warn for deleted
 controllers during reset
In-Reply-To: <20210607161215.3583176-1-sashal@kernel.org>
References: <20210607161215.3583176-1-sashal@kernel.org>
Message-ID: <20210607161215.3583176-39-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 6622f9acd29cd4f6272720e827e6406f5a970cb0 ]

During concurrent reset and delete calls the reset workqueue is
flushed, causing nvme_loop_reset_ctrl_work() to be executed when
the controller is in state DELETING or DELETING_NOIO.
But this is expected, so we shouldn't issue a WARN_ON here.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index fe14609d2254..0f22f333ff24 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -463,8 +463,10 @@ static void nvme_loop_reset_ctrl_work(struct work_struct *work)
 	nvme_loop_shutdown_ctrl(ctrl);
 
 	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {
-		/* state change failure should never happen */
-		WARN_ON_ONCE(1);
+		if (ctrl->ctrl.state != NVME_CTRL_DELETING &&
+		    ctrl->ctrl.state != NVME_CTRL_DELETING_NOIO)
+			/* state change failure for non-deleted ctrl? */
+			WARN_ON_ONCE(1);
 		return;
 	}
 
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:13:05 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:13:05 -0400
Subject: [PATCH AUTOSEL 5.10 26/39] nvme-loop: reset queue count to 1 in
 nvme_loop_destroy_io_queues()
In-Reply-To: <20210607161318.3583636-1-sashal@kernel.org>
References: <20210607161318.3583636-1-sashal@kernel.org>
Message-ID: <20210607161318.3583636-26-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit a6c144f3d2e230f2b3ac5ed8c51e0f0391556197 ]

The queue count is increased in nvme_loop_init_io_queues(), so we
need to reset it to 1 at the end of nvme_loop_destroy_io_queues().
Otherwise the function is not re-entrant safe, and crash will happen
during concurrent reset and remove calls.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index b869b686e962..1d3185c82596 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -287,6 +287,7 @@ static void nvme_loop_destroy_io_queues(struct nvme_loop_ctrl *ctrl)
 		clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[i].flags);
 		nvmet_sq_destroy(&ctrl->queues[i].nvme_sq);
 	}
+	ctrl->ctrl.queue_count = 1;
 }
 
 static int nvme_loop_init_io_queues(struct nvme_loop_ctrl *ctrl)
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:13:06 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:13:06 -0400
Subject: [PATCH AUTOSEL 5.10 27/39] nvme-loop: clear NVME_LOOP_Q_LIVE when
 nvme_loop_configure_admin_queue() fails
In-Reply-To: <20210607161318.3583636-1-sashal@kernel.org>
References: <20210607161318.3583636-1-sashal@kernel.org>
Message-ID: <20210607161318.3583636-27-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 1c5f8e882a05de5c011e8c3fbeceb0d1c590eb53 ]

When the call to nvme_enable_ctrl() in nvme_loop_configure_admin_queue()
fails the NVME_LOOP_Q_LIVE flag is not cleared.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 1d3185c82596..c07b4a14d477 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -394,6 +394,7 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
 	return 0;
 
 out_cleanup_queue:
+	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 out_cleanup_fabrics_q:
 	blk_cleanup_queue(ctrl->ctrl.fabrics_q);
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:13:07 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:13:07 -0400
Subject: [PATCH AUTOSEL 5.10 28/39] nvme-loop: check for NVME_LOOP_Q_LIVE in
 nvme_loop_destroy_admin_queue()
In-Reply-To: <20210607161318.3583636-1-sashal@kernel.org>
References: <20210607161318.3583636-1-sashal@kernel.org>
Message-ID: <20210607161318.3583636-28-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 4237de2f73a669e4f89ac0aa2b44fb1a1d9ec583 ]

We need to check the NVME_LOOP_Q_LIVE flag in
nvme_loop_destroy_admin_queue() to protect against duplicate
invocations eg during concurrent reset and remove calls.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index c07b4a14d477..df0e5288ae6e 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -251,7 +251,8 @@ static const struct blk_mq_ops nvme_loop_admin_mq_ops = {
 
 static void nvme_loop_destroy_admin_queue(struct nvme_loop_ctrl *ctrl)
 {
-	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
+	if (!test_and_clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags))
+		return;
 	nvmet_sq_destroy(&ctrl->queues[0].nvme_sq);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 	blk_cleanup_queue(ctrl->ctrl.fabrics_q);
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:13:08 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:13:08 -0400
Subject: [PATCH AUTOSEL 5.10 29/39] nvme-loop: do not warn for deleted
 controllers during reset
In-Reply-To: <20210607161318.3583636-1-sashal@kernel.org>
References: <20210607161318.3583636-1-sashal@kernel.org>
Message-ID: <20210607161318.3583636-29-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 6622f9acd29cd4f6272720e827e6406f5a970cb0 ]

During concurrent reset and delete calls the reset workqueue is
flushed, causing nvme_loop_reset_ctrl_work() to be executed when
the controller is in state DELETING or DELETING_NOIO.
But this is expected, so we shouldn't issue a WARN_ON here.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index df0e5288ae6e..16d71cc5a50e 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -453,8 +453,10 @@ static void nvme_loop_reset_ctrl_work(struct work_struct *work)
 	nvme_loop_shutdown_ctrl(ctrl);
 
 	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {
-		/* state change failure should never happen */
-		WARN_ON_ONCE(1);
+		if (ctrl->ctrl.state != NVME_CTRL_DELETING &&
+		    ctrl->ctrl.state != NVME_CTRL_DELETING_NOIO)
+			/* state change failure for non-deleted ctrl? */
+			WARN_ON_ONCE(1);
 		return;
 	}
 
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:14:01 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:14:01 -0400
Subject: [PATCH AUTOSEL 5.4 20/29] nvme-loop: reset queue count to 1 in
 nvme_loop_destroy_io_queues()
In-Reply-To: <20210607161410.3584036-1-sashal@kernel.org>
References: <20210607161410.3584036-1-sashal@kernel.org>
Message-ID: <20210607161410.3584036-20-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit a6c144f3d2e230f2b3ac5ed8c51e0f0391556197 ]

The queue count is increased in nvme_loop_init_io_queues(), so we
need to reset it to 1 at the end of nvme_loop_destroy_io_queues().
Otherwise the function is not re-entrant safe, and crash will happen
during concurrent reset and remove calls.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 82b87a4c50f6..b4f5503ae570 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -288,6 +288,7 @@ static void nvme_loop_destroy_io_queues(struct nvme_loop_ctrl *ctrl)
 		clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[i].flags);
 		nvmet_sq_destroy(&ctrl->queues[i].nvme_sq);
 	}
+	ctrl->ctrl.queue_count = 1;
 }
 
 static int nvme_loop_init_io_queues(struct nvme_loop_ctrl *ctrl)
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:14:02 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:14:02 -0400
Subject: [PATCH AUTOSEL 5.4 21/29] nvme-loop: clear NVME_LOOP_Q_LIVE when
 nvme_loop_configure_admin_queue() fails
In-Reply-To: <20210607161410.3584036-1-sashal@kernel.org>
References: <20210607161410.3584036-1-sashal@kernel.org>
Message-ID: <20210607161410.3584036-21-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 1c5f8e882a05de5c011e8c3fbeceb0d1c590eb53 ]

When the call to nvme_enable_ctrl() in nvme_loop_configure_admin_queue()
fails the NVME_LOOP_Q_LIVE flag is not cleared.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index b4f5503ae570..f752e9432676 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -395,6 +395,7 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
 	return 0;
 
 out_cleanup_queue:
+	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 out_cleanup_fabrics_q:
 	blk_cleanup_queue(ctrl->ctrl.fabrics_q);
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:14:03 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:14:03 -0400
Subject: [PATCH AUTOSEL 5.4 22/29] nvme-loop: check for NVME_LOOP_Q_LIVE in
 nvme_loop_destroy_admin_queue()
In-Reply-To: <20210607161410.3584036-1-sashal@kernel.org>
References: <20210607161410.3584036-1-sashal@kernel.org>
Message-ID: <20210607161410.3584036-22-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 4237de2f73a669e4f89ac0aa2b44fb1a1d9ec583 ]

We need to check the NVME_LOOP_Q_LIVE flag in
nvme_loop_destroy_admin_queue() to protect against duplicate
invocations eg during concurrent reset and remove calls.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index f752e9432676..f657a12bf1fe 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -252,7 +252,8 @@ static const struct blk_mq_ops nvme_loop_admin_mq_ops = {
 
 static void nvme_loop_destroy_admin_queue(struct nvme_loop_ctrl *ctrl)
 {
-	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
+	if (!test_and_clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags))
+		return;
 	nvmet_sq_destroy(&ctrl->queues[0].nvme_sq);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 	blk_cleanup_queue(ctrl->ctrl.fabrics_q);
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:14:41 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:14:41 -0400
Subject: [PATCH AUTOSEL 4.19 14/21] nvme-loop: reset queue count to 1 in
 nvme_loop_destroy_io_queues()
In-Reply-To: <20210607161448.3584332-1-sashal@kernel.org>
References: <20210607161448.3584332-1-sashal@kernel.org>
Message-ID: <20210607161448.3584332-14-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit a6c144f3d2e230f2b3ac5ed8c51e0f0391556197 ]

The queue count is increased in nvme_loop_init_io_queues(), so we
need to reset it to 1 at the end of nvme_loop_destroy_io_queues().
Otherwise the function is not re-entrant safe, and crash will happen
during concurrent reset and remove calls.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 137a27fa369c..7b6e44ed299a 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -309,6 +309,7 @@ static void nvme_loop_destroy_io_queues(struct nvme_loop_ctrl *ctrl)
 		clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[i].flags);
 		nvmet_sq_destroy(&ctrl->queues[i].nvme_sq);
 	}
+	ctrl->ctrl.queue_count = 1;
 }
 
 static int nvme_loop_init_io_queues(struct nvme_loop_ctrl *ctrl)
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:14:42 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:14:42 -0400
Subject: [PATCH AUTOSEL 4.19 15/21] nvme-loop: clear NVME_LOOP_Q_LIVE when
 nvme_loop_configure_admin_queue() fails
In-Reply-To: <20210607161448.3584332-1-sashal@kernel.org>
References: <20210607161448.3584332-1-sashal@kernel.org>
Message-ID: <20210607161448.3584332-15-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 1c5f8e882a05de5c011e8c3fbeceb0d1c590eb53 ]

When the call to nvme_enable_ctrl() in nvme_loop_configure_admin_queue()
fails the NVME_LOOP_Q_LIVE flag is not cleared.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 7b6e44ed299a..dba0b0145f48 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -418,6 +418,7 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
 	return 0;
 
 out_cleanup_queue:
+	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 out_free_tagset:
 	blk_mq_free_tag_set(&ctrl->admin_tag_set);
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:14:43 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:14:43 -0400
Subject: [PATCH AUTOSEL 4.19 16/21] nvme-loop: check for NVME_LOOP_Q_LIVE in
 nvme_loop_destroy_admin_queue()
In-Reply-To: <20210607161448.3584332-1-sashal@kernel.org>
References: <20210607161448.3584332-1-sashal@kernel.org>
Message-ID: <20210607161448.3584332-16-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 4237de2f73a669e4f89ac0aa2b44fb1a1d9ec583 ]

We need to check the NVME_LOOP_Q_LIVE flag in
nvme_loop_destroy_admin_queue() to protect against duplicate
invocations eg during concurrent reset and remove calls.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index dba0b0145f48..08b52f3ed0c3 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -274,7 +274,8 @@ static const struct blk_mq_ops nvme_loop_admin_mq_ops = {
 
 static void nvme_loop_destroy_admin_queue(struct nvme_loop_ctrl *ctrl)
 {
-	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
+	if (!test_and_clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags))
+		return;
 	nvmet_sq_destroy(&ctrl->queues[0].nvme_sq);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 	blk_mq_free_tag_set(&ctrl->admin_tag_set);
-- 
2.30.2



From kbusch at kernel.org  Mon Jun  7 09:15:12 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 8 Jun 2021 01:15:12 +0900
Subject: [PATCHv3 2/4] nvme: use blk_execute_rq() for passthrough commands
In-Reply-To: <CA+1E3rJM3sGLsOuPbYVH6kaTR8S9ogfe+wryuyWpqnA-pgDsEw@mail.gmail.com>
References: <20210521202145.3674904-1-kbusch@kernel.org>
 <20210521202145.3674904-3-kbusch@kernel.org>
 <CA+1E3rJM3sGLsOuPbYVH6kaTR8S9ogfe+wryuyWpqnA-pgDsEw@mail.gmail.com>
Message-ID: <20210607161512.GB21631@redsun51.ssa.fujisawa.hgst.com>

On Wed, May 26, 2021 at 02:17:23PM +0530, Kanchan Joshi wrote:
> On Sat, May 22, 2021 at 2:05 AM Keith Busch <kbusch at kernel.org> wrote:
> >
> > The generic blk_execute_rq() knows how to handle polled completions. Use
> > that instead of implementing an nvme specific handler.
> >
> > Signed-off-by: Keith Busch <kbusch at kernel.org>
> > ---
> > No changes since v2
> >
> >  drivers/nvme/host/core.c    | 38 +++++--------------------------------
> >  drivers/nvme/host/fabrics.c | 13 ++++++-------
> >  drivers/nvme/host/fabrics.h |  2 +-
> >  drivers/nvme/host/fc.c      |  2 +-
> >  drivers/nvme/host/nvme.h    |  2 +-
> >  drivers/nvme/host/rdma.c    |  3 +--
> >  drivers/nvme/host/tcp.c     |  2 +-
> >  drivers/nvme/target/loop.c  |  2 +-
> >  8 files changed, 17 insertions(+), 47 deletions(-)
> >
> > diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> > index 762125f2905f..1a73eed61eee 100644
> > --- a/drivers/nvme/host/core.c
> > +++ b/drivers/nvme/host/core.c
> > @@ -1012,31 +1012,6 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req)
> >  }
> >  EXPORT_SYMBOL_GPL(nvme_setup_cmd);
> >
> > -static void nvme_end_sync_rq(struct request *rq, blk_status_t error)
> > -{
> > -       struct completion *waiting = rq->end_io_data;
> > -
> > -       rq->end_io_data = NULL;
> > -       complete(waiting);
> > -}
> > -
> > -static void nvme_execute_rq_polled(struct request_queue *q,
> > -               struct gendisk *bd_disk, struct request *rq, int at_head)
> > -{
> > -       DECLARE_COMPLETION_ONSTACK(wait);
> > -
> > -       WARN_ON_ONCE(!test_bit(QUEUE_FLAG_POLL, &q->queue_flags));
> > -
> > -       rq->cmd_flags |= REQ_HIPRI;
> 
> The new code doesn't retain this flag.
> Looks good otherwise.

The flag is only used to select an apporpriate hctx. We've explicitly
selected a polling hctx in this path already, so the flag is
unnecessary.


From sashal at kernel.org  Mon Jun  7 09:15:09 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:15:09 -0400
Subject: [PATCH AUTOSEL 4.14 11/18] nvme-loop: reset queue count to 1 in
 nvme_loop_destroy_io_queues()
In-Reply-To: <20210607161517.3584577-1-sashal@kernel.org>
References: <20210607161517.3584577-1-sashal@kernel.org>
Message-ID: <20210607161517.3584577-11-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit a6c144f3d2e230f2b3ac5ed8c51e0f0391556197 ]

The queue count is increased in nvme_loop_init_io_queues(), so we
need to reset it to 1 at the end of nvme_loop_destroy_io_queues().
Otherwise the function is not re-entrant safe, and crash will happen
during concurrent reset and remove calls.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 3388d2788fe0..5f33c3a9469b 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -322,6 +322,7 @@ static void nvme_loop_destroy_io_queues(struct nvme_loop_ctrl *ctrl)
 		clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[i].flags);
 		nvmet_sq_destroy(&ctrl->queues[i].nvme_sq);
 	}
+	ctrl->ctrl.queue_count = 1;
 }
 
 static int nvme_loop_init_io_queues(struct nvme_loop_ctrl *ctrl)
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:15:10 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:15:10 -0400
Subject: [PATCH AUTOSEL 4.14 12/18] nvme-loop: clear NVME_LOOP_Q_LIVE when
 nvme_loop_configure_admin_queue() fails
In-Reply-To: <20210607161517.3584577-1-sashal@kernel.org>
References: <20210607161517.3584577-1-sashal@kernel.org>
Message-ID: <20210607161517.3584577-12-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 1c5f8e882a05de5c011e8c3fbeceb0d1c590eb53 ]

When the call to nvme_enable_ctrl() in nvme_loop_configure_admin_queue()
fails the NVME_LOOP_Q_LIVE flag is not cleared.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 5f33c3a9469b..963d8de932d1 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -430,6 +430,7 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
 	return 0;
 
 out_cleanup_queue:
+	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 out_free_tagset:
 	blk_mq_free_tag_set(&ctrl->admin_tag_set);
-- 
2.30.2



From sashal at kernel.org  Mon Jun  7 09:15:11 2021
From: sashal at kernel.org (Sasha Levin)
Date: Mon,  7 Jun 2021 12:15:11 -0400
Subject: [PATCH AUTOSEL 4.14 13/18] nvme-loop: check for NVME_LOOP_Q_LIVE in
 nvme_loop_destroy_admin_queue()
In-Reply-To: <20210607161517.3584577-1-sashal@kernel.org>
References: <20210607161517.3584577-1-sashal@kernel.org>
Message-ID: <20210607161517.3584577-13-sashal@kernel.org>

From: Hannes Reinecke <hare at suse.de>

[ Upstream commit 4237de2f73a669e4f89ac0aa2b44fb1a1d9ec583 ]

We need to check the NVME_LOOP_Q_LIVE flag in
nvme_loop_destroy_admin_queue() to protect against duplicate
invocations eg during concurrent reset and remove calls.

Signed-off-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Sasha Levin <sashal at kernel.org>
---
 drivers/nvme/target/loop.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 963d8de932d1..7a0a10777cd1 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -287,7 +287,8 @@ static const struct blk_mq_ops nvme_loop_admin_mq_ops = {
 
 static void nvme_loop_destroy_admin_queue(struct nvme_loop_ctrl *ctrl)
 {
-	clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
+	if (!test_and_clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags))
+		return;
 	nvmet_sq_destroy(&ctrl->queues[0].nvme_sq);
 	blk_cleanup_queue(ctrl->ctrl.admin_q);
 	blk_mq_free_tag_set(&ctrl->admin_tag_set);
-- 
2.30.2



From kbusch at kernel.org  Mon Jun  7 09:58:27 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 8 Jun 2021 01:58:27 +0900
Subject: [PATCHv3 4/4] nvme: use return value from blk_execute_rq()
In-Reply-To: <20210524080428.GA24488@lst.de>
References: <20210521202145.3674904-1-kbusch@kernel.org>
 <20210521202145.3674904-5-kbusch@kernel.org>
 <20210524080428.GA24488@lst.de>
Message-ID: <20210607165827.GC21631@redsun51.ssa.fujisawa.hgst.com>

On Mon, May 24, 2021 at 10:04:28AM +0200, Christoph Hellwig wrote:
> > @@ -168,7 +167,8 @@ static void nvmet_passthru_execute_cmd_work(struct work_struct *w)
> >  			nvmet_passthru_override_id_ns(req);
> >  			break;
> >  		}
> > -	}
> > +	} else if (status < 0)
> > +		status = NVME_SC_INTERNAL;
> 
> Don't we need a better translation here?

Did you have something in mind? I couldn't think of anything more
appropriate than the generic internal error. The errno's we get here are
-EINTR or -EIO. Both indicate we can't communicate with the back-end
device, but these problems are internal to the target from the host's
perspective.


From mario.limonciello at amd.com  Mon Jun  7 10:31:55 2021
From: mario.limonciello at amd.com (Mario Limonciello)
Date: Mon,  7 Jun 2021 12:31:55 -0500
Subject: [PATCH v6 1/2] ACPI: Move check for _DSD StorageD3Enable property to
 acpi
Message-ID: <20210607173156.5548-1-mario.limonciello@amd.com>

Although first implemented for NVME, this check may be usable by
other drivers as well. Microsoft's specification explicitly mentions
that is may be usable by SATA and AHCI devices.  Google also indicates
that they have used this with SDHCI in a downstream kernel tree that
a user can plug a storage device into.

Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
Suggested-by: Keith Busch <kbusch at kernel.org>
CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
CC: Alexander Deucher <Alexander.Deucher at amd.com>
CC: Rafael J. Wysocki <rjw at rjwysocki.net>
CC: Prike Liang <prike.liang at amd.com>
Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
Acked-by: Raul E Rangel <rrangel at chromium.org>
Acked-by: Rafael J. Wysocki <rafael.j.wysocki at intel.com>
Reviewed-by: Keith Busch <kbusch at kernel.org>
---
 drivers/acpi/device_pm.c | 25 +++++++++++++++++++++++++
 drivers/nvme/host/pci.c  | 28 +---------------------------
 include/linux/acpi.h     |  5 +++++
 3 files changed, 31 insertions(+), 27 deletions(-)

Changes from v4->v5:
 * Correct extra "Link:" word in commit message
Changes from v5->v6:
 * Add in commit message tags from Raul, Rafael and Keith

diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index d260bc1f3e6e..1edb68d00b8e 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -1340,4 +1340,29 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on)
 	return 1;
 }
 EXPORT_SYMBOL_GPL(acpi_dev_pm_attach);
+
+/**
+ * acpi_storage_d3 - Check if a storage device should use D3.
+ * @dev: Device to check
+ *
+ * Returns %true if @dev should be put into D3 when the ->suspend method is
+ * called, else %false.  The name of this function is somewhat misleading
+ * as it has nothing to do with storage except for the name of the ACPI
+ * property.  On some platforms resume will not work if this hint is ignored.
+ *
+ */
+bool acpi_storage_d3(struct device *dev)
+{
+	struct acpi_device *adev = ACPI_COMPANION(dev);
+	u8 val;
+
+	if (!adev)
+		return false;
+	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
+			&val))
+		return false;
+	return val == 1;
+}
+EXPORT_SYMBOL_GPL(acpi_storage_d3);
+
 #endif /* CONFIG_PM */
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..8fbc4c87a0d8 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2828,32 +2828,6 @@ static unsigned long check_vendor_combination_bug(struct pci_dev *pdev)
 	return 0;
 }
 
-#ifdef CONFIG_ACPI
-static bool nvme_acpi_storage_d3(struct pci_dev *dev)
-{
-	struct acpi_device *adev = ACPI_COMPANION(&dev->dev);
-	u8 val;
-
-	/*
-	 * Look for _DSD property specifying that the storage device on the port
-	 * must use D3 to support deep platform power savings during
-	 * suspend-to-idle.
-	 */
-
-	if (!adev)
-		return false;
-	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
-			&val))
-		return false;
-	return val == 1;
-}
-#else
-static inline bool nvme_acpi_storage_d3(struct pci_dev *dev)
-{
-	return false;
-}
-#endif /* CONFIG_ACPI */
-
 static void nvme_async_probe(void *data, async_cookie_t cookie)
 {
 	struct nvme_dev *dev = data;
@@ -2903,7 +2877,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	quirks |= check_vendor_combination_bug(pdev);
 
-	if (!noacpi && nvme_acpi_storage_d3(pdev)) {
+	if (!noacpi && acpi_storage_d3(&pdev->dev)) {
 		/*
 		 * Some systems use a bios work around to ask for D3 on
 		 * platforms that support kernel managed suspend.
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index c60745f657e9..dd0dafd21e33 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -1004,6 +1004,7 @@ int acpi_dev_resume(struct device *dev);
 int acpi_subsys_runtime_suspend(struct device *dev);
 int acpi_subsys_runtime_resume(struct device *dev);
 int acpi_dev_pm_attach(struct device *dev, bool power_on);
+bool acpi_storage_d3(struct device *dev);
 #else
 static inline int acpi_subsys_runtime_suspend(struct device *dev) { return 0; }
 static inline int acpi_subsys_runtime_resume(struct device *dev) { return 0; }
@@ -1011,6 +1012,10 @@ static inline int acpi_dev_pm_attach(struct device *dev, bool power_on)
 {
 	return 0;
 }
+static inline bool acpi_storage_d3(struct device *dev)
+{
+	return false;
+}
 #endif
 
 #if defined(CONFIG_ACPI) && defined(CONFIG_PM_SLEEP)
-- 
2.25.1



From mario.limonciello at amd.com  Mon Jun  7 10:31:56 2021
From: mario.limonciello at amd.com (Mario Limonciello)
Date: Mon,  7 Jun 2021 12:31:56 -0500
Subject: [PATCH v6 2/2] ACPI: Add quirks for AMD Renoir/Lucienne CPUs to force
 the D3 hint
In-Reply-To: <20210607173156.5548-1-mario.limonciello@amd.com>
References: <20210607173156.5548-1-mario.limonciello@amd.com>
Message-ID: <20210607173156.5548-2-mario.limonciello@amd.com>

AMD systems from Renoir and Lucienne require that the NVME controller
is put into D3 over a Modern Standby / suspend-to-idle
cycle.  This is "typically" accomplished using the `StorageD3Enable`
property in the _DSD, but this property was introduced after many
of these systems launched and most OEM systems don't have it in
their BIOS.

On AMD Renoir without these drives going into D3 over suspend-to-idle
the resume will fail with the NVME controller being reset and a trace
like this in the kernel logs:
```
[   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
[   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
[   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
[   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
[   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
[   95.332843] nvme nvme0: Abort status: 0x371
[   95.332852] nvme nvme0: Abort status: 0x371
[   95.332856] nvme nvme0: Abort status: 0x371
[   95.332859] nvme nvme0: Abort status: 0x371
[   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
[   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
```

The Microsoft documentation for StorageD3Enable mentioned that Windows has
a hardcoded allowlist for D3 support, which was used for these platforms.
Introduce quirks to hardcode them for Linux as well.

As this property is now "standardized", OEM systems using AMD Cezanne and
newer APU's have adopted this property, and quirks like this should not be
necessary.

CC: Julian Sikorski <belegdol at gmail.com>
CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
CC: Alexander Deucher <Alexander.Deucher at amd.com>
CC: Rafael J. Wysocki <rjw at rjwysocki.net>
CC: Prike Liang <prike.liang at amd.com>
Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
---
 drivers/acpi/device_pm.c |  3 +++
 drivers/acpi/x86/utils.c | 27 +++++++++++++++++++++++++++
 include/acpi/acpi_bus.h  |  5 +++++
 3 files changed, 35 insertions(+)

Changes from v4->v5:
 * Add this patch back in as it's been made apparent that the
   system needs to be hardcoded for these.
   Changes:
   - Drop Cezanne - it's now covered by StorageD3Enable
   - Rebase ontop of acpi_storage_d3 outside of NVME
Changes from v5->v6:
 * Move the quirk check into drivers/acpi/x86/ as suggested by
   Rafael.

diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index 1edb68d00b8e..985c17384192 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -1356,6 +1356,9 @@ bool acpi_storage_d3(struct device *dev)
 	struct acpi_device *adev = ACPI_COMPANION(dev);
 	u8 val;
 
+	if (force_storage_d3())
+		return true;
+
 	if (!adev)
 		return false;
 	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
diff --git a/drivers/acpi/x86/utils.c b/drivers/acpi/x86/utils.c
index bdc1ba00aee9..2b8d5b3c876f 100644
--- a/drivers/acpi/x86/utils.c
+++ b/drivers/acpi/x86/utils.c
@@ -135,3 +135,30 @@ bool acpi_device_always_present(struct acpi_device *adev)
 
 	return ret;
 }
+
+/*
+ * AMD systems from Renoir and Lucienne *require* that the NVME controller
+ * is put into D3 over a Modern Standby / suspend-to-idle cycle.
+ *
+ * This is "typically" accomplished using the `StorageD3Enable`
+ * property in the _DSD that is checked via the `acpi_storage_d3` function
+ * but this property was introduced after many of these systems launched
+ * and most OEM systems don't have it in their BIOS.
+ *
+ * The Microsoft documentation for StorageD3Enable mentioned that Windows has
+ * a hardcoded allowlist for D3 support, which was used for these platforms.
+ *
+ * This allows quirking on Linux in a similar fashion.
+ */
+const struct x86_cpu_id storage_d3_cpu_ids[] = {
+	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),	/* Renoir */
+	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL),	/* Lucienne */
+	{}
+};
+
+bool force_storage_d3(void)
+{
+	if (x86_match_cpu(storage_d3_cpu_ids))
+		return true;
+	return false;
+}
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index 3a82faac5767..9b0ddbae5617 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -607,11 +607,16 @@ int acpi_disable_wakeup_device_power(struct acpi_device *dev);
 
 #ifdef CONFIG_X86
 bool acpi_device_always_present(struct acpi_device *adev);
+bool force_storage_d3(void);
 #else
 static inline bool acpi_device_always_present(struct acpi_device *adev)
 {
 	return false;
 }
+static inline bool force_storage_d3(void)
+{
+	return false;
+}
 #endif
 
 #ifdef CONFIG_PM
-- 
2.25.1



From rrangel at chromium.org  Mon Jun  7 10:38:43 2021
From: rrangel at chromium.org (Raul Rangel)
Date: Mon, 7 Jun 2021 11:38:43 -0600
Subject: [PATCH v6 2/2] ACPI: Add quirks for AMD Renoir/Lucienne CPUs to
 force the D3 hint
In-Reply-To: <20210607173156.5548-2-mario.limonciello@amd.com>
References: <20210607173156.5548-1-mario.limonciello@amd.com>
 <20210607173156.5548-2-mario.limonciello@amd.com>
Message-ID: <CAHQZ30DXX+H=jPO74wKwsfN2nOgEjdyiUpkR8HWYN1L_EO-ucA@mail.gmail.com>

On Mon, Jun 7, 2021 at 11:32 AM Mario Limonciello
<mario.limonciello at amd.com> wrote:
>
> AMD systems from Renoir and Lucienne require that the NVME controller
> is put into D3 over a Modern Standby / suspend-to-idle
> cycle.  This is "typically" accomplished using the `StorageD3Enable`
> property in the _DSD, but this property was introduced after many
> of these systems launched and most OEM systems don't have it in
> their BIOS.
>
> On AMD Renoir without these drives going into D3 over suspend-to-idle
> the resume will fail with the NVME controller being reset and a trace
> like this in the kernel logs:
> ```
> [   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
> [   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
> [   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
> [   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
> [   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
> [   95.332843] nvme nvme0: Abort status: 0x371
> [   95.332852] nvme nvme0: Abort status: 0x371
> [   95.332856] nvme nvme0: Abort status: 0x371
> [   95.332859] nvme nvme0: Abort status: 0x371
> [   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
> [   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
> ```
>
> The Microsoft documentation for StorageD3Enable mentioned that Windows has
> a hardcoded allowlist for D3 support, which was used for these platforms.
> Introduce quirks to hardcode them for Linux as well.
>
> As this property is now "standardized", OEM systems using AMD Cezanne and
> newer APU's have adopted this property, and quirks like this should not be
> necessary.
>
> CC: Julian Sikorski <belegdol at gmail.com>
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> ---

It's unfortunate these systems have already shipped. I'm guessing they
can't have their firmware updated either. So I guess it's the best
option for now.

Acked-by: Raul E Rangel <rrangel at chromium.org>


From chaitanya.kulkarni at wdc.com  Mon Jun  7 12:54:50 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 12:54:50 -0700
Subject: [PATCH V2 1/5] nvme: use helper to remove duplicate code
In-Reply-To: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
References: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210607195454.16829-2-chaitanya.kulkarni@wdc.com>

Add a helper nvme_validate_passthru_nsid() to validate the nsid that
removes the nsid validation and error message print code from
nvme_user_cmd() and nvme_user_cmd64().

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/ioctl.c | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)

diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c
index 2e7780ea0354..d93928d1e5bd 100644
--- a/drivers/nvme/host/ioctl.c
+++ b/drivers/nvme/host/ioctl.c
@@ -177,6 +177,20 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
 			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
 }
 
+static bool nvme_validate_passthru_nsid(struct nvme_ctrl *ctrl,
+					struct nvme_ns *ns, __u32 nsid)
+{
+	if (ns && nsid != ns->head->ns_id) {
+		dev_err(ctrl->device,
+			"%s: nsid (%u) in cmd does not match nsid (%u)"
+			"of namespace\n",
+			current->comm, nsid, ns->head->ns_id);
+		return false;
+	}
+
+	return true;
+}
+
 static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			struct nvme_passthru_cmd __user *ucmd)
 {
@@ -192,12 +206,8 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 		return -EFAULT;
 	if (cmd.flags)
 		return -EINVAL;
-	if (ns && cmd.nsid != ns->head->ns_id) {
-		dev_err(ctrl->device,
-			"%s: nsid (%u) in cmd does not match nsid (%u) of namespace\n",
-			current->comm, cmd.nsid, ns->head->ns_id);
+	if (!nvme_validate_passthru_nsid(ctrl, ns, cmd.nsid))
 		return -EINVAL;
-	}
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = cmd.opcode;
@@ -242,12 +252,8 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 		return -EFAULT;
 	if (cmd.flags)
 		return -EINVAL;
-	if (ns && cmd.nsid != ns->head->ns_id) {
-		dev_err(ctrl->device,
-			"%s: nsid (%u) in cmd does not match nsid (%u) of namespace\n",
-			current->comm, cmd.nsid, ns->head->ns_id);
+	if (!nvme_validate_passthru_nsid(ctrl, ns, cmd.nsid))
 		return -EINVAL;
-	}
 
 	memset(&c, 0, sizeof(c));
 	c.common.opcode = cmd.opcode;
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun  7 12:54:51 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 12:54:51 -0700
Subject: [PATCH V2 2/5] nvme-fc: add a helper to check ctrl sgl support
In-Reply-To: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
References: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210607195454.16829-3-chaitanya.kulkarni@wdc.com>

For transports it is common to check if NVMe SGLs are supported or not
by the controller.

Add a helper instead of open coding controller SGL support and use it
in fc transport.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/fc.c   | 2 +-
 drivers/nvme/host/nvme.h | 8 ++++++++
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index f183f9fa03d0..7600863f7752 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -3112,7 +3112,7 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	}
 
 	/* FC-NVME supports normal SGL Data Block Descriptors */
-	if (!(ctrl->ctrl.sgls & ((1 << 0) | (1 << 1)))) {
+	if (!nvme_ctrl_sgl_supported(&ctrl->ctrl)) {
 		dev_err(ctrl->ctrl.device,
 			"Mandatory sgls are not supported!\n");
 		ret = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1f397ecba16c..1aab74128d40 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -869,6 +869,14 @@ static inline void nvme_hwmon_exit(struct nvme_ctrl *ctrl)
 }
 #endif
 
+static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
+{
+	if (!(ctrl->sgls & ((1 << 0) | (1 << 1))))
+		return true;
+
+	return false;
+}
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 void nvme_execute_passthru_rq(struct request *rq);
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun  7 12:54:49 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 12:54:49 -0700
Subject: [PATCH V2 0/5] nvme: small cleanups
Message-ID: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>

Hi,

This adds a couple of helpers to remove the duplicate code. The last
patch removes the exta white lines from nvme-pci.

If this makes sense, please consider this series for 5.14.

-ck

v1->v2

1. use nvme_ctrl_sgl_supported() helper in nvme_map_data().

 drivers/nvme/host/fc.c    |  2 +-
 drivers/nvme/host/ioctl.c | 26 ++++++++++++++++----------
 drivers/nvme/host/nvme.h  |  8 ++++++++
 drivers/nvme/host/pci.c   |  6 ++----
 drivers/nvme/host/tcp.c   |  2 +-
 5 files changed, 28 insertions(+), 16 deletions(-)

-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun  7 12:54:52 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 12:54:52 -0700
Subject: [PATCH V2 3/5] nvme-tcp: use helper for ctrl sgl check
In-Reply-To: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
References: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210607195454.16829-4-chaitanya.kulkarni@wdc.com>

Use helper to check the controller's SGL support instead of open coding.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/tcp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 5fc6c568c626..8d279aacb027 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1992,7 +1992,7 @@ static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new)
 		goto destroy_admin;
 	}
 
-	if (!(ctrl->sgls & ((1 << 0) | (1 << 1)))) {
+	if (!nvme_ctrl_sgl_supported(ctrl)) {
 		dev_err(ctrl->device, "Mandatory sgls are not supported!\n");
 		goto destroy_admin;
 	}
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun  7 12:54:53 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 12:54:53 -0700
Subject: [PATCH V2 4/5] nvme-pci: use helper for ctrl sgl check
In-Reply-To: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
References: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210607195454.16829-5-chaitanya.kulkarni@wdc.com>

Use helper to check the controller's SGL support instead of open coding.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/pci.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..f86ad4b7a9b6 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -536,7 +536,7 @@ static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
 
 	avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req), nseg);
 
-	if (!(dev->ctrl.sgls & ((1 << 0) | (1 << 1))))
+	if (!nvme_ctrl_sgl_supported(&dev->ctrl))
 		return false;
 	if (!iod->nvmeq->qid)
 		return false;
@@ -855,7 +855,7 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 							     &cmnd->rw, &bv);
 
 			if (iod->nvmeq->qid && sgl_threshold &&
-			    dev->ctrl.sgls & ((1 << 0) | (1 << 1)))
+			    nvme_ctrl_sgl_supported(&dev->ctrl))
 				return nvme_setup_sgl_simple(dev, req,
 							     &cmnd->rw, &bv);
 		}
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun  7 12:54:54 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 12:54:54 -0700
Subject: [PATCH V2 5/5] nvme-pci: remove trailing lines for helpers
In-Reply-To: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
References: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210607195454.16829-6-chaitanya.kulkarni@wdc.com>

Remove the extra white line at the end of the functions.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/pci.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index f86ad4b7a9b6..6933f843f3f2 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -559,7 +559,6 @@ static void nvme_free_prps(struct nvme_dev *dev, struct request *req)
 		dma_pool_free(dev->prp_page_pool, prp_list, dma_addr);
 		dma_addr = next_dma_addr;
 	}
-
 }
 
 static void nvme_free_sgls(struct nvme_dev *dev, struct request *req)
@@ -576,7 +575,6 @@ static void nvme_free_sgls(struct nvme_dev *dev, struct request *req)
 		dma_pool_free(dev->prp_page_pool, sg_list, dma_addr);
 		dma_addr = next_dma_addr;
 	}
-
 }
 
 static void nvme_unmap_sg(struct nvme_dev *dev, struct request *req)
-- 
2.22.1



From Chaitanya.Kulkarni at wdc.com  Mon Jun  7 12:59:24 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 7 Jun 2021 19:59:24 +0000
Subject: [PATCH v2 1/4] nvmet: change sn size and check validity
References: <20210607092324.1031046-1-mgurtovoy@nvidia.com>
Message-ID: <BYAPR04MB4965541CEAEB7DDD129D497386389@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/7/21 02:23, Max Gurtovoy wrote:
> From: Noam Gottlieb <ngottlieb at nvidia.com>
>
> According to the NVM specification, the serial_number should be 20 bytes
> (bytes 23:04 of the Identify Controller data structure), and should
> contain only ASCII characters.
>
> In accordance, the serial_number size is changed to 20 bytes and before
> any attempt to store a new value in serial_number we check that the
> input is valid - i.e. contains only ASCII characters, is not empty and
> does not exceed 20 bytes.
>
> Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
> Signed-off-by: Noam Gottlieb <ngottlieb at nvidia.com>
> ---

Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>




From Chaitanya.Kulkarni at wdc.com  Mon Jun  7 13:37:38 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 7 Jun 2021 20:37:38 +0000
Subject: [PATCH v2 2/4] nvmet: make sn stable once connection was
 established
References: <20210607092324.1031046-1-mgurtovoy@nvidia.com>
 <20210607092324.1031046-2-mgurtovoy@nvidia.com>
Message-ID: <BYAPR04MB4965C4BD68DD41755F8CA15386389@BYAPR04MB4965.namprd04.prod.outlook.com>

Max,

On 6/7/21 02:23, Max Gurtovoy wrote:
> diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
> index 0ae809ca428c..bd0a0b91d843 100644
> --- a/drivers/nvme/target/nvmet.h
> +++ b/drivers/nvme/target/nvmet.h
> @@ -231,6 +231,7 @@ struct nvmet_subsys {
>  
>  	u64			ver;
>  	char			serial[NVMET_SN_MAX_SIZE];
> +	bool			subsys_discovered;

When subsystem is discovered (connected to the host) we will have non empty
controller list, instead of adding a new bool variable why not use
something like following tested patch on the top of this one ?
(that also fixes the indentation of
nvmet_subsys_attr_serial_store_locked()).

diff --git a/drivers/nvme/target/admin-cmd.c
b/drivers/nvme/target/admin-cmd.c
index bec84c231d37..897f419be618 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -366,12 +366,6 @@ static void nvmet_execute_identify_ctrl(struct
nvmet_req *req)
        u32 cmd_capsule_size;
        u16 status = 0;
 
-       if (!subsys->subsys_discovered) {
-               mutex_lock(&subsys->lock);
-               subsys->subsys_discovered = true;
-               mutex_unlock(&subsys->lock);
-       }
-
        /*
         * If there is no model number yet, set it now.  It will then remain
         * stable for the life time of the subsystem.
diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index a13da86fb374..94e23dca763f 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -1046,11 +1046,11 @@ static ssize_t
nvmet_subsys_attr_serial_show(struct config_item *item,
 
 static ssize_t
 nvmet_subsys_attr_serial_store_locked(struct nvmet_subsys *subsys,
-               const char *page, size_t count)
+                                     const char *page, size_t count)
 {
        int pos, len = strcspn(page, "\n");
 
-       if (subsys->subsys_discovered) {
+       if (!list_empty(&subsys->ctrls)) {
                pr_err("Can't set serial number. %s is already assigned\n",
                       subsys->serial);
                return -EINVAL;
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index f7fbf9690ce8..73c6c4dc8997 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -232,7 +232,6 @@ struct nvmet_subsys {
 
        u64                     ver;
        char                    serial[NVMET_SN_MAX_SIZE];
-       bool                    subsys_discovered;
        char                    *subsysnqn;
        bool                    pi_support;
 

# nvme list
Node                  SN                  
Model                                    Namespace
Usage                      Format           FW Rev 
--------------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------
/dev/nvme0n1          foo                  QEMU NVMe
Ctrl                           1           1.07  GB /   1.07  GB   
512   B +  0 B   1.0    
/dev/nvme1n1          bf854da611a264d448c2
Linux                                    1           2.15  GB /   2.15 
GB      4 KiB +  0 B   5.13.0-r
# echo 1 > /sys/kernel/config/nvmet/subsystems/nqn/
allowed_hosts/       attr_cntlid_min      attr_serial          passthru/
attr_allow_any_host  attr_model           attr_version        
attr_cntlid_max      attr_pi_enable       namespaces/         
 # echo "1234567890" > /sys/kernel/config/nvmet/subsystems/nqn/attr_serial
bash: echo: write error: Invalid argument
# dmesg  -c
[311081.155148] nvmet: Can't set serial number. bf854da611a264d448c2 is
already assigned




From wubo40 at huawei.com  Mon Jun  7 21:13:36 2021
From: wubo40 at huawei.com (Wu Bo)
Date: Tue, 8 Jun 2021 12:13:36 +0800
Subject: [PATCH] nvme-multipath: combine grpid and ANA state checks in
 nvme_parse_ana_log()
Message-ID: <1623125616-629270-1-git-send-email-wubo40@huawei.com>

Hi,

Combine grpid and ANA state checks in nvme_parse_ana_log() to
reduce repetitive code and more intuitive meaning.

Signed-off-by: Wu Bo <wubo40 at huawei.com>
---
 drivers/nvme/host/multipath.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index f81871c..19fe16d 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -522,13 +522,11 @@ static int nvme_parse_ana_log(struct nvme_ctrl *ctrl, void *data,
 		nr_nsids = le32_to_cpu(desc->nnsids);
 		nsid_buf_size = nr_nsids * sizeof(__le32);
 
-		if (WARN_ON_ONCE(desc->grpid == 0))
+		if (WARN_ON_ONCE(desc->grpid == 0 ||
+			le32_to_cpu(desc->grpid) > ctrl->anagrpmax))
 			return -EINVAL;
-		if (WARN_ON_ONCE(le32_to_cpu(desc->grpid) > ctrl->anagrpmax))
-			return -EINVAL;
-		if (WARN_ON_ONCE(desc->state == 0))
-			return -EINVAL;
-		if (WARN_ON_ONCE(desc->state > NVME_ANA_CHANGE))
+		if (WARN_ON_ONCE(desc->state == 0 ||
+			desc->state > NVME_ANA_CHANGE))
 			return -EINVAL;
 
 		offset += sizeof(*desc);
-- 
2.30.2



From Chaitanya.Kulkarni at wdc.com  Mon Jun  7 21:55:52 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue, 8 Jun 2021 04:55:52 +0000
Subject: [PATCH] nvme-multipath: combine grpid and ANA state checks in
 nvme_parse_ana_log()
References: <1623125616-629270-1-git-send-email-wubo40@huawei.com>
Message-ID: <BYAPR04MB4965B1255EFAA2987414EEA386379@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/7/21 20:59, Wu Bo wrote:
> Hi,
>
> Combine grpid and ANA state checks in nvme_parse_ana_log() to
> reduce repetitive code and more intuitive meaning.
>
> Signed-off-by: Wu Bo <wubo40 at huawei.com>

Existing code just looks fine...




From hch at lst.de  Mon Jun  7 22:23:20 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 07:23:20 +0200
Subject: [PATCH] nvme-multipath: combine grpid and ANA state checks in
 nvme_parse_ana_log()
In-Reply-To: <1623125616-629270-1-git-send-email-wubo40@huawei.com>
References: <1623125616-629270-1-git-send-email-wubo40@huawei.com>
Message-ID: <20210608052320.GA13828@lst.de>

On Tue, Jun 08, 2021 at 12:13:36PM +0800, Wu Bo wrote:
> -		if (WARN_ON_ONCE(desc->grpid == 0))
> +		if (WARN_ON_ONCE(desc->grpid == 0 ||
> +			le32_to_cpu(desc->grpid) > ctrl->anagrpmax))
>  			return -EINVAL;
> -		if (WARN_ON_ONCE(le32_to_cpu(desc->grpid) > ctrl->anagrpmax))
> -			return -EINVAL;
> -		if (WARN_ON_ONCE(desc->state == 0))
> -			return -EINVAL;
> -		if (WARN_ON_ONCE(desc->state > NVME_ANA_CHANGE))
> +		if (WARN_ON_ONCE(desc->state == 0 ||
> +			desc->state > NVME_ANA_CHANGE))

So besides making the code impossibl to read due to the incorrect
indentation this also makes each WARN_ON_ONCE cover multiple conditions.
Not very useful for debugging.


From hch at lst.de  Mon Jun  7 22:26:07 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 07:26:07 +0200
Subject: [PATCHv3 4/4] nvme: use return value from blk_execute_rq()
In-Reply-To: <20210607165827.GC21631@redsun51.ssa.fujisawa.hgst.com>
References: <20210521202145.3674904-1-kbusch@kernel.org>
 <20210521202145.3674904-5-kbusch@kernel.org> <20210524080428.GA24488@lst.de>
 <20210607165827.GC21631@redsun51.ssa.fujisawa.hgst.com>
Message-ID: <20210608052607.GB13828@lst.de>

On Tue, Jun 08, 2021 at 01:58:27AM +0900, Keith Busch wrote:
> On Mon, May 24, 2021 at 10:04:28AM +0200, Christoph Hellwig wrote:
> > > @@ -168,7 +167,8 @@ static void nvmet_passthru_execute_cmd_work(struct work_struct *w)
> > >  			nvmet_passthru_override_id_ns(req);
> > >  			break;
> > >  		}
> > > -	}
> > > +	} else if (status < 0)
> > > +		status = NVME_SC_INTERNAL;
> > 
> > Don't we need a better translation here?
> 
> Did you have something in mind? I couldn't think of anything more
> appropriate than the generic internal error. The errno's we get here are
> -EINTR or -EIO. Both indicate we can't communicate with the back-end
> device, but these problems are internal to the target from the host's
> perspective.

Ok.


From hch at lst.de  Mon Jun  7 22:26:39 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 07:26:39 +0200
Subject: [PATCHv3 2/4] nvme: use blk_execute_rq() for passthrough commands
In-Reply-To: <20210607161512.GB21631@redsun51.ssa.fujisawa.hgst.com>
References: <20210521202145.3674904-1-kbusch@kernel.org>
 <20210521202145.3674904-3-kbusch@kernel.org>
 <CA+1E3rJM3sGLsOuPbYVH6kaTR8S9ogfe+wryuyWpqnA-pgDsEw@mail.gmail.com>
 <20210607161512.GB21631@redsun51.ssa.fujisawa.hgst.com>
Message-ID: <20210608052639.GC13828@lst.de>

On Tue, Jun 08, 2021 at 01:15:12AM +0900, Keith Busch wrote:
> > The new code doesn't retain this flag.
> > Looks good otherwise.
> 
> The flag is only used to select an apporpriate hctx. We've explicitly
> selected a polling hctx in this path already, so the flag is
> unnecessary.

I think setting the flag would still be useful, if only to prevent issues
with new callers and/or better document what we are doing.


From hch at lst.de  Mon Jun  7 22:35:46 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 07:35:46 +0200
Subject: [PATCH v6 1/2] ACPI: Move check for _DSD StorageD3Enable
 property to acpi
In-Reply-To: <20210607173156.5548-1-mario.limonciello@amd.com>
References: <20210607173156.5548-1-mario.limonciello@amd.com>
Message-ID: <20210608053546.GA14116@lst.de>

On Mon, Jun 07, 2021 at 12:31:55PM -0500, Mario Limonciello wrote:
> +/**
> + * acpi_storage_d3 - Check if a storage device should use D3.
> + * @dev: Device to check
> + *
> + * Returns %true if @dev should be put into D3 when the ->suspend method is
> + * called, else %false.  The name of this function is somewhat misleading
> + * as it has nothing to do with storage except for the name of the ACPI
> + * property.  On some platforms resume will not work if this hint is ignored.
> + *
> + */

I still do not like this comment.  There is nothing about only using this
for storage devices.  It is specific to a PCIe slot, and if I plug
something that is not a storage device into it the same restrictions
still apply.

Also no need for the empty-ish line at the end.


From chaitanya.kulkarni at wdc.com  Mon Jun  7 22:46:30 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 22:46:30 -0700
Subject: [PATCH 0/5] nvmet: add ZBD backened support
Message-ID: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>

Hi,

NVMeOF Host is capable of handling the NVMe Protocol based Zoned Block
Devices (ZBD) in the Zoned Namespaces (ZNS) mode with the passthru
backend. There is no support for a generic block device backend to
handle the ZBDs which are not NVMe protocol compliant.

This adds support to export the ZBDs (which are not NVMe drives) to host
from the target via NVMeOF using the host side ZNS interface.

Note: This patch-series is based on nvme-5.14.

-ck

Chaitanya Kulkarni (5):
  block: export blk_next_bio()
  nvmet: add req cns error complete helper
  nvmet: add nvmet_req_bio put helper for backends
  nvmet: add Command Set Identifier support
  nvmet: add ZBD over ZNS backend support

 block/blk-lib.c                   |   1 +
 drivers/nvme/target/Makefile      |   1 +
 drivers/nvme/target/admin-cmd.c   | 121 +++++-
 drivers/nvme/target/core.c        |  43 ++-
 drivers/nvme/target/io-cmd-bdev.c |  29 +-
 drivers/nvme/target/nvmet.h       |  35 ++
 drivers/nvme/target/passthru.c    |   3 +-
 drivers/nvme/target/zns.c         | 622 ++++++++++++++++++++++++++++++
 include/linux/bio.h               |   2 +
 include/linux/nvme.h              |   8 +
 10 files changed, 828 insertions(+), 37 deletions(-)
 create mode 100644 drivers/nvme/target/zns.c

Changes since V13:-

1. Move the code cleanup patches upfront and export blk_next_bio().
2. Remove NVM in the subject line for the CSS patch.
3. Remove the switch from nvmet_execute_identify_desclist_csi()
   and use ns->csi directly.
4. Add csi switch in the nvmet_parse_io_cmd().
5. Rename nvmet_cc_css_check() -> nvmet_css_supported().
6. Remove switch from nvmet_set_csi_zns_effects().
7. Rename nvmet_set_csi_nvm_effects() ->
   nvmet_get_cmd_effects_nvm().
7. Add separate parse case for the different csi.
8. Remove the bdev_is_zoned().
9. Remove the csi check from the admin cmd handlers and move
   to the caller.
10. Add Zone State change emulation and add state machine check
    to handle zone mgmt send command.
12. Abort the zone append command with error when transfer len = 0.
13. Don't allocate the buffer for zone mgmt recv report zones instead
    use the command data buffer.
14. Overall small cleanups to reflect the new code changes.
15. Offload zone-mgmt commands to workqueue.

Changes since V12:-

1. Remove the comment :-
  /* bdev_is_zoned() is stubbed out of CONFIG_BLK_DEV_ZONED */
2. Update copyright header.
3. Use folliwng expression in nvmet_zasl(). 
   return ilog2(zone_append_sects >> (NVMET_MPSMIN_SHIFT - 9));
4. Add comment in nvmet_bdev_zns_enable().
5. Add zone filtering and partial bit handling.
6. Use if else pattern in nvmet_bdev_execute_zone_mgmt_send().
7. Make zone append async. 
8. Add a comment for nvmet_check_transfer_len().
9. Add a helper nvmet_req_cns_error_compplete() to remove the duplicate
   code.

Changes from V11:-

1. Use bdev_logical_block_size() in nvmet_bdev_zns_enable().
2. Return error if nr_zones calculation ends up having value 0.
4. Drop the comment patch from this series :-
   nvme: add comments to nvme_zns_alloc_report_buffer

Changes from V10:-

1.  Move CONFIG_BLK_DEV_ZONED check into the caller of
    nvmet_set_csi_zns_effects().
2.  Move ZNS related csi code from csi patch to its own ZNS backend
    patch.
3.  For ZNS command effects logs set default command effects with 
    nvmet_set_csi_nvm_effects() along with nvmet_set_csi_zns_effects().
4.  Use goto for failure case in the nvmet_set_csi_zns_effects().
5.  Return directly from swicth in nvmet_execute_identify_desclist_csi().
6.  Merge Patch 2nd/3rd into one patch and move ZNS related code
    into its own :-
     [PATCH V10 2/8] nvmet: add NVM Command Set Identifier support
     [PATCH V10 3/8] nvmet: add command set supported ctrl cap
    Merged into new patch minux ZNS calls :-
     [PATCH V11 1/4] nvmet: add NVM Command Set Identifier support
7.  Move req->cmd->identify.csi == NVME_CSI_ZNS checks in to respective
    caller in nvmet_execute_identify().
8.  Update error log page in nvmet_bdev_zns_checks().
9.  Remove the terney expression nvmet_zns_update_zasl().
10. Drop following patches:-
     [PATCH V10 1/8] nvmet: trim args for nvmet_copy_ns_identifier()
     [PATCH V10 6/8] nvme-core: check ctrl css before setting up zns
     [PATCH V10 7/8] nvme-core: add a helper to print css related error

Changes from V9:-

1.  Use bio_add_zone_append_page() for REQ_OP_ZONE_APPEND.
2.  Add a separate prep patch to reduce the arguments for
    nvmet_copy_ns_identifier().
3.  Add a separate patch for nvmet CSI support.
4.  Add a separate patch for nvmet CSS support.
5.  Move nvmet_cc_css_check() to multi-css supported patch.
6.  Return error in csi cmd effects helper when !CONFIG_BLK_DEV_ZONED.
7.  Return error in id desc list helper when !CONFIG_BLK_DEV_ZONED.
8.  Remove goto and return from nvmet_bdev_zns_checks().
9.  Move nr_zones calculations near call to blkdev_report_zones() in
    nvmet_bdev_execute_zone_mgmt_recv().
10. Split command effects logs into respective CSI helpers.
11. Don't use the local variables to pass NVME_CSI_XXX values, instead
    use req->ns->csi, also move this from ZBD support patch to nvmet
    CSI support.
12. Fix the bug that is chekcing cns value instead of csi in identify.
13. bdev_is_zoned() is stubbed out the CONFIG_BLK_DEV_ZONED, so remove
    the check for CONFIG_BLK_DEV_ZONED before calling bdev_is_zoned().
14  Drop following patches :-
    [PATCH V9 1/9] block: export bio_add_hw_pages().
    [PATCH V9 5/9] nvmet: add bio get helper.
    [PATCH V9 6/9] nvmet: add bio init helper.
    [PATCH V9 8/9] nvmet: add common I/O length check helper.
    [PATCH V9 9/9] nvmet: call nvmet_bio_done() for zone append.
15. Add a patch to check for ctrl bits to make sure ctrl supports the
    multi css on the host side when setting up Zoned Namespace.
17. Add a documentation patch for the host side when calculating the
    buffer size allocation size for the report zones.
16. Rebase and retest 5.12-rc1.

Changes from V8:-

1. Rebase and retest on latest nvme-5.11.
2. Export ctrl->cap csi support only if CONFIG_BLK_DEV_ZONE is set.
3. Add a fix to admin ns-desc list handler for handling default csi.

Changes from V7:-

1. Just like what block layer provides an API for bio_init(), provide
   nvmet_bio_init() such that we move bio initialization code for
   nvme-read-write commands from bdev and zns backend into the centralize
   helper. 
2. With bdev/zns/file now we have three backends that are checking for
   req->sg_cnt and calling nvmet_check_transfer_len() before we process
   nvme-read-write commands. Move this duplicate code from three
   backeneds into the helper.
3. Export and use nvmet_bio_done() callback in
   nvmet_execute_zone_append() instead of the open coding the function.
   This also avoids code duplication for bio & request completion with
   error log page update.
4. Add zonefs tests log for dm linear device created on the top of SMR HDD
   exported with NVMeOF ZNS backend with the help of nvme-loop.

Changes from V6:-

1. Instead of calling report zones to find conventional zones in the 
   loop use the loop inside LLD blkdev_report_zones()->LLD_report_zones,
   that also simplifies the report zone callback.
2. Fix the bug in nvmet_bdev_has_conv_zones().
3. Remove conditional operators in the nvmet_bdev_execute_zone_append().

Changes from V5:-

1.  Use bio->bi_iter.bi_sector for result of the REQ_OP_ZONE_APPEND
    command.
2.  Add endianness to the helper nvmet_sect_to_lba().
3.  Make bufsize u32 in zone mgmt recv command handler.
4.  Add __GFP_ZERO for report zone data buffer to return clean buffer.

Changes from V4:-

1.  Don't use bio_iov_iter_get_pages() instead add a patch to export
    bio_add_hw_page() and call it directly for zone append.
2.  Add inline vector optimization for append bio.
3.  Update the commit logs for the patches.
4.  Remove ZNS related identify data structures, use individual members.
5.  Add a comment for macro NVMET_MPSMIN_SHIFT.
6.  Remove nvmet_bdev() helper.
7.  Move the command set identifier code into common code.
8.  Use IS_ENABLED() and move helpers fomr zns.c into common code.
9.  Add a patch to support Command Set identifiers.
10. Open code nvmet_bdev_validate_zns_zones().
11. Remove the per namespace min zasl calculation and don't allow
    namespaces with zasl value > the first ns zasl value.
12. Move the stubs into the header file.
13. Add lba to/from sector conversion helpers and update the
    io-cmd-bdev.c to avoid the code duplication.
14. Add everything into one patch for zns command handlers and 
    respective calls from the target code.
15. Remove the trim ns-desclist admin callback patch from this series.
16. Add bio get and put helpers patches to reduce the duplicate code in
    generic bdev, passthru, and generic zns backend.

Changes from V3:-

1.  Get rid of the bio_max_zasl check.
2.  Remove extra lines.
3.  Remove the block layer api export patch.
4.  Remove the bvec check in the bio_iov_iter_get_pages() for
    REQ_OP_ZONE_APPEND so that we can reuse the code.

Changes from V2:-

1.  Move conventional zone bitmap check into 
    nvmet_bdev_validate_zns_zones(). 
2.  Don't use report zones call to check the runt zone.
3.  Trim nvmet_zasl() helper.
4.  Fix typo in the nvmet_zns_update_zasl().
5.  Remove the comment and fix the mdts calculation in
    nvmet_execute_identify_cns_cs_ctrl().
6.  Use u64 for bufsize in nvmet_bdev_execute_zone_mgmt_recv().
7.  Remove nvmet_zones_to_desc_size() and fix the nr_zones
    calculation.
8.  Remove the op variable in nvmet_bdev_execute_zone_append().
9.  Fix the nr_zones calculation nvmet_bdev_execute_zone_mgmt_recv().
10. Update cover letter subject.

Changes from V1:-

1.  Remove the nvmet-$(CONFIG_BLK_DEV_ZONED) += zns.o.
2.  Mark helpers inline.
3.  Fix typos in the comments and update the comments.
4.  Get rid of the curly brackets.
5.  Don't allow drives with last smaller zones.
6.  Calculate the zasl as a function of ax_zone_append_sectors,
    bio_max_pages so we don't have to split the bio.
7.  Add global subsys->zasl and update the zasl when new namespace
    is enabled.
8.  Rmove the loop in the nvmet_bdev_execute_zone_mgmt_recv() and
    move functionality in to the report zone callback.
9.  Add goto for default case in nvmet_bdev_execute_zone_mgmt_send().
10. Allocate the zones buffer with zones size instead of bdev nr_zones.

-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun  7 22:46:31 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 22:46:31 -0700
Subject: [PATCH 1/5] block: export blk_next_bio()
In-Reply-To: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
References: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210608054635.5661-2-chaitanya.kulkarni@wdc.com>

The block layer provides emulation of zone management operations
targeting all zones of a zoned block device only for the zone reset
operation (REQ_OP_ZONE_RESET). In order to correctly implement
exporting of zoned block devices with NVMeOF, emulating zone management
operations targeting all zones of a device is also necessary for the
open, close and finish zone operations (REQ_OP_ZONE_OPEN,
REQ_OP_ZONE_CLOSE and REQ_OP_ZONE_FINISH).

Instead of duplicating the code, export the existing helper from block
layer so we can use a bio chaining pattern that is present in the block
layer for REQ_OP_ZONE RESET all emulation in the NVMeOF zoned block
device backend.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 block/blk-lib.c     | 1 +
 include/linux/bio.h | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/block/blk-lib.c b/block/blk-lib.c
index 7b256131b20b..9f09beadcbe3 100644
--- a/block/blk-lib.c
+++ b/block/blk-lib.c
@@ -21,6 +21,7 @@ struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp)
 
 	return new;
 }
+EXPORT_SYMBOL_GPL(blk_next_bio);
 
 int __blkdev_issue_discard(struct block_device *bdev, sector_t sector,
 		sector_t nr_sects, gfp_t gfp_mask, int flags,
diff --git a/include/linux/bio.h b/include/linux/bio.h
index a0b4cfdf62a4..b2491ead22a0 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -822,4 +822,6 @@ static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb)
 		bio->bi_opf |= REQ_NOWAIT;
 }
 
+struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp);
+
 #endif /* __LINUX_BIO_H */
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun  7 22:46:32 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 22:46:32 -0700
Subject: [PATCH 2/5] nvmet: add req cns error complete helper
In-Reply-To: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
References: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210608054635.5661-3-chaitanya.kulkarni@wdc.com>

We report error and complete the request when identify cns value is not
handled in nvmet_execute_identify(). This error reporting is also needed
for Zone Block Device backend for NVMeOF target.

Add a helper nvmet_req_cns_error_compplete() to report an error and
complete the request when idenitfy command cns not handled value.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 5 +----
 drivers/nvme/target/nvmet.h     | 8 ++++++++
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index dcd49a72f2f3..94230803c75a 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -659,10 +659,7 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 		return nvmet_execute_identify_desclist(req);
 	}
 
-	pr_debug("unhandled identify cns %d on qid %d\n",
-	       req->cmd->identify.cns, req->sq->qid);
-	req->error_loc = offsetof(struct nvme_identify, cns);
-	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
+	nvmet_req_cns_error_complete(req);
 }
 
 /*
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index d69a409515d6..058b6d3d48fa 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -622,4 +622,12 @@ static inline bool nvmet_use_inline_bvec(struct nvmet_req *req)
 	       req->sg_cnt <= NVMET_MAX_INLINE_BIOVEC;
 }
 
+static inline void nvmet_req_cns_error_complete(struct nvmet_req *req)
+{
+	pr_debug("unhandled identify cns %d on qid %d\n",
+	       req->cmd->identify.cns, req->sq->qid);
+	req->error_loc = offsetof(struct nvme_identify, cns);
+	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
+}
+
 #endif /* _NVMET_H */
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun  7 22:46:33 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 22:46:33 -0700
Subject: [PATCH 3/5] nvmet: add nvmet_req_bio put helper for backends
In-Reply-To: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
References: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210608054635.5661-4-chaitanya.kulkarni@wdc.com>

In current code there exists two backends which are using inline bio
optimization, that adds a duplicate code for freeing the bio.

For Zoned Block Device backend we also use the same optimzation and it
will lead to having duplicate code in the three backends: generic
bdev, passsthru, and generic zns.

Add a helper function to avoid duplicate code and update the respective
backends.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/io-cmd-bdev.c | 3 +--
 drivers/nvme/target/nvmet.h       | 6 ++++++
 drivers/nvme/target/passthru.c    | 3 +--
 3 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index f673679d258a..40534d9a86d1 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -164,8 +164,7 @@ static void nvmet_bio_done(struct bio *bio)
 	struct nvmet_req *req = bio->bi_private;
 
 	nvmet_req_complete(req, blk_to_nvme_status(req, bio->bi_status));
-	if (bio != &req->b.inline_bio)
-		bio_put(bio);
+	nvmet_req_bio_put(req, bio);
 }
 
 #ifdef CONFIG_BLK_DEV_INTEGRITY
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 058b6d3d48fa..a7d035be9f13 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -630,4 +630,10 @@ static inline void nvmet_req_cns_error_complete(struct nvmet_req *req)
 	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
 }
 
+static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio)
+{
+	if (bio != &req->b.inline_bio)
+		bio_put(bio);
+}
+
 #endif /* _NVMET_H */
diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
index 39b1473f7204..fced52de33ce 100644
--- a/drivers/nvme/target/passthru.c
+++ b/drivers/nvme/target/passthru.c
@@ -206,8 +206,7 @@ static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq)
 	for_each_sg(req->sg, sg, req->sg_cnt, i) {
 		if (bio_add_pc_page(rq->q, bio, sg_page(sg), sg->length,
 				    sg->offset) < sg->length) {
-			if (bio != &req->p.inline_bio)
-				bio_put(bio);
+			nvmet_req_bio_put(req, bio);
 			return -EINVAL;
 		}
 	}
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun  7 22:46:34 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 22:46:34 -0700
Subject: [PATCH 4/5] nvmet: add Command Set Identifier support
In-Reply-To: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
References: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210608054635.5661-5-chaitanya.kulkarni@wdc.com>

NVMe TP 4056 allows controllers to support different command sets.
NVMeoF target currently only supports namespaces that contain
traditional logical blocks that may be randomly read and written. In
some applications there is a value in exposing namespaces that contain
logical blocks that have special access rules (e.g. sequentially write
required namespace such as Zoned Namespace (ZNS)).

In order to support the Zoned Block Devices (ZBD) backend, controllers
need to have support for ZNS Command Set Identifier (CSI).

In this preparation patch, we adjust the code such that it can now
support the default command set identifier. We update the namespace data
structure to store the CSI value which defaults to NVME_CSI_NVM
that represents traditional logical blocks namespace type.

The CSI support is required to implement the ZBD backend for NVMeOF
with host side NVMe ZNS interface, since ZNS commands belong to
the different command set than the default one.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 75 +++++++++++++++++++++++++++------
 drivers/nvme/target/core.c      | 28 +++++++++---
 drivers/nvme/target/nvmet.h     |  1 +
 include/linux/nvme.h            |  1 +
 4 files changed, 87 insertions(+), 18 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 94230803c75a..118b81a34bbb 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -162,15 +162,8 @@ static void nvmet_execute_get_log_page_smart(struct nvmet_req *req)
 	nvmet_req_complete(req, status);
 }
 
-static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
+static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log)
 {
-	u16 status = NVME_SC_INTERNAL;
-	struct nvme_effects_log *log;
-
-	log = kzalloc(sizeof(*log), GFP_KERNEL);
-	if (!log)
-		goto out;
-
 	log->acs[nvme_admin_get_log_page]	= cpu_to_le32(1 << 0);
 	log->acs[nvme_admin_identify]		= cpu_to_le32(1 << 0);
 	log->acs[nvme_admin_abort_cmd]		= cpu_to_le32(1 << 0);
@@ -184,9 +177,30 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
 	log->iocs[nvme_cmd_flush]		= cpu_to_le32(1 << 0);
 	log->iocs[nvme_cmd_dsm]			= cpu_to_le32(1 << 0);
 	log->iocs[nvme_cmd_write_zeroes]	= cpu_to_le32(1 << 0);
+}
 
-	status = nvmet_copy_to_sgl(req, 0, log, sizeof(*log));
+static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
+{
+	struct nvme_effects_log *log;
+	u16 status = NVME_SC_SUCCESS;
+
+	log = kzalloc(sizeof(*log), GFP_KERNEL);
+	if (!log) {
+		status = NVME_SC_INTERNAL;
+		goto out;
+	}
+
+	switch (req->cmd->get_log_page.csi) {
+	case NVME_CSI_NVM:
+		nvmet_get_cmd_effects_nvm(log);
+		break;
+	default:
+		status = NVME_SC_INVALID_LOG_PAGE;
+		goto free;
+	}
 
+	status = nvmet_copy_to_sgl(req, 0, log, sizeof(*log));
+free:
 	kfree(log);
 out:
 	nvmet_req_complete(req, status);
@@ -635,6 +649,12 @@ static void nvmet_execute_identify_desclist(struct nvmet_req *req)
 			goto out;
 	}
 
+	status = nvmet_copy_ns_identifier(req, NVME_NIDT_CSI,
+					  NVME_NIDT_CSI_LEN,
+					  &req->ns->csi, &off);
+	if (status)
+		goto out;
+
 	if (sg_zero_buffer(req->sg, req->sg_cnt, NVME_IDENTIFY_DATA_SIZE - off,
 			off) != NVME_IDENTIFY_DATA_SIZE - off)
 		status = NVME_SC_INTERNAL | NVME_SC_DNR;
@@ -643,6 +663,17 @@ static void nvmet_execute_identify_desclist(struct nvmet_req *req)
 	nvmet_req_complete(req, status);
 }
 
+static bool nvmet_handle_identify_desclist(struct nvmet_req *req)
+{
+	switch (req->cmd->identify.csi) {
+	case NVME_CSI_NVM:
+		nvmet_execute_identify_desclist(req);
+		return true;
+	default:
+		return false;
+	}
+}
+
 static void nvmet_execute_identify(struct nvmet_req *req)
 {
 	if (!nvmet_check_transfer_len(req, NVME_IDENTIFY_DATA_SIZE))
@@ -650,13 +681,31 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 
 	switch (req->cmd->identify.cns) {
 	case NVME_ID_CNS_NS:
-		return nvmet_execute_identify_ns(req);
+		switch (req->cmd->identify.csi) {
+		case NVME_CSI_NVM:
+			return nvmet_execute_identify_ns(req);
+		default:
+			break;
+		}
+		break;
 	case NVME_ID_CNS_CTRL:
-		return nvmet_execute_identify_ctrl(req);
+		switch (req->cmd->identify.csi) {
+		case NVME_CSI_NVM:
+			return nvmet_execute_identify_ctrl(req);
+		}
+		break;
 	case NVME_ID_CNS_NS_ACTIVE_LIST:
-		return nvmet_execute_identify_nslist(req);
+		switch (req->cmd->identify.csi) {
+		case NVME_CSI_NVM:
+			return nvmet_execute_identify_nslist(req);
+		default:
+			break;
+		}
+		break;
 	case NVME_ID_CNS_NS_DESC_LIST:
-		return nvmet_execute_identify_desclist(req);
+		if (nvmet_handle_identify_desclist(req) == true)
+			return;
+		break;
 	}
 
 	nvmet_req_cns_error_complete(req);
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 4ae4bea6625d..7ee9418cd9b6 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -692,6 +692,7 @@ struct nvmet_ns *nvmet_ns_alloc(struct nvmet_subsys *subsys, u32 nsid)
 
 	uuid_gen(&ns->uuid);
 	ns->buffered_io = false;
+	ns->csi = NVME_CSI_NVM;
 
 	return ns;
 }
@@ -887,10 +888,14 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 		return ret;
 	}
 
-	if (req->ns->file)
-		return nvmet_file_parse_io_cmd(req);
-
-	return nvmet_bdev_parse_io_cmd(req);
+	switch (req->ns->csi) {
+	case NVME_CSI_NVM:
+		if (req->ns->file)
+			return nvmet_file_parse_io_cmd(req);
+		return nvmet_bdev_parse_io_cmd(req);
+	default:
+		return NVME_SC_INVALID_IO_CMD_SET;
+	}
 }
 
 bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
@@ -1112,6 +1117,17 @@ static inline u8 nvmet_cc_iocqes(u32 cc)
 	return (cc >> NVME_CC_IOCQES_SHIFT) & 0xf;
 }
 
+static inline bool nvmet_css_supported(u8 cc_css)
+{
+	switch (cc_css <<= NVME_CC_CSS_SHIFT) {
+	case NVME_CC_CSS_NVM:
+	case NVME_CC_CSS_CSI:
+		return true;
+	default:
+		return false;
+	}
+}
+
 static void nvmet_start_ctrl(struct nvmet_ctrl *ctrl)
 {
 	lockdep_assert_held(&ctrl->lock);
@@ -1131,7 +1147,7 @@ static void nvmet_start_ctrl(struct nvmet_ctrl *ctrl)
 
 	if (nvmet_cc_mps(ctrl->cc) != 0 ||
 	    nvmet_cc_ams(ctrl->cc) != 0 ||
-	    nvmet_cc_css(ctrl->cc) != 0) {
+	    !nvmet_css_supported(nvmet_cc_css(ctrl->cc))) {
 		ctrl->csts = NVME_CSTS_CFS;
 		return;
 	}
@@ -1182,6 +1198,8 @@ static void nvmet_init_cap(struct nvmet_ctrl *ctrl)
 {
 	/* command sets supported: NVMe command set: */
 	ctrl->cap = (1ULL << 37);
+	/* Controller supports one or more I/O Command Sets */
+	ctrl->cap |= (1ULL << 43);
 	/* CC.EN timeout in 500msec units: */
 	ctrl->cap |= (15ULL << 24);
 	/* maximum queue entries supported: */
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index a7d035be9f13..f709238014f5 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -82,6 +82,7 @@ struct nvmet_ns {
 	struct pci_dev		*p2p_dev;
 	int			pi_type;
 	int			metadata_size;
+	u8			csi;
 };
 
 static inline struct nvmet_ns *to_nvmet_ns(struct config_item *item)
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index edcbd60b88b9..c7ba83144d52 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -1504,6 +1504,7 @@ enum {
 	NVME_SC_NS_WRITE_PROTECTED	= 0x20,
 	NVME_SC_CMD_INTERRUPTED		= 0x21,
 	NVME_SC_TRANSIENT_TR_ERR	= 0x22,
+	NVME_SC_INVALID_IO_CMD_SET	= 0x2C,
 
 	NVME_SC_LBA_RANGE		= 0x80,
 	NVME_SC_CAP_EXCEEDED		= 0x81,
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun  7 22:46:35 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon,  7 Jun 2021 22:46:35 -0700
Subject: [PATCH 5/5] nvmet: add ZBD over ZNS backend support
In-Reply-To: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
References: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210608054635.5661-6-chaitanya.kulkarni@wdc.com>

NVMe TP 4053 ? Zoned Namespaces (ZNS) allows host software to
communicate with a non-volatile memory subsystem using zones for NVMe
protocol-based controllers. NVMeOF already support the ZNS NVMe
Protocol compliant devices on the target in the passthru mode. There
are generic zoned block devices like  Shingled Magnetic Recording (SMR)
HDDs that are not based on the NVMe protocol.

This patch adds ZNS backend support for non-ZNS zoned block devices as
NVMeOF targets.

This support includes implementing the new command set NVME_CSI_ZNS,
adding different command handlers for ZNS command set such as NVMe
Identify Controller, NVMe Identify Namespace, NVMe Zone Append,
NVMe Zone Management Send and NVMe Zone Management Receive.

With the new command set identifier, we also update the target command
effects logs to reflect the ZNS compliant commands.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/Makefile      |   1 +
 drivers/nvme/target/admin-cmd.c   |  41 ++
 drivers/nvme/target/core.c        |  15 +-
 drivers/nvme/target/io-cmd-bdev.c |  26 +-
 drivers/nvme/target/nvmet.h       |  20 +
 drivers/nvme/target/zns.c         | 622 ++++++++++++++++++++++++++++++
 include/linux/nvme.h              |   7 +
 7 files changed, 721 insertions(+), 11 deletions(-)
 create mode 100644 drivers/nvme/target/zns.c

diff --git a/drivers/nvme/target/Makefile b/drivers/nvme/target/Makefile
index ebf91fc4c72e..9837e580fa7e 100644
--- a/drivers/nvme/target/Makefile
+++ b/drivers/nvme/target/Makefile
@@ -12,6 +12,7 @@ obj-$(CONFIG_NVME_TARGET_TCP)		+= nvmet-tcp.o
 nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
 			discovery.o io-cmd-file.o io-cmd-bdev.o
 nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
+nvmet-$(CONFIG_BLK_DEV_ZONED)		+= zns.o
 nvme-loop-y	+= loop.o
 nvmet-rdma-y	+= rdma.o
 nvmet-fc-y	+= fc.o
diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 118b81a34bbb..72470e815146 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -179,6 +179,13 @@ static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log)
 	log->iocs[nvme_cmd_write_zeroes]	= cpu_to_le32(1 << 0);
 }
 
+static void nvmet_get_cmd_effects_zns(struct nvme_effects_log *log)
+{
+	log->iocs[nvme_cmd_zone_append]		= cpu_to_le32(1 << 0);
+	log->iocs[nvme_cmd_zone_mgmt_send]	= cpu_to_le32(1 << 0);
+	log->iocs[nvme_cmd_zone_mgmt_recv]	= cpu_to_le32(1 << 0);
+}
+
 static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
 {
 	struct nvme_effects_log *log;
@@ -194,6 +201,14 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
 	case NVME_CSI_NVM:
 		nvmet_get_cmd_effects_nvm(log);
 		break;
+	case NVME_CSI_ZNS:
+		if (!IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
+			status = NVME_SC_INVALID_IO_CMD_SET;
+			goto free;
+		}
+		nvmet_get_cmd_effects_nvm(log);
+		nvmet_get_cmd_effects_zns(log);
+		break;
 	default:
 		status = NVME_SC_INVALID_LOG_PAGE;
 		goto free;
@@ -669,6 +684,12 @@ static bool nvmet_handle_identify_desclist(struct nvmet_req *req)
 	case NVME_CSI_NVM:
 		nvmet_execute_identify_desclist(req);
 		return true;
+	case NVME_CSI_ZNS:
+		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
+			nvmet_execute_identify_desclist(req);
+			return true;
+		}
+		return false;
 	default:
 		return false;
 	}
@@ -688,12 +709,32 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 			break;
 		}
 		break;
+	case NVME_ID_CNS_CS_NS:
+		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
+			switch (req->cmd->identify.csi) {
+			case NVME_CSI_ZNS:
+				return nvmet_execute_identify_cns_cs_ns(req);
+			default:
+				break;
+			}
+		}
+		break;
 	case NVME_ID_CNS_CTRL:
 		switch (req->cmd->identify.csi) {
 		case NVME_CSI_NVM:
 			return nvmet_execute_identify_ctrl(req);
 		}
 		break;
+	case NVME_ID_CNS_CS_CTRL:
+		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
+			switch (req->cmd->identify.csi) {
+			case NVME_CSI_ZNS:
+				return nvmet_execute_identify_cns_cs_ctrl(req);
+			default:
+				break;
+			}
+		}
+		break;
 	case NVME_ID_CNS_NS_ACTIVE_LIST:
 		switch (req->cmd->identify.csi) {
 		case NVME_CSI_NVM:
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 7ee9418cd9b6..3ca85e948fc4 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -16,6 +16,7 @@
 #include "nvmet.h"
 
 struct workqueue_struct *buffered_io_wq;
+struct workqueue_struct *zbd_wq;
 static const struct nvmet_fabrics_ops *nvmet_transports[NVMF_TRTYPE_MAX];
 static DEFINE_IDA(cntlid_ida);
 
@@ -893,6 +894,10 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 		if (req->ns->file)
 			return nvmet_file_parse_io_cmd(req);
 		return nvmet_bdev_parse_io_cmd(req);
+	case NVME_CSI_ZNS:
+		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED))
+			return nvmet_bdev_zns_parse_io_cmd(req);
+		return NVME_SC_INVALID_IO_CMD_SET;
 	default:
 		return NVME_SC_INVALID_IO_CMD_SET;
 	}
@@ -1587,11 +1592,15 @@ static int __init nvmet_init(void)
 
 	nvmet_ana_group_enabled[NVMET_DEFAULT_ANA_GRPID] = 1;
 
+	zbd_wq = alloc_workqueue("nvmet-zbd-wq", WQ_MEM_RECLAIM, 0);
+	if (!zbd_wq)
+		return -ENOMEM;
+
 	buffered_io_wq = alloc_workqueue("nvmet-buffered-io-wq",
 			WQ_MEM_RECLAIM, 0);
 	if (!buffered_io_wq) {
 		error = -ENOMEM;
-		goto out;
+		goto out_free_zbd_work_queue;
 	}
 
 	error = nvmet_init_discovery();
@@ -1607,7 +1616,8 @@ static int __init nvmet_init(void)
 	nvmet_exit_discovery();
 out_free_work_queue:
 	destroy_workqueue(buffered_io_wq);
-out:
+out_free_zbd_work_queue:
+	destroy_workqueue(zbd_wq);
 	return error;
 }
 
@@ -1617,6 +1627,7 @@ static void __exit nvmet_exit(void)
 	nvmet_exit_discovery();
 	ida_destroy(&cntlid_ida);
 	destroy_workqueue(buffered_io_wq);
+	destroy_workqueue(zbd_wq);
 
 	BUILD_BUG_ON(sizeof(struct nvmf_disc_rsp_page_entry) != 1024);
 	BUILD_BUG_ON(sizeof(struct nvmf_disc_rsp_page_hdr) != 1024);
diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index 40534d9a86d1..57f619aa7cba 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -47,6 +47,14 @@ void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id)
 	id->nows = to0based(ql->io_opt / ql->logical_block_size);
 }
 
+void nvmet_bdev_ns_disable(struct nvmet_ns *ns)
+{
+	if (ns->bdev) {
+		blkdev_put(ns->bdev, FMODE_WRITE | FMODE_READ);
+		ns->bdev = NULL;
+	}
+}
+
 static void nvmet_bdev_ns_enable_integrity(struct nvmet_ns *ns)
 {
 	struct blk_integrity *bi = bdev_get_integrity(ns->bdev);
@@ -86,15 +94,15 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
 	if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY_T10))
 		nvmet_bdev_ns_enable_integrity(ns);
 
-	return 0;
-}
-
-void nvmet_bdev_ns_disable(struct nvmet_ns *ns)
-{
-	if (ns->bdev) {
-		blkdev_put(ns->bdev, FMODE_WRITE | FMODE_READ);
-		ns->bdev = NULL;
+	if (bdev_is_zoned(ns->bdev)) {
+		if (!nvmet_bdev_zns_enable(ns)) {
+			nvmet_bdev_ns_disable(ns);
+			return -EINVAL;
+		}
+		ns->csi = NVME_CSI_ZNS;
 	}
+
+	return 0;
 }
 
 void nvmet_bdev_ns_revalidate(struct nvmet_ns *ns)
@@ -102,7 +110,7 @@ void nvmet_bdev_ns_revalidate(struct nvmet_ns *ns)
 	ns->size = i_size_read(ns->bdev->bd_inode);
 }
 
-static u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts)
+u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts)
 {
 	u16 status = NVME_SC_SUCCESS;
 
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index f709238014f5..a105982e8f8a 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -248,6 +248,10 @@ struct nvmet_subsys {
 	unsigned int		admin_timeout;
 	unsigned int		io_timeout;
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
+#ifdef CONFIG_BLK_DEV_ZONED
+	u8			zasl;
+#endif /* CONFIG_BLK_DEV_ZONED */
 };
 
 static inline struct nvmet_subsys *to_subsys(struct config_item *item)
@@ -333,6 +337,12 @@ struct nvmet_req {
 			struct work_struct      work;
 			bool			use_workqueue;
 		} p;
+#ifdef CONFIG_BLK_DEV_ZONED
+		struct {
+			struct bio		inline_bio;
+			struct work_struct	zmgmt_work;
+		} z;
+#endif /* CONFIG_BLK_DEV_ZONED */
 	};
 	int			sg_cnt;
 	int			metadata_sg_cnt;
@@ -352,6 +362,7 @@ struct nvmet_req {
 };
 
 extern struct workqueue_struct *buffered_io_wq;
+extern struct workqueue_struct *zbd_wq;
 
 static inline void nvmet_set_result(struct nvmet_req *req, u32 result)
 {
@@ -401,6 +412,7 @@ u16 nvmet_parse_connect_cmd(struct nvmet_req *req);
 void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id);
 u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req);
 u16 nvmet_file_parse_io_cmd(struct nvmet_req *req);
+u16 nvmet_bdev_zns_parse_io_cmd(struct nvmet_req *req);
 u16 nvmet_parse_admin_cmd(struct nvmet_req *req);
 u16 nvmet_parse_discovery_cmd(struct nvmet_req *req);
 u16 nvmet_parse_fabrics_cmd(struct nvmet_req *req);
@@ -528,6 +540,14 @@ void nvmet_ns_changed(struct nvmet_subsys *subsys, u32 nsid);
 void nvmet_bdev_ns_revalidate(struct nvmet_ns *ns);
 int nvmet_file_ns_revalidate(struct nvmet_ns *ns);
 void nvmet_ns_revalidate(struct nvmet_ns *ns);
+u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts);
+
+bool nvmet_bdev_zns_enable(struct nvmet_ns *ns);
+void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req);
+void nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req);
+void nvmet_bdev_execute_zone_mgmt_recv(struct nvmet_req *req);
+void nvmet_bdev_execute_zone_mgmt_send(struct nvmet_req *req);
+void nvmet_bdev_execute_zone_append(struct nvmet_req *req);
 
 static inline u32 nvmet_rw_data_len(struct nvmet_req *req)
 {
diff --git a/drivers/nvme/target/zns.c b/drivers/nvme/target/zns.c
new file mode 100644
index 000000000000..34135cbdd7e5
--- /dev/null
+++ b/drivers/nvme/target/zns.c
@@ -0,0 +1,622 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe ZNS-ZBD command implementation.
+ * Copyright (C) 2021 Western Digital Corporation or its affiliates.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/nvme.h>
+#include <linux/blkdev.h>
+#include "nvmet.h"
+
+/*
+ * We set the Memory Page Size Minimum (MPSMIN) for target controller to 0
+ * which gets added by 12 in the nvme_enable_ctrl() which results in 2^12 = 4k
+ * as page_shift value. When calculating the ZASL use shift by 12.
+ */
+#define NVMET_MPSMIN_SHIFT	12
+
+static inline u8 nvmet_zasl(unsigned int zone_append_sects)
+{
+	/*
+	 * Zone Append Size Limit (zasl) is expressed as a power of 2 value
+	 * with the minimum memory page size (i.e. 12) as unit.
+	 */
+	return ilog2(zone_append_sects >> (NVMET_MPSMIN_SHIFT - 9));
+}
+
+static int validate_conv_zones_cb(struct blk_zone *z,
+				  unsigned int i, void *data)
+{
+	if (z->type == BLK_ZONE_TYPE_CONVENTIONAL)
+		return -EOPNOTSUPP;
+	return 0;
+}
+
+bool nvmet_bdev_zns_enable(struct nvmet_ns *ns)
+{
+	struct request_queue *q = ns->bdev->bd_disk->queue;
+	u8 zasl = nvmet_zasl(queue_max_zone_append_sectors(q));
+	struct gendisk *bd_disk = ns->bdev->bd_disk;
+	int ret;
+
+	if (ns->subsys->zasl) {
+		if (ns->subsys->zasl > zasl)
+			return false;
+	}
+	ns->subsys->zasl = zasl;
+
+	/*
+	 * Generic zoned block devices may have a smaller last zone which is
+	 * not supported by ZNS. Exclude zoned drives that have such smaller
+	 * last zone.
+	 */
+	if (get_capacity(bd_disk) & (bdev_zone_sectors(ns->bdev) - 1))
+		return false;
+	/*
+	 * ZNS does not define a conventional zone type. If the underlying
+	 * device has a bitmap set indicating the existence of conventional
+	 * zones, reject the device. Otherwise, use report zones to detect if
+	 * the device has conventional zones.
+	 */
+	if (ns->bdev->bd_disk->queue->conv_zones_bitmap)
+		return false;
+
+	ret = blkdev_report_zones(ns->bdev, 0, blkdev_nr_zones(bd_disk),
+				  validate_conv_zones_cb, NULL);
+	if (ret < 0)
+		return false;
+
+	ns->blksize_shift = blksize_bits(bdev_logical_block_size(ns->bdev));
+
+	return true;
+}
+
+void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req)
+{
+	u8 zasl = req->sq->ctrl->subsys->zasl;
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	struct nvme_id_ctrl_zns *id;
+	u16 status;
+
+	id = kzalloc(sizeof(*id), GFP_KERNEL);
+	if (!id) {
+		status = NVME_SC_INTERNAL;
+		goto out;
+	}
+
+	if (ctrl->ops->get_mdts)
+		id->zasl = min_t(u8, ctrl->ops->get_mdts(ctrl), zasl);
+	else
+		id->zasl = zasl;
+
+	status = nvmet_copy_to_sgl(req, 0, id, sizeof(*id));
+
+	kfree(id);
+out:
+	nvmet_req_complete(req, status);
+}
+
+void nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req)
+{
+	struct nvme_id_ns_zns *id_zns;
+	u64 zsze;
+	u16 status;
+
+	if (le32_to_cpu(req->cmd->identify.nsid) == NVME_NSID_ALL) {
+		req->error_loc = offsetof(struct nvme_identify, nsid);
+		status = NVME_SC_INVALID_NS | NVME_SC_DNR;
+		goto out;
+	}
+
+	id_zns = kzalloc(sizeof(*id_zns), GFP_KERNEL);
+	if (!id_zns) {
+		status = NVME_SC_INTERNAL;
+		goto out;
+	}
+
+	status = nvmet_req_find_ns(req);
+	if (status) {
+		status = NVME_SC_INTERNAL;
+		goto done;
+	}
+
+	if (!bdev_is_zoned(req->ns->bdev)) {
+		req->error_loc = offsetof(struct nvme_identify, nsid);
+		status = NVME_SC_INVALID_NS | NVME_SC_DNR;
+		goto done;
+	}
+
+	nvmet_ns_revalidate(req->ns);
+	zsze = (bdev_zone_sectors(req->ns->bdev) << 9) >>
+					req->ns->blksize_shift;
+	id_zns->lbafe[0].zsze = cpu_to_le64(zsze);
+	id_zns->mor = cpu_to_le32(bdev_max_open_zones(req->ns->bdev));
+	id_zns->mar = cpu_to_le32(bdev_max_active_zones(req->ns->bdev));
+
+done:
+	status = nvmet_copy_to_sgl(req, 0, id_zns, sizeof(*id_zns));
+	kfree(id_zns);
+out:
+	nvmet_req_complete(req, status);
+}
+
+static u16 nvmet_bdev_validate_zone_mgmt_recv(struct nvmet_req *req)
+{
+	sector_t sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
+	u32 out_bufsize = (le32_to_cpu(req->cmd->zmr.numd) + 1) << 2;
+
+	if (sect >= get_capacity(req->ns->bdev->bd_disk)) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, slba);
+		return NVME_SC_LBA_RANGE | NVME_SC_DNR;
+	}
+
+	if (out_bufsize < sizeof(struct nvme_zone_report)) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, numd);
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+	}
+
+	if (req->cmd->zmr.zra != NVME_ZRA_ZONE_REPORT) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, zra);
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+	}
+
+	switch (req->cmd->zmr.pr) {
+	case 0:
+	case 1:
+		break;
+	default:
+		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, pr);
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+	}
+
+	switch (req->cmd->zmr.zrasf) {
+	case NVME_ZRASF_ZONE_REPORT_ALL:
+	case NVME_ZRASF_ZONE_STATE_EMPTY:
+	case NVME_ZRASF_ZONE_STATE_IMP_OPEN:
+	case NVME_ZRASF_ZONE_STATE_EXP_OPEN:
+	case NVME_ZRASF_ZONE_STATE_CLOSED:
+	case NVME_ZRASF_ZONE_STATE_FULL:
+	case NVME_ZRASF_ZONE_STATE_READONLY:
+	case NVME_ZRASF_ZONE_STATE_OFFLINE:
+		break;
+	default:
+		req->error_loc =
+			offsetof(struct nvme_zone_mgmt_recv_cmd, zrasf);
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+	}
+
+	return NVME_SC_SUCCESS;
+}
+
+struct nvmet_report_zone_data {
+	struct nvmet_req *req;
+	u64 out_buf_offset;
+	u64 out_nr_zones;
+	u64 nr_zones;
+	u8 zrasf;
+};
+
+static int nvmet_bdev_report_zone_cb(struct blk_zone *z, unsigned i, void *d)
+{
+	static const unsigned int nvme_zrasf_to_blk_zcond[] = {
+		[NVME_ZRASF_ZONE_STATE_EMPTY]	 = BLK_ZONE_COND_EMPTY,
+		[NVME_ZRASF_ZONE_STATE_IMP_OPEN] = BLK_ZONE_COND_IMP_OPEN,
+		[NVME_ZRASF_ZONE_STATE_EXP_OPEN] = BLK_ZONE_COND_EXP_OPEN,
+		[NVME_ZRASF_ZONE_STATE_CLOSED]	 = BLK_ZONE_COND_CLOSED,
+		[NVME_ZRASF_ZONE_STATE_READONLY] = BLK_ZONE_COND_READONLY,
+		[NVME_ZRASF_ZONE_STATE_FULL]	 = BLK_ZONE_COND_FULL,
+		[NVME_ZRASF_ZONE_STATE_OFFLINE]	 = BLK_ZONE_COND_OFFLINE,
+	};
+	struct nvmet_report_zone_data *rz = d;
+
+	if (rz->zrasf != NVME_ZRASF_ZONE_REPORT_ALL &&
+	    z->cond != nvme_zrasf_to_blk_zcond[rz->zrasf])
+		return 0;
+
+	if (rz->nr_zones < rz->out_nr_zones) {
+		struct nvme_zone_descriptor zdesc = { };
+		u16 status;
+
+		zdesc.zcap = nvmet_sect_to_lba(rz->req->ns, z->capacity);
+		zdesc.zslba = nvmet_sect_to_lba(rz->req->ns, z->start);
+		zdesc.wp = nvmet_sect_to_lba(rz->req->ns, z->wp);
+		zdesc.za = z->reset ? 1 << 2 : 0;
+		zdesc.zs = z->cond << 4;
+		zdesc.zt = z->type;
+
+		status = nvmet_copy_to_sgl(rz->req, rz->out_buf_offset, &zdesc,
+					   sizeof(zdesc));
+		if (status)
+			return -EINVAL;
+
+		rz->out_buf_offset += sizeof(zdesc);
+	}
+
+	rz->nr_zones++;
+
+	return 0;
+}
+
+static unsigned long nvmet_req_nr_zones_from_slba(struct nvmet_req *req)
+{
+	unsigned int sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
+
+	return blkdev_nr_zones(req->ns->bdev->bd_disk) -
+		(sect >> ilog2(bdev_zone_sectors(req->ns->bdev)));
+}
+
+static unsigned long get_nr_zones_from_buf(struct nvmet_req *req, u32 bufsize)
+{
+	if (bufsize <= sizeof(struct nvme_zone_report))
+		return 0;
+
+	return (bufsize - sizeof(struct nvme_zone_report)) /
+		sizeof(struct nvme_zone_descriptor);
+}
+
+void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
+{
+	struct nvmet_req *req = container_of(w, struct nvmet_req, z.zmgmt_work);
+	sector_t start_sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
+	unsigned long req_slba_nr_zones = nvmet_req_nr_zones_from_slba(req);
+	u32 out_bufsize = (le32_to_cpu(req->cmd->zmr.numd) + 1) << 2;
+	__le64 nr_zones;
+	u16 status;
+	int ret;
+	struct nvmet_report_zone_data rz_data = {
+		.out_nr_zones = get_nr_zones_from_buf(req, out_bufsize),
+		/* leave the place for report zone header */
+		.out_buf_offset = sizeof(struct nvme_zone_report),
+		.zrasf = req->cmd->zmr.zrasf,
+		.nr_zones = 0,
+		.req = req,
+	};
+
+	status = nvmet_bdev_validate_zone_mgmt_recv(req);
+	if (status)
+		goto out;
+
+	if (!req_slba_nr_zones) {
+		status = NVME_SC_SUCCESS;
+		goto out;
+	}
+
+	ret = blkdev_report_zones(req->ns->bdev, start_sect, req_slba_nr_zones,
+				 nvmet_bdev_report_zone_cb, &rz_data);
+	if (ret < 0) {
+		status = NVME_SC_INTERNAL;
+		goto out;
+	}
+
+	/*
+	 * When partial bit is set nr_zones must indicate the number of zone
+	 * descriptors actually transferred.
+	 */
+	if (req->cmd->zmr.pr)
+		rz_data.nr_zones = min(rz_data.nr_zones, rz_data.out_nr_zones);
+
+	nr_zones = cpu_to_le64(rz_data.nr_zones);
+	status = nvmet_copy_to_sgl(req, 0, &nr_zones, sizeof(nr_zones));
+
+out:
+	nvmet_req_complete(req, status);
+}
+
+void nvmet_bdev_execute_zone_mgmt_recv(struct nvmet_req *req)
+{
+	INIT_WORK(&req->z.zmgmt_work, nvmet_bdev_zone_zmgmt_recv_work);
+	queue_work(zbd_wq, &req->z.zmgmt_work);
+}
+
+static inline enum req_opf zsa_req_op(u8 zsa)
+{
+	switch (zsa) {
+	case NVME_ZONE_OPEN:
+		return REQ_OP_ZONE_OPEN;
+	case NVME_ZONE_CLOSE:
+		return REQ_OP_ZONE_CLOSE;
+	case NVME_ZONE_FINISH:
+		return REQ_OP_ZONE_FINISH;
+	case NVME_ZONE_RESET:
+		return REQ_OP_ZONE_RESET;
+	default:
+		return REQ_OP_LAST;
+	}
+}
+
+static u16 blkdev_zone_mgmt_errno_to_nvme_status(int ret)
+{
+	switch (ret) {
+	case 0:
+		return NVME_SC_SUCCESS;
+	case -EINVAL:
+	case -EIO:
+		return NVME_SC_ZONE_INVALID_TRANSITION | NVME_SC_DNR;
+	default:
+		return NVME_SC_INTERNAL;
+	}
+}
+
+struct nvmet_zone_mgmt_send_all_data {
+	unsigned long *zbitmap;
+	struct nvmet_req *req;
+};
+
+static int zmgmt_send_scan_cb(struct blk_zone *z, unsigned i, void *d)
+{
+	struct nvmet_zone_mgmt_send_all_data *data = d;
+
+	switch (zsa_req_op(data->req->cmd->zms.zsa)) {
+	case REQ_OP_ZONE_OPEN:
+		switch (z->cond) {
+		case BLK_ZONE_COND_CLOSED:
+			break;
+		default:
+			return 0;
+		}
+		break;
+	case REQ_OP_ZONE_CLOSE:
+		switch (z->cond) {
+		case BLK_ZONE_COND_IMP_OPEN:
+		case BLK_ZONE_COND_EXP_OPEN:
+			break;
+		default:
+			return 0;
+		}
+		break;
+	case REQ_OP_ZONE_FINISH:
+		switch (z->cond) {
+		case BLK_ZONE_COND_IMP_OPEN:
+		case BLK_ZONE_COND_EXP_OPEN:
+		case BLK_ZONE_COND_CLOSED:
+			break;
+		default:
+			return 0;
+		}
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	set_bit(i, (unsigned long *)(data->zbitmap));
+
+	return 0;
+}
+
+static u16 nvmet_bdev_zone_mgmt_emulate_all(struct nvmet_req *req)
+{
+	struct block_device *bdev = req->ns->bdev;
+	unsigned int nr_zones = blkdev_nr_zones(bdev->bd_disk);
+	struct request_queue *q = bdev_get_queue(bdev);
+	struct bio *bio = NULL;
+	sector_t sector = 0;
+	int ret;
+	struct nvmet_zone_mgmt_send_all_data d = {
+		.req = req,
+	};
+
+	d.zbitmap = kcalloc_node(BITS_TO_LONGS(nr_zones), sizeof(*(d.zbitmap)),
+				 GFP_NOIO, q->node);
+	if (!d.zbitmap) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	/* Scan and build bitmap of the eligible zones */
+	ret = blkdev_report_zones(bdev, 0, nr_zones, zmgmt_send_scan_cb, &d);
+	if (ret != nr_zones) {
+		if (ret > 0)
+			ret = -EIO;
+		goto out;
+	} else {
+		/* We scanned all the zones */
+		ret = 0;
+	}
+
+	while (sector < get_capacity(bdev->bd_disk)) {
+		if (test_bit(blk_queue_zone_no(q, sector), d.zbitmap)) {
+			bio = blk_next_bio(bio, 0, GFP_KERNEL);
+			bio->bi_opf = zsa_req_op(req->cmd->zms.zsa) | REQ_SYNC;
+			bio->bi_iter.bi_sector = sector;
+			bio_set_dev(bio, bdev);
+			/* This may take a while, so be nice to others */
+			cond_resched();
+		}
+		sector += blk_queue_zone_sectors(q);
+	}
+
+	if (bio) {
+		ret = submit_bio_wait(bio);
+		bio_put(bio);
+	}
+
+out:
+	kfree(d.zbitmap);
+
+	return blkdev_zone_mgmt_errno_to_nvme_status(ret);
+}
+
+static u16 nvmet_bdev_execute_zmgmt_send_all(struct nvmet_req *req)
+{
+	int ret;
+
+	switch (zsa_req_op(req->cmd->zms.zsa)) {
+	case REQ_OP_ZONE_RESET:
+		ret = blkdev_zone_mgmt(req->ns->bdev, REQ_OP_ZONE_RESET, 0,
+				       get_capacity(req->ns->bdev->bd_disk),
+				       GFP_KERNEL);
+		if (ret < 0)
+			return blkdev_zone_mgmt_errno_to_nvme_status(ret);
+		break;
+	case REQ_OP_ZONE_OPEN:
+	case REQ_OP_ZONE_CLOSE:
+	case REQ_OP_ZONE_FINISH:
+		return nvmet_bdev_zone_mgmt_emulate_all(req);
+	default:
+		/* this is needed to quiet compiler warning */
+		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, zsa);
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+	}
+
+	return NVME_SC_SUCCESS;
+}
+
+static void nvmet_bdev_zmgmt_send_work(struct work_struct *w)
+{
+	struct nvmet_req *req = container_of(w, struct nvmet_req, z.zmgmt_work);
+	sector_t sect = nvmet_lba_to_sect(req->ns, req->cmd->zms.slba);
+	enum req_opf op = zsa_req_op(req->cmd->zms.zsa);
+	struct block_device *bdev = req->ns->bdev;
+	sector_t zone_sectors = bdev_zone_sectors(bdev);
+	u16 status = NVME_SC_SUCCESS;
+	int ret;
+
+	if (op == REQ_OP_LAST) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, zsa);
+		status = NVME_SC_ZONE_INVALID_TRANSITION | NVME_SC_DNR;
+		goto out;
+	}
+
+	/* when select all bit is set slba field is ignored */
+	if (req->cmd->zms.select_all) {
+		status = nvmet_bdev_execute_zmgmt_send_all(req);
+		goto out;
+	}
+
+	if (sect >= get_capacity(bdev->bd_disk)) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, slba);
+		status = NVME_SC_LBA_RANGE | NVME_SC_DNR;
+		goto out;
+	}
+
+	if (sect & (zone_sectors - 1)) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, slba);
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		goto out;
+	}
+
+	ret = blkdev_zone_mgmt(bdev, op, sect, zone_sectors, GFP_KERNEL);
+	if (ret < 0)
+		status = blkdev_zone_mgmt_errno_to_nvme_status(ret);
+
+out:
+	nvmet_req_complete(req, status);
+}
+
+void nvmet_bdev_execute_zone_mgmt_send(struct nvmet_req *req)
+{
+	INIT_WORK(&req->z.zmgmt_work, nvmet_bdev_zmgmt_send_work);
+	queue_work(zbd_wq, &req->z.zmgmt_work);
+}
+
+static void nvmet_bdev_zone_append_bio_done(struct bio *bio)
+{
+	struct nvmet_req *req = bio->bi_private;
+
+	if (bio->bi_status == BLK_STS_OK) {
+		req->cqe->result.u64 =
+			nvmet_sect_to_lba(req->ns, bio->bi_iter.bi_sector);
+	}
+
+	nvmet_req_complete(req, blk_to_nvme_status(req, bio->bi_status));
+	nvmet_req_bio_put(req, bio);
+}
+
+void nvmet_bdev_execute_zone_append(struct nvmet_req *req)
+{
+	sector_t sect = nvmet_lba_to_sect(req->ns, req->cmd->rw.slba);
+	u16 nr_sect = nvmet_lba_to_sect(req->ns, req->cmd->rw.length);
+	u16 status = NVME_SC_SUCCESS;
+	unsigned int total_len = 0;
+	struct scatterlist *sg;
+	struct bio *bio;
+	int sg_cnt;
+
+	/* Request is completed on len mismatch in nvmet_check_transter_len() */
+	if (!nvmet_check_transfer_len(req, nvmet_rw_data_len(req)))
+		return;
+
+	if (!req->sg_cnt) {
+		nvmet_req_complete(req, 0);
+		return;
+	}
+
+	if (sect >= get_capacity(req->ns->bdev->bd_disk)) {
+		req->error_loc = offsetof(struct nvme_rw_command, slba);
+		status = NVME_SC_LBA_RANGE | NVME_SC_DNR;
+		goto out;
+	}
+
+	if ((sect + nr_sect > get_capacity(req->ns->bdev->bd_disk))) {
+		req->error_loc = offsetof(struct nvme_rw_command, length);
+		status = NVME_SC_LBA_RANGE | NVME_SC_DNR;
+		goto out;
+	}
+
+	if (sect & (bdev_zone_sectors(req->ns->bdev) - 1)) {
+		req->error_loc = offsetof(struct nvme_rw_command, slba);
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		goto out;
+	}
+
+	if (nvmet_use_inline_bvec(req)) {
+		bio = &req->z.inline_bio;
+		bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec));
+	} else {
+		bio = bio_alloc(GFP_KERNEL, req->sg_cnt);
+	}
+
+	bio->bi_opf = REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE;
+	bio->bi_end_io = nvmet_bdev_zone_append_bio_done;
+	bio_set_dev(bio, req->ns->bdev);
+	bio->bi_iter.bi_sector = sect;
+	bio->bi_private = req;
+	if (req->cmd->rw.control & cpu_to_le16(NVME_RW_FUA))
+		bio->bi_opf |= REQ_FUA;
+
+	for_each_sg(req->sg, sg, req->sg_cnt, sg_cnt) {
+		struct page *p = sg_page(sg);
+		unsigned int l = sg->length;
+		unsigned int o = sg->offset;
+		unsigned int ret;
+
+		ret = bio_add_zone_append_page(bio, p, l, o);
+		if (ret != sg->length) {
+			status = NVME_SC_INTERNAL;
+			goto out_put_bio;
+		}
+		total_len += sg->length;
+	}
+
+	if (total_len != nvmet_rw_data_len(req)) {
+		status = NVME_SC_INTERNAL | NVME_SC_DNR;
+		goto out_put_bio;
+	}
+
+	submit_bio(bio);
+	return;
+
+out_put_bio:
+	nvmet_req_bio_put(req, bio);
+out:
+	nvmet_req_complete(req, status);
+}
+
+u16 nvmet_bdev_zns_parse_io_cmd(struct nvmet_req *req)
+{
+	struct nvme_command *cmd = req->cmd;
+
+	switch (cmd->common.opcode) {
+	case nvme_cmd_zone_append:
+		req->execute = nvmet_bdev_execute_zone_append;
+		return 0;
+	case nvme_cmd_zone_mgmt_recv:
+		req->execute = nvmet_bdev_execute_zone_mgmt_recv;
+		return 0;
+	case nvme_cmd_zone_mgmt_send:
+		req->execute = nvmet_bdev_execute_zone_mgmt_send;
+		return 0;
+	default:
+		return nvmet_bdev_parse_io_cmd(req);
+	}
+}
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index c7ba83144d52..cb1197f1cfed 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -944,6 +944,13 @@ struct nvme_zone_mgmt_recv_cmd {
 enum {
 	NVME_ZRA_ZONE_REPORT		= 0,
 	NVME_ZRASF_ZONE_REPORT_ALL	= 0,
+	NVME_ZRASF_ZONE_STATE_EMPTY	= 0x01,
+	NVME_ZRASF_ZONE_STATE_IMP_OPEN	= 0x02,
+	NVME_ZRASF_ZONE_STATE_EXP_OPEN	= 0x03,
+	NVME_ZRASF_ZONE_STATE_CLOSED	= 0x04,
+	NVME_ZRASF_ZONE_STATE_READONLY	= 0x05,
+	NVME_ZRASF_ZONE_STATE_FULL	= 0x06,
+	NVME_ZRASF_ZONE_STATE_OFFLINE	= 0x07,
 	NVME_REPORT_ZONE_PARTIAL	= 1,
 };
 
-- 
2.22.1



From belegdol at gmail.com  Mon Jun  7 23:01:38 2021
From: belegdol at gmail.com (Julian Sikorski)
Date: Tue, 8 Jun 2021 08:01:38 +0200
Subject: [PATCH v6 2/2] ACPI: Add quirks for AMD Renoir/Lucienne CPUs to
 force the D3 hint
In-Reply-To: <20210607173156.5548-2-mario.limonciello@amd.com>
References: <20210607173156.5548-1-mario.limonciello@amd.com>
 <20210607173156.5548-2-mario.limonciello@amd.com>
Message-ID: <328f5fd7-f8d1-0543-a328-44ef0641c64b@gmail.com>

W dniu 07.06.2021 o?19:31, Mario Limonciello pisze:
> AMD systems from Renoir and Lucienne require that the NVME controller
> is put into D3 over a Modern Standby / suspend-to-idle
> cycle.  This is "typically" accomplished using the `StorageD3Enable`
> property in the _DSD, but this property was introduced after many
> of these systems launched and most OEM systems don't have it in
> their BIOS.
> 
> On AMD Renoir without these drives going into D3 over suspend-to-idle
> the resume will fail with the NVME controller being reset and a trace
> like this in the kernel logs:
> ```
> [   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
> [   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
> [   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
> [   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
> [   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
> [   95.332843] nvme nvme0: Abort status: 0x371
> [   95.332852] nvme nvme0: Abort status: 0x371
> [   95.332856] nvme nvme0: Abort status: 0x371
> [   95.332859] nvme nvme0: Abort status: 0x371
> [   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
> [   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
> ```
> 
> The Microsoft documentation for StorageD3Enable mentioned that Windows has
> a hardcoded allowlist for D3 support, which was used for these platforms.
> Introduce quirks to hardcode them for Linux as well.
> 
> As this property is now "standardized", OEM systems using AMD Cezanne and
> newer APU's have adopted this property, and quirks like this should not be
> necessary.
> 
> CC: Julian Sikorski <belegdol at gmail.com>
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> ---

Tested-by: Julian Sikorski <belegdol at gmail.com>


From pmladek at suse.com  Tue Jun  8 02:28:53 2021
From: pmladek at suse.com (Petr Mladek)
Date: Tue, 8 Jun 2021 11:28:53 +0200
Subject: [RFC PATCH v5 01/27] nvme-tcp-offload: Add nvme-tcp-offload -
 NVMeTCP HW offload ULP
In-Reply-To: <20210519111340.20613-2-smalin@marvell.com>
References: <20210519111340.20613-1-smalin@marvell.com>
 <20210519111340.20613-2-smalin@marvell.com>
Message-ID: <YL84VamVh78Ds2Eg@alley>

On Wed 2021-05-19 14:13:14, Shai Malin wrote:
> This patch will present the structure for the NVMeTCP offload common
> layer driver. This module is added under "drivers/nvme/host/" and future
> offload drivers which will register to it will be placed under
> "drivers/nvme/hw".
> This new driver will be enabled by the Kconfig "NVM Express over Fabrics
> TCP offload commmon layer".
> In order to support the new transport type, for host mode, no change is
> needed.
> 
> Each new vendor-specific offload driver will register to this ULP during
> its probe function, by filling out the nvme_tcp_ofld_dev->ops and
> nvme_tcp_ofld_dev->private_data and calling nvme_tcp_ofld_register_dev
> with the initialized struct.
> 
> The internal implementation:
> - tcp-offload.h:
>   Includes all common structs and ops to be used and shared by offload
>   drivers.
> 
> - tcp-offload.c:
>   Includes the init function which registers as a NVMf transport just
>   like any other transport.
> 
> Acked-by: Igor Russkikh <irusskikh at marvell.com>
> Signed-off-by: Dean Balandin <dbalandin at marvell.com>
> Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
> Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
> Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
> Signed-off-by: Ariel Elior <aelior at marvell.com>
> Signed-off-by: Shai Malin <smalin at marvell.com>
> Reviewed-by: Hannes Reinecke <hare at suse.de>

> --- a/drivers/nvme/host/Kconfig
> +++ b/drivers/nvme/host/Kconfig
> @@ -84,3 +84,19 @@ config NVME_TCP
>  	  from https://github.com/linux-nvme/nvme-cli.
>  
>  	  If unsure, say N.
> +
> +config NVME_TCP_OFFLOAD
> +	tristate "NVM Express over Fabrics TCP offload common layer"
> +	default m

Is this intentional, please?

> +	depends on INET
> +	depends on BLK_DEV_NVME
> +	select NVME_FABRICS
> +	help
> +	  This provides support for the NVMe over Fabrics protocol using
> +	  the TCP offload transport. This allows you to use remote block devices
> +	  exported using the NVMe protocol set.
> +
> +	  To configure a NVMe over Fabrics controller use the nvme-cli tool
> +	  from https://github.com/linux-nvme/nvme-cli.
> +
> +	  If unsure, say N.

I would expect that the default would be "n" so that people that are
not sure or do not care about NWMe just take the default. IMHO, it is
the usual behavior.

Best Regards,
Petr


From geert+renesas at glider.be  Tue Jun  8 03:56:09 2021
From: geert+renesas at glider.be (Geert Uytterhoeven)
Date: Tue,  8 Jun 2021 12:56:09 +0200
Subject: [PATCH] nvme: NVME_TCP_OFFLOAD should not default to m
Message-ID: <39b1a3684880e1d85ef76e34403886e8f1d22508.1623149635.git.geert+renesas@glider.be>

The help text for the symbol controlling support for the NVM Express
over Fabrics TCP offload common layer suggests to not enable this
support when unsure.

Hence drop the "default m", which actually means "default y" if
CONFIG_MODULES is not enabled.

Fixes: f0e8cb6106da2703 ("nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW offload ULP")
Signed-off-by: Geert Uytterhoeven <geert+renesas at glider.be>
---
 drivers/nvme/host/Kconfig | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 9c6f4d776daf14cf..f76cc4690bfc37bc 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -88,7 +88,6 @@ config NVME_TCP
 
 config NVME_TCP_OFFLOAD
 	tristate "NVM Express over Fabrics TCP offload common layer"
-	default m
 	depends on BLOCK
 	depends on INET
 	select NVME_CORE
-- 
2.25.1



From rafael at kernel.org  Tue Jun  8 04:20:43 2021
From: rafael at kernel.org (Rafael J. Wysocki)
Date: Tue, 8 Jun 2021 13:20:43 +0200
Subject: [PATCH v6 1/2] ACPI: Move check for _DSD StorageD3Enable property
 to acpi
In-Reply-To: <20210608053546.GA14116@lst.de>
References: <20210607173156.5548-1-mario.limonciello@amd.com>
 <20210608053546.GA14116@lst.de>
Message-ID: <CAJZ5v0gMgwJ9kq_mo_dYoTUAqQzhXmrgp9wnY4TRPU1pJp=TyQ@mail.gmail.com>

On Tue, Jun 8, 2021 at 7:35 AM Christoph Hellwig <hch at lst.de> wrote:
>
> On Mon, Jun 07, 2021 at 12:31:55PM -0500, Mario Limonciello wrote:
> > +/**
> > + * acpi_storage_d3 - Check if a storage device should use D3.

Let's be specific about what D3 means here in the first place and
that's D3hot AFAICS.

And the comment should be something like "Check whether or not to use
D3hot in the suspend path".

> > + * @dev: Device to check
> > + *
> > + * Returns %true if @dev should be put into D3 when the ->suspend method is
> > + * called, else %false.  The name of this function is somewhat misleading
> > + * as it has nothing to do with storage except for the name of the ACPI
> > + * property.  On some platforms resume will not work if this hint is ignored.

I would write it this way:

"Return %true if the platform firmware wants @dev to be programmed
into D3hot in the suspend path, or %false when there is no specific
preference. On some platforms, if this hint is ignored, @dev may
remain unresponsive after suspending the platform as a whole."

And I'm not sure if it is necessary to mention "storage" in this comment at all.

> > + *
> > + */
>
> I still do not like this comment.  There is nothing about only using this
> for storage devices.  It is specific to a PCIe slot, and if I plug
> something that is not a storage device into it the same restrictions
> still apply.

Originally, it was about storage devices IIUC (and that's why the
property name is what it is).

IIRC PCIe link management was broken in some NVMe drives and putting
them into D3hot would allow the link to go down later and it would
never go up again.  So you-know-who decided to avoid using D3hot for
all NVMe drives and they were generally characterized as "storage".
Alas, that turned out to break other things depending on the devices
being put into D3hot (like PS_ON) so the property was invented to
allow OEMs to mark the configurations including the devices that
actually worked (and Linux was/is on the receiving end). And
originally the property applied to the endpoint device under the root
port it was present for (my personal opinion about this doesn't really
matter, so let me avoid wasting time on expressing it).

Now, it evidently has been re-purposed to mark PCIe root ports that
require devices under them to be put into D3hot in the suspend path
(and it now means any device under the root port with this property,
at least in some cases).

> Also no need for the empty-ish line at the end.

Right.


From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:43 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:43 +0530
Subject: [PATCH v11 00/13] blkcg:Support to track FC storage blk io traffic
Message-ID: <20210608043556.274139-1-muneendra.kumar@broadcom.com>

From: Muneendra <muneendra.kumar at broadcom.com>

This Patch added a unique application identifier i.e
app_id  knob to  blkcg which allows identification of traffic
sources at an individual cgroup based Applications
(ex:virtual machine (VM))level in both host and
fabric infrastructure.

Added a new sysfs attribute appid_store to set the application identfier
in  the blkcg associted with cgroup id under
/sys/class/fc/fc_udev_device/*

With this new interface the user can set the application identfier
in  the blkcg associted with cgroup id.

This capability can be utilized by multiple block transport infrastructure
like fc,iscsi,roce.

Existing FC fabric will use this feature and the description of
the use case is below.

Various virtualization technologies used in Fibre Channel
SAN deployments have created the opportunity to identify
and associate traffic with specific virtualized applications.
The concepts behind the T11 Application Services standard is
to provide the general mechanisms needed to identify
virtualized services.
It enables the Fabric and the storage targets to
identify, monitor, and handle FC traffic
based on vm tags by inserting application specific identification
into the FC frame.

The patches were cut against  5.14/scsi-queue tree

v11:
add Tejun Heo Acks.
Add comment on race condition

v10:
Fixed the spelling mistakes and function name corrections
Removed the redundant code

v9:
Addressed the issues reported by kernel test robot
Replaced lpfc_get_vmid_from_hashtable with the
generic kernel based hashtable (include/linux/hashtable.h)
and made the changes in the code accordingly.
Addressed the locking issue and also merged few patches

v8:
Modified the structure member,log messages and function declarations
Added proper error codes and return values

v7:
Modified the Kconfig comments

v6:
Addressed the issues reported by kernel test robot
Modified the Kconfig files as per standard

v5:
Renamed the function cgroup_get_from_kernfs_id to
cgroup_get_from_id.

Moved the input validation at the beginning of the function in 
Renamed the arguments appropriatley.

Changed Return code to non-numeric/SymbolChanged Return code
to non-numeric/Symbol

Modified the comments.

v4:
Addressed the error reported by  kernel test robot

v3:
removed RFC.

Renamed the functions and app_id to more specific
Addressed the reference leaks in blkcg_set_app_identifier
Added a new config BLK_CGROUP_FC_APPID and made changes to 
select the same under SCSI_FC_ATTRS

V2:
renamed app_identifier to app_id.
removed the  sysfs interface blkio.app_identifie under
/sys/fs/cgroup/blkio
Ported the patch on top of 5.10/scsi-queue.
Removed redundant code due to changes since last submit.
Added a fix for issuing QFPA command.



Gaurav Srivastava (10):
  lpfc: vmid: Add the datastructure for supporting VMID in lpfc
  lpfc: vmid: VMID params initialization
  lpfc: vmid: Add support for vmid in mailbox command, does vmid
    resource allocation and vmid cleanup
  lpfc: vmid: Implements ELS commands for appid patch
  lpfc: vmid: Functions to manage vmids
  lpfc: vmid: Implements CT commands for appid.
  lpfc: vmid: Appends the vmid in the wqe before sending
  lpfc: vmid: Timeout implementation for vmid
  lpfc: vmid: Adding qfpa and vmid timeout check in worker thread
  lpfc: vmid: Introducing vmid in io path.

Muneendra (3):
  cgroup: Added cgroup_get_from_id
  blkcg: Added a app identifier support for blkcg
  nvme: Added a newsysfs attribute appid_store

 block/Kconfig                    |   9 +
 drivers/nvme/host/fc.c           |  73 +++++-
 drivers/scsi/Kconfig             |  13 ++
 drivers/scsi/lpfc/lpfc.h         | 122 +++++++++++
 drivers/scsi/lpfc/lpfc_attr.c    |  48 ++++
 drivers/scsi/lpfc/lpfc_crtn.h    |  11 +
 drivers/scsi/lpfc/lpfc_ct.c      | 255 +++++++++++++++++++++
 drivers/scsi/lpfc/lpfc_disc.h    |   1 +
 drivers/scsi/lpfc/lpfc_els.c     | 366 ++++++++++++++++++++++++++++++-
 drivers/scsi/lpfc/lpfc_hbadisc.c | 148 +++++++++++++
 drivers/scsi/lpfc/lpfc_hw.h      | 124 ++++++++++-
 drivers/scsi/lpfc/lpfc_hw4.h     |  12 +
 drivers/scsi/lpfc/lpfc_init.c    | 104 +++++++++
 drivers/scsi/lpfc/lpfc_mbox.c    |   6 +
 drivers/scsi/lpfc/lpfc_scsi.c    | 321 +++++++++++++++++++++++++++
 drivers/scsi/lpfc/lpfc_sli.c     |  23 ++
 drivers/scsi/lpfc/lpfc_sli.h     |   8 +
 include/linux/blk-cgroup.h       |  64 ++++++
 include/linux/cgroup.h           |   6 +
 kernel/cgroup/cgroup.c           |  26 +++
 20 files changed, 1731 insertions(+), 9 deletions(-)

-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:45 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:45 +0530
Subject: [PATCH v11 02/13] blkcg: Added a app identifier support for blkcg
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-3-muneendra.kumar@broadcom.com>

From: Muneendra <muneendra.kumar at broadcom.com>

This Patch added a unique application identifier i.e
fc_app_id  member in blkcg which allows identification of traffic
sources at an individual cgroup based Applications
(ex:virtual machine (VM))level in both host and
fabric infrastructure.

Added a new function blkcg_get_fc_appid to
grab the app identifier associated with a bio.

Added a new function blkcg_set_fc_appid to
set the app identifier in a blkcgrp associated with cgroup id

Added a new config BLK_CGROUP_FC_APPID and moved the changes
under this config

Merged the patch 16 of previous version in which we
added a new config FC_APPID to select BLK_CGROUP_FC_APPID which Enable
support to track FC io Traffic.

Acked-by: Tejun Heo <tj at kernel.org>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Muneendra <muneendra.kumar at broadcom.com>

---
v11:
Added a comment on race condition

v10:
No change

v9:
Merged patch16 of previosu version to this patch
where we are using Kconfig settings

v8:
No change

v7:
Modified the Kconfig file

v6:
Modified the Kconfig file as per standard specified
in Documentation/process/coding-style.rst

v5:
Renamed the arguments appropriatley
Renamed APPID_LEN  to FC_APPID_LEN
Moved the input validation at the begining of the function
Modified the comments

v4:
No change

v3:
Renamed the functions and app_id to more specific

Addressed the reference leaks in blkcg_set_app_identifier

Added a new config BLK_CGROUP_FC_APPID and moved the changes
under this config

Added blkcg_get_fc_appid,blkcg_set_fc_appid as inline functions

v2:
renamed app_identifier to app_id
removed the  sysfs interface blkio.app_identifie under
---
 block/Kconfig              |  9 ++++++
 drivers/scsi/Kconfig       | 13 ++++++++
 include/linux/blk-cgroup.h | 64 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 86 insertions(+)

diff --git a/block/Kconfig b/block/Kconfig
index a2297edfdde8..03886d105301 100644
--- a/block/Kconfig
+++ b/block/Kconfig
@@ -144,6 +144,15 @@ config BLK_CGROUP_IOLATENCY
 
 	Note, this is an experimental interface and could be changed someday.
 
+config BLK_CGROUP_FC_APPID
+	bool "Enable support to track FC I/O Traffic across cgroup applications"
+	depends on BLK_CGROUP=y
+	help
+	  Enabling this option enables the support to track FC I/O traffic across
+	  cgroup applications. It enables the Fabric and the storage targets to
+	  identify, monitor, and handle FC traffic based on VM tags by inserting
+	  application specific identification into the FC frame.
+
 config BLK_CGROUP_IOCOST
 	bool "Enable support for cost model based cgroup IO controller"
 	depends on BLK_CGROUP=y
diff --git a/drivers/scsi/Kconfig b/drivers/scsi/Kconfig
index 23e015af0e93..cd5fbd9d46ba 100644
--- a/drivers/scsi/Kconfig
+++ b/drivers/scsi/Kconfig
@@ -235,6 +235,19 @@ config SCSI_FC_ATTRS
 	  each attached FiberChannel device to sysfs, say Y.
 	  Otherwise, say N.
 
+config FC_APPID
+	bool "Enable support to track FC I/O Traffic"
+	depends on BLOCK && BLK_CGROUP
+	depends on SCSI
+	select BLK_CGROUP_FC_APPID
+	default y
+	help
+	  If you say Y here, it enables the support to track
+	  FC I/O traffic over fabric. It enables the Fabric and the
+	  storage targets to identify, monitor, and handle FC traffic
+	  based on VM tags by inserting application specific
+	  identification into the FC frame.
+
 config SCSI_ISCSI_ATTRS
 	tristate "iSCSI Transport Attributes"
 	depends on SCSI && NET
diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h
index b9f3c246c3c9..aaf9ef174302 100644
--- a/include/linux/blk-cgroup.h
+++ b/include/linux/blk-cgroup.h
@@ -30,6 +30,8 @@
 
 /* Max limits for throttle policy */
 #define THROTL_IOPS_MAX		UINT_MAX
+#define FC_APPID_LEN              129
+
 
 #ifdef CONFIG_BLK_CGROUP
 
@@ -55,6 +57,9 @@ struct blkcg {
 	struct blkcg_policy_data	*cpd[BLKCG_MAX_POLS];
 
 	struct list_head		all_blkcgs_node;
+#ifdef CONFIG_BLK_CGROUP_FC_APPID
+	char                            fc_app_id[FC_APPID_LEN];
+#endif
 #ifdef CONFIG_CGROUP_WRITEBACK
 	struct list_head		cgwb_list;
 #endif
@@ -660,4 +665,63 @@ static inline void blk_cgroup_bio_start(struct bio *bio) { }
 
 #endif	/* CONFIG_BLOCK */
 #endif	/* CONFIG_BLK_CGROUP */
+
+#ifdef CONFIG_BLK_CGROUP_FC_APPID
+/*
+ * Sets the fc_app_id field associted to blkcg
+ * @app_id: application identifier
+ * @cgrp_id: cgroup id
+ * @app_id_len: size of application identifier
+ */
+static inline int blkcg_set_fc_appid(char *app_id, u64 cgrp_id, size_t app_id_len)
+{
+	struct cgroup *cgrp;
+	struct cgroup_subsys_state *css;
+	struct blkcg *blkcg;
+	int ret  = 0;
+
+	if (app_id_len > FC_APPID_LEN)
+		return -EINVAL;
+
+	cgrp = cgroup_get_from_id(cgrp_id);
+	if (!cgrp)
+		return -ENOENT;
+	css = cgroup_get_e_css(cgrp, &io_cgrp_subsys);
+	if (!css) {
+		ret = -ENOENT;
+		goto out_cgrp_put;
+	}
+	blkcg = css_to_blkcg(css);
+	/*
+	 * There is a slight race condition on the setting of the appid,
+	 * which should not be an issue. At worst an io may not find the
+	 * appid, but it will be picked up in subsequent io.
+	 * This is no different from the io we let pass while obtaining
+	 * the vmid from the fabric.
+	 * Adding the overhead of a lock is not necessary.
+	 */
+	strlcpy(blkcg->fc_app_id, app_id, app_id_len);
+	css_put(css);
+out_cgrp_put:
+	cgroup_put(cgrp);
+	return ret;
+}
+
+/**
+ * blkcg_get_fc_appid - get the fc app identifier associated with a bio
+ * @bio: target bio
+ *
+ * On success it returns the fc_app_id on failure it returns NULL
+ */
+static inline char *blkcg_get_fc_appid(struct bio *bio)
+{
+	if (bio && bio->bi_blkg &&
+		(bio->bi_blkg->blkcg->fc_app_id[0] != '\0'))
+		return bio->bi_blkg->blkcg->fc_app_id;
+	return NULL;
+}
+#else
+static inline int blkcg_set_fc_appid(char *buf, u64 id, size_t len) { return -EINVAL; }
+static inline char *blkcg_get_fc_appid(struct bio *bio) { return NULL; }
+#endif /*CONFIG_BLK_CGROUP_FC_APPID*/
 #endif	/* _BLK_CGROUP_H */
-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:44 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:44 +0530
Subject: [PATCH v11 01/13] cgroup: Added cgroup_get_from_id
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-2-muneendra.kumar@broadcom.com>

From: Muneendra <muneendra.kumar at broadcom.com>

Added a new function cgroup_get_from_id  to retrieve the cgroup
associated with cgroup id.
Exported the same as this can be used by blk-cgorup.c

Added function declaration of cgroup_get_from_id in cgorup.h

This patch also exported the function cgroup_get_e_css
as this is getting used in blk-cgroup.h

Acked-by: Tejun Heo <tj at kernel.org>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Muneendra <muneendra.kumar at broadcom.com>

---
v11:
No change

v10:
No change

v9:
Addressed the issues reported by kernel test robot

v8:
No change

v7:
No change

v6:
No change

v5:
renamed the function cgroup_get_from_kernfs_id to
cgroup_get_from_id

v4:
No change

v3:
Exported the cgroup_get_e_css

v2:
New patch
---
 include/linux/cgroup.h |  6 ++++++
 kernel/cgroup/cgroup.c | 26 ++++++++++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index 4f2f79de083e..d2eace88d9d1 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -696,6 +696,7 @@ static inline void cgroup_kthread_ready(void)
 }
 
 void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen);
+struct cgroup *cgroup_get_from_id(u64 id);
 #else /* !CONFIG_CGROUPS */
 
 struct cgroup_subsys_state;
@@ -743,6 +744,11 @@ static inline bool task_under_cgroup_hierarchy(struct task_struct *task,
 
 static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen)
 {}
+
+static inline struct cgroup *cgroup_get_from_id(u64 id)
+{
+	return NULL;
+}
 #endif /* !CONFIG_CGROUPS */
 
 #ifdef CONFIG_CGROUPS
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index e049edd66776..bd3251d4fcc7 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -577,6 +577,7 @@ struct cgroup_subsys_state *cgroup_get_e_css(struct cgroup *cgrp,
 	rcu_read_unlock();
 	return css;
 }
+EXPORT_SYMBOL_GPL(cgroup_get_e_css);
 
 static void cgroup_get_live(struct cgroup *cgrp)
 {
@@ -5776,6 +5777,31 @@ void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen)
 	kernfs_put(kn);
 }
 
+/*
+ * cgroup_get_from_id : get the cgroup associated with cgroup id
+ * @id: cgroup id
+ * On success it returns the cgrp on failure it returns NULL
+ */
+struct cgroup *cgroup_get_from_id(u64 id)
+{
+	struct kernfs_node *kn;
+	struct cgroup *cgrp = NULL;
+
+	mutex_lock(&cgroup_mutex);
+	kn = kernfs_find_and_get_node_by_id(cgrp_dfl_root.kf_root, id);
+	if (!kn)
+		goto out_unlock;
+
+	cgrp = kn->priv;
+	if (cgroup_is_dead(cgrp) || !cgroup_tryget(cgrp))
+		cgrp = NULL;
+	kernfs_put(kn);
+out_unlock:
+	mutex_unlock(&cgroup_mutex);
+	return cgrp;
+}
+EXPORT_SYMBOL_GPL(cgroup_get_from_id);
+
 /*
  * proc_cgroup_show()
  *  - Print task's cgroup paths into seq_file, one line for each hierarchy
-- 
2.26.2



From rafael at kernel.org  Tue Jun  8 04:27:58 2021
From: rafael at kernel.org (Rafael J. Wysocki)
Date: Tue, 8 Jun 2021 13:27:58 +0200
Subject: [PATCH v6 2/2] ACPI: Add quirks for AMD Renoir/Lucienne CPUs to
 force the D3 hint
In-Reply-To: <20210607173156.5548-2-mario.limonciello@amd.com>
References: <20210607173156.5548-1-mario.limonciello@amd.com>
 <20210607173156.5548-2-mario.limonciello@amd.com>
Message-ID: <CAJZ5v0jFop-M6dT18+eDDgUepFY=CxjHWjfYf6nxrk5B64=vwQ@mail.gmail.com>

On Mon, Jun 7, 2021 at 7:32 PM Mario Limonciello
<mario.limonciello at amd.com> wrote:
>
> AMD systems from Renoir and Lucienne require that the NVME controller
> is put into D3 over a Modern Standby / suspend-to-idle
> cycle.  This is "typically" accomplished using the `StorageD3Enable`
> property in the _DSD, but this property was introduced after many
> of these systems launched and most OEM systems don't have it in
> their BIOS.
>
> On AMD Renoir without these drives going into D3 over suspend-to-idle
> the resume will fail with the NVME controller being reset and a trace
> like this in the kernel logs:
> ```
> [   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
> [   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
> [   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
> [   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
> [   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
> [   95.332843] nvme nvme0: Abort status: 0x371
> [   95.332852] nvme nvme0: Abort status: 0x371
> [   95.332856] nvme nvme0: Abort status: 0x371
> [   95.332859] nvme nvme0: Abort status: 0x371
> [   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
> [   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
> ```
>
> The Microsoft documentation for StorageD3Enable mentioned that Windows has
> a hardcoded allowlist for D3 support, which was used for these platforms.
> Introduce quirks to hardcode them for Linux as well.
>
> As this property is now "standardized", OEM systems using AMD Cezanne and
> newer APU's have adopted this property, and quirks like this should not be
> necessary.
>
> CC: Julian Sikorski <belegdol at gmail.com>
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> ---
>  drivers/acpi/device_pm.c |  3 +++
>  drivers/acpi/x86/utils.c | 27 +++++++++++++++++++++++++++
>  include/acpi/acpi_bus.h  |  5 +++++
>  3 files changed, 35 insertions(+)
>
> Changes from v4->v5:
>  * Add this patch back in as it's been made apparent that the
>    system needs to be hardcoded for these.
>    Changes:
>    - Drop Cezanne - it's now covered by StorageD3Enable
>    - Rebase ontop of acpi_storage_d3 outside of NVME
> Changes from v5->v6:
>  * Move the quirk check into drivers/acpi/x86/ as suggested by
>    Rafael.
>
> diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
> index 1edb68d00b8e..985c17384192 100644
> --- a/drivers/acpi/device_pm.c
> +++ b/drivers/acpi/device_pm.c
> @@ -1356,6 +1356,9 @@ bool acpi_storage_d3(struct device *dev)
>         struct acpi_device *adev = ACPI_COMPANION(dev);
>         u8 val;
>
> +       if (force_storage_d3())
> +               return true;
> +
>         if (!adev)
>                 return false;
>         if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> diff --git a/drivers/acpi/x86/utils.c b/drivers/acpi/x86/utils.c
> index bdc1ba00aee9..2b8d5b3c876f 100644
> --- a/drivers/acpi/x86/utils.c
> +++ b/drivers/acpi/x86/utils.c
> @@ -135,3 +135,30 @@ bool acpi_device_always_present(struct acpi_device *adev)
>
>         return ret;
>  }
> +
> +/*
> + * AMD systems from Renoir and Lucienne *require* that the NVME controller
> + * is put into D3 over a Modern Standby / suspend-to-idle cycle.
> + *
> + * This is "typically" accomplished using the `StorageD3Enable`
> + * property in the _DSD that is checked via the `acpi_storage_d3` function
> + * but this property was introduced after many of these systems launched
> + * and most OEM systems don't have it in their BIOS.
> + *
> + * The Microsoft documentation for StorageD3Enable mentioned that Windows has
> + * a hardcoded allowlist for D3 support, which was used for these platforms.
> + *
> + * This allows quirking on Linux in a similar fashion.
> + */
> +const struct x86_cpu_id storage_d3_cpu_ids[] = {
> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),  /* Renoir */
> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL), /* Lucienne */
> +       {}
> +};
> +
> +bool force_storage_d3(void)
> +{
> +       if (x86_match_cpu(storage_d3_cpu_ids))
> +               return true;
> +       return false;

Well, what about doing

  return x86_match_cpu(storage_d3_cpu_ids);

instead?

> +}
> diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
> index 3a82faac5767..9b0ddbae5617 100644
> --- a/include/acpi/acpi_bus.h
> +++ b/include/acpi/acpi_bus.h
> @@ -607,11 +607,16 @@ int acpi_disable_wakeup_device_power(struct acpi_device *dev);
>
>  #ifdef CONFIG_X86
>  bool acpi_device_always_present(struct acpi_device *adev);
> +bool force_storage_d3(void);

This doesn't need to go into acpi_bus.h, because it will only be used
in device_pm.c.

You may as well put it into drivers/acpi/internal.h.

>  #else
>  static inline bool acpi_device_always_present(struct acpi_device *adev)
>  {
>         return false;
>  }
> +static inline bool force_storage_d3(void)
> +{
> +       return false;
> +}
>  #endif
>
>  #ifdef CONFIG_PM
> --


From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:46 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:46 +0530
Subject: [PATCH v11 03/13] nvme: Added a newsysfs attribute appid_store
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-4-muneendra.kumar@broadcom.com>

From: Muneendra <muneendra.kumar at broadcom.com>

Added a new sysfs attribute appid_store under
/sys/class/fc/fc_udev_device/*

With this new interface the user can set the application identfier
in  the blkcg associted with cgroup id.

Once the application identifer has set with this interface it allows
identification of traffic sources at an individual cgroup based
Applications (ex:virtual machine (VM))level in both host and
fabric infrastructure(FC).

Below is the interface provided to set the app_id

echo "<cgroupid>:<appid>" >> /sys/class/fc/fc_udev_device/appid_store
echo "457E:100000109b521d27" >> /sys/class/fc/fc_udev_device/appid_store

Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Muneendra <muneendra.kumar at broadcom.com>

---
v11:
No change

v10:
No change

v9:
No change

v8:
No change

v7:
No change

v6:
No change

v5:
Replaced APPID_LEN with FC_APPID_LEN

v4:
No change

v3:
Replaced blkcg_set_app_identifier function with blkcg_set_fc_appid

v2:
New Patch
---
 drivers/nvme/host/fc.c | 73 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 72 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index d9ab9e7871d0..57baddfe208b 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -9,7 +9,7 @@
 #include <uapi/scsi/fc/fc_els.h>
 #include <linux/delay.h>
 #include <linux/overflow.h>
-
+#include <linux/blk-cgroup.h>
 #include "nvme.h"
 #include "fabrics.h"
 #include <linux/nvme-fc-driver.h>
@@ -3787,10 +3787,81 @@ static ssize_t nvme_fc_nvme_discovery_store(struct device *dev,
 
 	return count;
 }
+
+/*parse the Cgroup id from a buf and returns the length of cgrpid*/
+static int fc_parse_cgrpid(const char *buf, u64 *id)
+{
+	char cgrp_id[16+1];
+	int cgrpid_len, j;
+
+	memset(cgrp_id, 0x0, sizeof(cgrp_id));
+	for (cgrpid_len = 0, j = 0; cgrpid_len < 17; cgrpid_len++) {
+		if (buf[cgrpid_len] != ':')
+			cgrp_id[cgrpid_len] = buf[cgrpid_len];
+		else {
+			j = 1;
+			break;
+		}
+	}
+	if (!j)
+		return -EINVAL;
+	if (kstrtou64(cgrp_id, 16, id) < 0)
+		return -EINVAL;
+	return cgrpid_len;
+}
+
+/*
+ * fc_update_appid :parses and updates the appid in the blkcg associated with
+ * cgroupid.
+ * @buf: buf contains both cgrpid and appid info
+ * @count: size of the buffer
+ */
+static int fc_update_appid(const char *buf, size_t count)
+{
+	u64 cgrp_id;
+	int appid_len = 0;
+	int cgrpid_len = 0;
+	char app_id[FC_APPID_LEN];
+	int ret = 0;
+
+	if (buf[count-1] == '\n')
+		count--;
+
+	if ((count > (16+1+FC_APPID_LEN)) || (!strchr(buf, ':')))
+		return -EINVAL;
+
+	cgrpid_len = fc_parse_cgrpid(buf, &cgrp_id);
+	if (cgrpid_len < 0)
+		return -EINVAL;
+	/*appid len is count - cgrpid_len -1 (: + \n) */
+	appid_len = count - cgrpid_len - 1;
+	if (appid_len > FC_APPID_LEN)
+		return -EINVAL;
+
+	memset(app_id, 0x0, sizeof(app_id));
+	memcpy(app_id, &buf[cgrpid_len+1], appid_len);
+	ret = blkcg_set_fc_appid(app_id, cgrp_id, sizeof(app_id));
+	if (ret < 0)
+		return ret;
+	return count;
+}
+
+static ssize_t fc_appid_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	int ret  = 0;
+
+	ret = fc_update_appid(buf, count);
+	if (ret < 0)
+		return -EINVAL;
+	return count;
+}
 static DEVICE_ATTR(nvme_discovery, 0200, NULL, nvme_fc_nvme_discovery_store);
+static DEVICE_ATTR(appid_store, 0200, NULL, fc_appid_store);
 
 static struct attribute *nvme_fc_attrs[] = {
 	&dev_attr_nvme_discovery.attr,
+	&dev_attr_appid_store.attr,
 	NULL
 };
 
-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:47 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:47 +0530
Subject: [PATCH v11 04/13] lpfc: vmid: Add the datastructure for supporting
 VMID in lpfc
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-5-muneendra.kumar@broadcom.com>

From: Gaurav Srivastava <gaurav.srivastava at broadcom.com>

This patch adds the primary datastructures needed to implement
VMID in lpfc driver. It maintains the capability, current state,
hash table for the vmid/appid along with other information.
The implementation supports the two versions of vmid implementation
(app header and priority tagging)

Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Gaurav Srivastava <gaurav.srivastava at broadcom.com>
Signed-off-by: James Smart <jsmart2021 at gmail.com>

---
v11:
No change

v10:
Function name correction

v9:
Updated the data structures
Merged patch 5 and 6 of previous version v8 to this patch

v8:
modify structure member to uniform data type naming scheme

v7:
No change

v6:
No change

v5:
No Change

v4:
No change

v3:
No change

v2:
Ported the patch on top of 5.10/scsi-queue
Removed unused variable.
---
 drivers/scsi/lpfc/lpfc.h      | 122 +++++++++++++++++++++++++++++++++
 drivers/scsi/lpfc/lpfc_crtn.h |  11 +++
 drivers/scsi/lpfc/lpfc_disc.h |   1 +
 drivers/scsi/lpfc/lpfc_hw.h   | 124 ++++++++++++++++++++++++++++++++--
 drivers/scsi/lpfc/lpfc_sli.h  |   8 +++
 5 files changed, 262 insertions(+), 4 deletions(-)

diff --git a/drivers/scsi/lpfc/lpfc.h b/drivers/scsi/lpfc/lpfc.h
index eb4472951cc6..4eb3a09928ab 100644
--- a/drivers/scsi/lpfc/lpfc.h
+++ b/drivers/scsi/lpfc/lpfc.h
@@ -304,6 +304,64 @@ struct lpfc_stats {
 struct lpfc_hba;
 
 
+#define LPFC_VMID_TIMER   300	/* timer interval in seconds. */
+
+#define LPFC_MAX_VMID_SIZE      256
+#define LPFC_COMPRESS_VMID_SIZE 16
+
+union lpfc_vmid_io_tag {
+	u32 app_id;	/* App Id vmid */
+	u8 cs_ctl_vmid;	/* Priority tag vmid */
+};
+
+#define JIFFIES_PER_HR	(HZ * 60 * 60)
+
+struct lpfc_vmid {
+	u8 flag;
+#define LPFC_VMID_SLOT_FREE     0x0
+#define LPFC_VMID_SLOT_USED     0x1
+#define LPFC_VMID_REQ_REGISTER  0x2
+#define LPFC_VMID_REGISTERED    0x4
+#define LPFC_VMID_DE_REGISTER   0x8
+	char host_vmid[LPFC_MAX_VMID_SIZE];
+	union lpfc_vmid_io_tag un;
+	struct hlist_node hnode;
+	u64 io_rd_cnt;
+	u64 io_wr_cnt;
+	u8 vmid_len;
+	u8 delete_inactive; /* Delete if inactive flag 0 = no, 1 = yes */
+	u32 hash_index;
+	u64 __percpu *last_io_time;
+};
+
+#define lpfc_vmid_is_type_priority_tag(vport)\
+	(vport->vmid_priority_tagging ? 1 : 0)
+
+#define LPFC_VMID_HASH_SIZE     256
+#define LPFC_VMID_HASH_MASK     255
+#define LPFC_VMID_HASH_SHIFT    6
+
+struct lpfc_vmid_context {
+	struct lpfc_vmid *vmp;
+	struct lpfc_nodelist *nlp;
+	bool instantiated;
+};
+
+struct lpfc_vmid_priority_range {
+	u8 low;
+	u8 high;
+	u8 qos;
+};
+
+struct lpfc_vmid_priority_info {
+	u32 num_descriptors;
+	struct lpfc_vmid_priority_range *vmid_range;
+};
+
+#define QFPA_EVEN_ONLY 0x01
+#define QFPA_ODD_ONLY  0x02
+#define QFPA_EVEN_ODD  0x03
+
 enum discovery_state {
 	LPFC_VPORT_UNKNOWN     =  0,    /* vport state is unknown */
 	LPFC_VPORT_FAILED      =  1,    /* vport has failed */
@@ -443,6 +501,9 @@ struct lpfc_vport {
 #define WORKER_RAMP_DOWN_QUEUE         0x800	/* hba: Decrease Q depth */
 #define WORKER_RAMP_UP_QUEUE           0x1000	/* hba: Increase Q depth */
 #define WORKER_SERVICE_TXQ             0x2000	/* hba: IOCBs on the txq */
+#define WORKER_CHECK_INACTIVE_VMID     0x4000	/* hba: check inactive vmids */
+#define WORKER_CHECK_VMID_ISSUE_QFPA   0x8000	/* vport: Check if qfpa need */
+						/* to issue */
 
 	struct timer_list els_tmofunc;
 	struct timer_list delayed_disc_tmo;
@@ -453,6 +514,8 @@ struct lpfc_vport {
 #define FC_LOADING		0x1	/* HBA in process of loading drvr */
 #define FC_UNLOADING		0x2	/* HBA in process of unloading drvr */
 #define FC_ALLOW_FDMI		0x4	/* port is ready for FDMI requests */
+#define FC_ALLOW_VMID		0x8	/* Allow VMID IO's */
+#define FC_DEREGISTER_ALL_APP_ID	0x10	/* Deregister all vmid's */
 	/* Vport Config Parameters */
 	uint32_t cfg_scan_down;
 	uint32_t cfg_lun_queue_depth;
@@ -471,9 +534,36 @@ struct lpfc_vport {
 	uint32_t cfg_tgt_queue_depth;
 	uint32_t cfg_first_burst_size;
 	uint32_t dev_loss_tmo_changed;
+	/* VMID parameters */
+	u8 lpfc_vmid_host_uuid[LPFC_COMPRESS_VMID_SIZE];
+	u32 max_vmid;	/* maximum VMIDs allowed per port */
+	u32 cur_vmid_cnt;	/* Current VMID count */
+#define LPFC_MIN_VMID	4
+#define LPFC_MAX_VMID	255
+	u32 vmid_inactivity_timeout;	/* Time after which the VMID */
+						/* deregisters from switch */
+	u32 vmid_priority_tagging;
+#define LPFC_VMID_PRIO_TAG_DISABLE	0 /* Disable */
+#define LPFC_VMID_PRIO_TAG_SUP_TARGETS	1 /* Allow supported targets only */
+#define LPFC_VMID_PRIO_TAG_ALL_TARGETS	2 /* Allow all targets */
+	unsigned long *vmid_priority_range;
+#define LPFC_VMID_MAX_PRIORITY_RANGE    256
+#define LPFC_VMID_PRIORITY_BITMAP_SIZE  32
+	u8 vmid_flag;
+#define LPFC_VMID_IN_USE		0x1
+#define LPFC_VMID_ISSUE_QFPA		0x2
+#define LPFC_VMID_QFPA_CMPL		0x4
+#define LPFC_VMID_QOS_ENABLED		0x8
+#define LPFC_VMID_TIMER_ENBLD		0x10
+	struct fc_qfpa_res *qfpa_res;
 
 	struct fc_vport *fc_vport;
 
+	struct lpfc_vmid *vmid;
+	DECLARE_HASHTABLE(hash_table, 8);
+	rwlock_t vmid_lock;
+	struct lpfc_vmid_priority_info vmid_priority;
+
 #ifdef CONFIG_SCSI_LPFC_DEBUG_FS
 	struct dentry *debug_disc_trc;
 	struct dentry *debug_nodelist;
@@ -940,6 +1030,13 @@ struct lpfc_hba {
 	struct nvmet_fc_target_port *targetport;
 	lpfc_vpd_t vpd;		/* vital product data */
 
+	u32 cfg_max_vmid;	/* maximum VMIDs allowed per port */
+	u32 cfg_vmid_app_header;
+#define LPFC_VMID_APP_HEADER_DISABLE	0
+#define LPFC_VMID_APP_HEADER_ENABLE	1
+	u32 cfg_vmid_priority_tagging;
+	u32 cfg_vmid_inactivity_timeout;	/* Time after which the VMID */
+						/*  deregisters from switch */
 	struct pci_dev *pcidev;
 	struct list_head      work_list;
 	uint32_t              work_ha;      /* Host Attention Bits for WT */
@@ -1180,6 +1277,7 @@ struct lpfc_hba {
 	struct list_head ct_ev_waiters;
 	struct unsol_rcv_ct_ctx ct_ctx[LPFC_CT_CTX_MAX];
 	uint32_t ctx_idx;
+	struct timer_list inactive_vmid_poll;
 
 	/* RAS Support */
 	struct lpfc_ras_fwlog ras_fwlog;
@@ -1421,3 +1519,27 @@ static const char *routine(enum enum_name table_key)			\
 	}								\
 	return name;							\
 }
+
+/**
+ * lpfc_is_vmid_enabled - returns if VMID is enabled for either switch types
+ * @phba: Pointer to HBA context object.
+ *
+ * Relationship between the enable, target support and if vmid tag is required
+ * for the particular combination
+ * ---------------------------------------------------
+ * Switch    Enable Flag  Target Support  VMID Needed
+ * ---------------------------------------------------
+ * App Id     0              NA              N
+ * App Id     1               0              N
+ * App Id     1               1              Y
+ * Pr Tag     0              NA              N
+ * Pr Tag     1               0              N
+ * Pr Tag     1               1              Y
+ * Pr Tag     2               *              Y
+ ---------------------------------------------------
+ *
+ **/
+static inline int lpfc_is_vmid_enabled(struct lpfc_hba *phba)
+{
+	return phba->cfg_vmid_app_header || phba->cfg_vmid_priority_tagging;
+}
diff --git a/drivers/scsi/lpfc/lpfc_crtn.h b/drivers/scsi/lpfc/lpfc_crtn.h
index 8f56e8f66311..737483c3f01d 100644
--- a/drivers/scsi/lpfc/lpfc_crtn.h
+++ b/drivers/scsi/lpfc/lpfc_crtn.h
@@ -608,3 +608,14 @@ extern unsigned long lpfc_no_hba_reset[];
 extern union lpfc_wqe128 lpfc_iread_cmd_template;
 extern union lpfc_wqe128 lpfc_iwrite_cmd_template;
 extern union lpfc_wqe128 lpfc_icmnd_cmd_template;
+
+/* vmid interface */
+int lpfc_vmid_uvem(struct lpfc_vport *vport, struct lpfc_vmid *vmid, bool ins);
+uint32_t lpfc_vmid_get_cs_ctl(struct lpfc_vport *vport);
+int lpfc_vmid_cmd(struct lpfc_vport *vport,
+		  int cmdcode, struct lpfc_vmid *vmid);
+int lpfc_vmid_hash_fn(const char *vmid, int len);
+struct lpfc_vmid *lpfc_get_vmid_from_hashtable(struct lpfc_vport *vport,
+					      uint32_t hash, uint8_t *buf);
+void lpfc_vmid_vport_cleanup(struct lpfc_vport *vport);
+int lpfc_issue_els_qfpa(struct lpfc_vport *vport);
diff --git a/drivers/scsi/lpfc/lpfc_disc.h b/drivers/scsi/lpfc/lpfc_disc.h
index 27d1a971a7f8..131374a61d7e 100644
--- a/drivers/scsi/lpfc/lpfc_disc.h
+++ b/drivers/scsi/lpfc/lpfc_disc.h
@@ -125,6 +125,7 @@ struct lpfc_nodelist {
 	uint8_t         nlp_fcp_info;	        /* class info, bits 0-3 */
 #define NLP_FCP_2_DEVICE   0x10			/* FCP-2 device */
 	u8		nlp_nvme_info;	        /* NVME NSLER Support */
+	uint8_t		vmid_support;		/* destination VMID support */
 #define NLP_NVME_NSLER     0x1			/* NVME NSLER device */
 
 	struct timer_list   nlp_delayfunc;	/* Used for delayed ELS cmds */
diff --git a/drivers/scsi/lpfc/lpfc_hw.h b/drivers/scsi/lpfc/lpfc_hw.h
index 42682d95af52..4a5a85ed42ec 100644
--- a/drivers/scsi/lpfc/lpfc_hw.h
+++ b/drivers/scsi/lpfc/lpfc_hw.h
@@ -275,6 +275,7 @@ struct lpfc_sli_ct_request {
 #define  SLI_CT_ACCESS_DENIED             0x10
 #define  SLI_CT_INVALID_PORT_ID           0x11
 #define  SLI_CT_DATABASE_EMPTY            0x12
+#define  SLI_CT_APP_ID_NOT_AVAILABLE      0x40
 
 /*
  * Name Server Command Codes
@@ -400,16 +401,16 @@ struct csp {
 	uint16_t altBbCredit:1;	/* FC Word 1, bit 27 */
 	uint16_t edtovResolution:1;	/* FC Word 1, bit 26 */
 	uint16_t multicast:1;	/* FC Word 1, bit 25 */
-	uint16_t broadcast:1;	/* FC Word 1, bit 24 */
+	uint16_t app_hdr_support:1;	/* FC Word 1, bit 24 */
 
-	uint16_t huntgroup:1;	/* FC Word 1, bit 23 */
+	uint16_t priority_tagging:1;	/* FC Word 1, bit 23 */
 	uint16_t simplex:1;	/* FC Word 1, bit 22 */
 	uint16_t word1Reserved1:3;	/* FC Word 1, bit 21:19 */
 	uint16_t dhd:1;		/* FC Word 1, bit 18 */
 	uint16_t contIncSeqCnt:1;	/* FC Word 1, bit 17 */
 	uint16_t payloadlength:1;	/* FC Word 1, bit 16 */
 #else	/*  __LITTLE_ENDIAN_BITFIELD */
-	uint16_t broadcast:1;	/* FC Word 1, bit 24 */
+	uint16_t app_hdr_support:1;	/* FC Word 1, bit 24 */
 	uint16_t multicast:1;	/* FC Word 1, bit 25 */
 	uint16_t edtovResolution:1;	/* FC Word 1, bit 26 */
 	uint16_t altBbCredit:1;	/* FC Word 1, bit 27 */
@@ -423,7 +424,7 @@ struct csp {
 	uint16_t dhd:1;		/* FC Word 1, bit 18 */
 	uint16_t word1Reserved1:3;	/* FC Word 1, bit 21:19 */
 	uint16_t simplex:1;	/* FC Word 1, bit 22 */
-	uint16_t huntgroup:1;	/* FC Word 1, bit 23 */
+	uint16_t priority_tagging:1;	/* FC Word 1, bit 23 */
 #endif
 
 	uint8_t bbRcvSizeMsb;	/* Upper nibble is reserved */
@@ -607,6 +608,8 @@ struct fc_vft_header {
 #define ELS_CMD_LIRR      0x7A000000
 #define ELS_CMD_LCB	  0x81000000
 #define ELS_CMD_FPIN	  0x16000000
+#define ELS_CMD_QFPA      0xB0000000
+#define ELS_CMD_UVEM      0xB1000000
 #else	/*  __LITTLE_ENDIAN_BITFIELD */
 #define ELS_CMD_MASK      0xffff
 #define ELS_RSP_MASK      0xff
@@ -649,6 +652,8 @@ struct fc_vft_header {
 #define ELS_CMD_LIRR      0x7A
 #define ELS_CMD_LCB	  0x81
 #define ELS_CMD_FPIN	  ELS_FPIN
+#define ELS_CMD_QFPA      0xB0
+#define ELS_CMD_UVEM      0xB1
 #endif
 
 /*
@@ -1317,6 +1322,117 @@ struct fc_rdp_res_frame {
 };
 
 
+/* UVEM */
+
+#define LPFC_UVEM_SIZE 60
+#define LPFC_UVEM_VEM_ID_DESC_SIZE 16
+#define LPFC_UVEM_VE_MAP_DESC_SIZE 20
+
+#define VEM_ID_DESC_TAG  0x0001000A
+struct lpfc_vem_id_desc {
+	uint32_t tag;
+	uint32_t length;
+	uint8_t vem_id[16];
+};
+
+#define LPFC_QFPA_SIZE	4
+
+#define INSTANTIATED_VE_DESC_TAG  0x0001000B
+struct instantiated_ve_desc {
+	uint32_t tag;
+	uint32_t length;
+	uint8_t global_vem_id[16];
+	uint32_t word6;
+#define lpfc_instantiated_local_id_SHIFT   0
+#define lpfc_instantiated_local_id_MASK    0x000000ff
+#define lpfc_instantiated_local_id_WORD    word6
+#define lpfc_instantiated_nport_id_SHIFT   8
+#define lpfc_instantiated_nport_id_MASK    0x00ffffff
+#define lpfc_instantiated_nport_id_WORD    word6
+};
+
+#define DEINSTANTIATED_VE_DESC_TAG  0x0001000C
+struct deinstantiated_ve_desc {
+	uint32_t tag;
+	uint32_t length;
+	uint8_t global_vem_id[16];
+	uint32_t word6;
+#define lpfc_deinstantiated_nport_id_SHIFT   0
+#define lpfc_deinstantiated_nport_id_MASK    0x000000ff
+#define lpfc_deinstantiated_nport_id_WORD    word6
+#define lpfc_deinstantiated_local_id_SHIFT   24
+#define lpfc_deinstantiated_local_id_MASK    0x00ffffff
+#define lpfc_deinstantiated_local_id_WORD    word6
+};
+
+/* Query Fabric Priority Allocation Response */
+#define LPFC_PRIORITY_RANGE_DESC_SIZE 12
+
+struct priority_range_desc {
+	uint32_t tag;
+	uint32_t length;
+	uint8_t lo_range;
+	uint8_t hi_range;
+	uint8_t qos_priority;
+	uint8_t local_ve_id;
+};
+
+struct fc_qfpa_res {
+	uint32_t reply_sequence;	/* LS_ACC or LS_RJT */
+	uint32_t length;	/* FC Word 1    */
+	struct priority_range_desc desc[1];
+};
+
+/* Application Server command code */
+/* VMID               */
+
+#define SLI_CT_APP_SEV_Subtypes     0x20	/* Application Server subtype */
+
+#define SLI_CTAS_GAPPIA_ENT    0x0100	/* Get Application Identifier */
+#define SLI_CTAS_GALLAPPIA     0x0101	/* Get All Application Identifier */
+#define SLI_CTAS_GALLAPPIA_ID  0x0102	/* Get All Application Identifier */
+					/* for Nport */
+#define SLI_CTAS_GAPPIA_IDAPP  0x0103	/* Get Application Identifier */
+					/* for Nport */
+#define SLI_CTAS_RAPP_IDENT    0x0200	/* Register Application Identifier */
+#define SLI_CTAS_DAPP_IDENT    0x0300	/* Deregister Application */
+					/* Identifier */
+#define SLI_CTAS_DALLAPP_ID    0x0301	/* Deregister All Application */
+					/* Identifier */
+
+struct entity_id_object {
+	uint8_t entity_id_len;
+	uint8_t entity_id[255];	/* VM UUID */
+};
+
+struct app_id_object {
+	uint32_t port_id;
+	uint32_t app_id;
+	struct entity_id_object obj;
+};
+
+struct lpfc_vmid_rapp_ident_list {
+	uint32_t no_of_objects;
+	struct entity_id_object obj[1];
+};
+
+struct lpfc_vmid_dapp_ident_list {
+	uint32_t no_of_objects;
+	struct entity_id_object obj[1];
+};
+
+#define GALLAPPIA_ID_LAST  0x80
+struct lpfc_vmid_gallapp_ident_list {
+	uint8_t control;
+	uint8_t reserved[3];
+	struct app_id_object app_id;
+};
+
+#define RAPP_IDENT_OFFSET  (offsetof(struct lpfc_sli_ct_request, un) + 4)
+#define DAPP_IDENT_OFFSET  (offsetof(struct lpfc_sli_ct_request, un) + 4)
+#define GALLAPPIA_ID_SIZE  (offsetof(struct lpfc_sli_ct_request, un) + 4)
+#define DALLAPP_ID_SIZE    (offsetof(struct lpfc_sli_ct_request, un) + 4)
+
 /******** FDMI ********/
 
 /* lpfc_sli_ct_request defines the CT_IU preamble for FDMI commands */
diff --git a/drivers/scsi/lpfc/lpfc_sli.h b/drivers/scsi/lpfc/lpfc_sli.h
index e40da1a7ff77..dde8eb9d796d 100644
--- a/drivers/scsi/lpfc/lpfc_sli.h
+++ b/drivers/scsi/lpfc/lpfc_sli.h
@@ -35,6 +35,12 @@ typedef enum _lpfc_ctx_cmd {
 	LPFC_CTX_HOST
 } lpfc_ctx_cmd;
 
+union lpfc_vmid_iocb_tag {
+	uint32_t app_id;
+	uint8_t cs_ctl_vmid;
+	struct lpfc_vmid_context *vmid_context;	/* UVEM context information */
+};
+
 struct lpfc_cq_event {
 	struct list_head list;
 	uint16_t hdwq;
@@ -100,6 +106,7 @@ struct lpfc_iocbq {
 #define LPFC_IO_NVME	        0x200000 /* NVME FCP command */
 #define LPFC_IO_NVME_LS		0x400000 /* NVME LS command */
 #define LPFC_IO_NVMET		0x800000 /* NVMET command */
+#define LPFC_IO_VMID            0x1000000 /* VMID tagged IO */
 
 	uint32_t drvrTimeout;	/* driver timeout in seconds */
 	struct lpfc_vport *vport;/* virtual port pointer */
@@ -115,6 +122,7 @@ struct lpfc_iocbq {
 		struct lpfc_node_rrq *rrq;
 	} context_un;
 
+	union lpfc_vmid_iocb_tag vmid_tag;
 	void (*fabric_iocb_cmpl)(struct lpfc_hba *, struct lpfc_iocbq *,
 			   struct lpfc_iocbq *);
 	void (*wait_iocb_cmpl)(struct lpfc_hba *, struct lpfc_iocbq *,
-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:48 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:48 +0530
Subject: [PATCH v11 05/13] lpfc: vmid: VMID params initialization
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-6-muneendra.kumar@broadcom.com>

From: Gaurav Srivastava <gaurav.srivastava at broadcom.com>

This patch initializes the VMID parameters like the type of vmid, max
number of vmids supported and timeout value for the vmid registration
based on the user input.

Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Gaurav Srivastava <gaurav.srivastava at broadcom.com>
Signed-off-by: James Smart <jsmart2021 at gmail.com>

---
v11:
No change

v10:
No change

v9:
updated comments

v8:
No change

v7:
No change

v6:
No change

v5:
No change

v4:
No change

v3:
No change

v2:
Ported the patch on top of 5.10/scsi-queue
---
 drivers/scsi/lpfc/lpfc_attr.c | 48 +++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/drivers/scsi/lpfc/lpfc_attr.c b/drivers/scsi/lpfc/lpfc_attr.c
index c5e96cb904c8..eb88aaaf36eb 100644
--- a/drivers/scsi/lpfc/lpfc_attr.c
+++ b/drivers/scsi/lpfc/lpfc_attr.c
@@ -6162,6 +6162,45 @@ LPFC_ATTR_RW(enable_dpp, 1, 0, 1, "Enable Direct Packet Push");
  */
 LPFC_ATTR_R(enable_mi, 1, 0, 1, "Enable MI");
 
+/*
+ * lpfc_max_vmid: Maximum number of VMs to be tagged. This is valid only if
+ * either vmid_app_header or vmid_priority_tagging is enabled.
+ *       4 - 255  = vmid support enabled for 4-255 VMs
+ *       Value range is [4,255].
+ */
+LPFC_ATTR_RW(max_vmid, LPFC_MIN_VMID, LPFC_MIN_VMID, LPFC_MAX_VMID,
+	     "Maximum number of VMs supported");
+
+/*
+ * lpfc_vmid_inactivity_timeout: Inactivity timeout duration in hours
+ *       0  = Timeout is disabled
+ * Value range is [0,24].
+ */
+LPFC_ATTR_RW(vmid_inactivity_timeout, 4, 0, 24,
+	     "Inactivity timeout in hours");
+
+/*
+ * lpfc_vmid_app_header: Enable App Header VMID support
+ *       0  = Support is disabled (default)
+ *       1  = Support is enabled
+ * Value range is [0,1].
+ */
+LPFC_ATTR_RW(vmid_app_header, LPFC_VMID_APP_HEADER_DISABLE,
+	     LPFC_VMID_APP_HEADER_DISABLE, LPFC_VMID_APP_HEADER_ENABLE,
+	     "Enable App Header VMID support");
+
+/*
+ * lpfc_vmid_priority_tagging: Enable Priority Tagging VMID support
+ *       0  = Support is disabled (default)
+ *       1  = Allow supported targets only
+ *       2  = Allow all targets
+ * Value range is [0,2].
+ */
+LPFC_ATTR_RW(vmid_priority_tagging, LPFC_VMID_PRIO_TAG_DISABLE,
+	     LPFC_VMID_PRIO_TAG_DISABLE,
+	     LPFC_VMID_PRIO_TAG_ALL_TARGETS,
+	     "Enable Priority Tagging VMID support");
+
 struct device_attribute *lpfc_hba_attrs[] = {
 	&dev_attr_nvme_info,
 	&dev_attr_scsi_stat,
@@ -6281,6 +6320,10 @@ struct device_attribute *lpfc_hba_attrs[] = {
 	&dev_attr_lpfc_enable_bbcr,
 	&dev_attr_lpfc_enable_dpp,
 	&dev_attr_lpfc_enable_mi,
+	&dev_attr_lpfc_max_vmid,
+	&dev_attr_lpfc_vmid_inactivity_timeout,
+	&dev_attr_lpfc_vmid_app_header,
+	&dev_attr_lpfc_vmid_priority_tagging,
 	NULL,
 };
 
@@ -7357,6 +7400,11 @@ lpfc_get_cfgparam(struct lpfc_hba *phba)
 	lpfc_enable_hba_heartbeat_init(phba, lpfc_enable_hba_heartbeat);
 
 	lpfc_EnableXLane_init(phba, lpfc_EnableXLane);
+	/* VMID Inits */
+	lpfc_max_vmid_init(phba, lpfc_max_vmid);
+	lpfc_vmid_inactivity_timeout_init(phba, lpfc_vmid_inactivity_timeout);
+	lpfc_vmid_app_header_init(phba, lpfc_vmid_app_header);
+	lpfc_vmid_priority_tagging_init(phba, lpfc_vmid_priority_tagging);
 	if (phba->sli_rev != LPFC_SLI_REV4)
 		phba->cfg_EnableXLane = 0;
 	lpfc_XLanePriority_init(phba, lpfc_XLanePriority);
-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:49 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:49 +0530
Subject: [PATCH v11 06/13] lpfc: vmid: Add support for vmid in mailbox command,
 does vmid resource allocation and vmid cleanup
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-7-muneendra.kumar@broadcom.com>

From: Gaurav Srivastava <gaurav.srivastava at broadcom.com>

This patch does the following -
1.adds supporting datastructures for mailbox command which helps in
determining if the firmware supports appid or not.
2.This patch allocates the resource for vmid and checks if the firmware
supports the feature or not.
3.The patch cleans up the vmid resources and stops the timer.

Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Gaurav Srivastava <gaurav.srivastava at broadcom.com>
Signed-off-by: James Smart <jsmart2021 at gmail.com>

---
v11:
No change

v10:
No change

v9:
Modified code for use of hashtable

v8:
Added new function declaration, return error code and
memory allocation API changes

v7:
No change

v6:
Added Forward declarations and functions to static

v5:
Merged patches 8 and 11 of v4 to this patch
Changed Return code to non-numeric/Symbol

v4:
No change

v3:
No change

v2:
Ported the patch on top of 5.10/scsi-queue
---
 drivers/scsi/lpfc/lpfc_hw4.h  | 12 +++++++
 drivers/scsi/lpfc/lpfc_init.c | 64 +++++++++++++++++++++++++++++++++++
 drivers/scsi/lpfc/lpfc_mbox.c |  6 ++++
 drivers/scsi/lpfc/lpfc_scsi.c | 29 ++++++++++++++++
 drivers/scsi/lpfc/lpfc_sli.c  |  9 +++++
 5 files changed, 120 insertions(+)

diff --git a/drivers/scsi/lpfc/lpfc_hw4.h b/drivers/scsi/lpfc/lpfc_hw4.h
index f77e71e6dbbd..eb8c735a243b 100644
--- a/drivers/scsi/lpfc/lpfc_hw4.h
+++ b/drivers/scsi/lpfc/lpfc_hw4.h
@@ -273,6 +273,9 @@ struct lpfc_sli4_flags {
 #define lpfc_vfi_rsrc_rdy_MASK		0x00000001
 #define lpfc_vfi_rsrc_rdy_WORD		word0
 #define LPFC_VFI_RSRC_RDY		1
+#define lpfc_ftr_ashdr_SHIFT            4
+#define lpfc_ftr_ashdr_MASK             0x00000001
+#define lpfc_ftr_ashdr_WORD             word0
 };
 
 struct sli4_bls_rsp {
@@ -2944,6 +2947,9 @@ struct lpfc_mbx_request_features {
 #define lpfc_mbx_rq_ftr_rq_mrqp_SHIFT		16
 #define lpfc_mbx_rq_ftr_rq_mrqp_MASK		0x00000001
 #define lpfc_mbx_rq_ftr_rq_mrqp_WORD		word2
+#define lpfc_mbx_rq_ftr_rq_ashdr_SHIFT          17
+#define lpfc_mbx_rq_ftr_rq_ashdr_MASK           0x00000001
+#define lpfc_mbx_rq_ftr_rq_ashdr_WORD           word2
 	uint32_t word3;
 #define lpfc_mbx_rq_ftr_rsp_iaab_SHIFT		0
 #define lpfc_mbx_rq_ftr_rsp_iaab_MASK		0x00000001
@@ -2975,6 +2981,9 @@ struct lpfc_mbx_request_features {
 #define lpfc_mbx_rq_ftr_rsp_mrqp_SHIFT		16
 #define lpfc_mbx_rq_ftr_rsp_mrqp_MASK		0x00000001
 #define lpfc_mbx_rq_ftr_rsp_mrqp_WORD		word3
+#define lpfc_mbx_rq_ftr_rsp_ashdr_SHIFT         17
+#define lpfc_mbx_rq_ftr_rsp_ashdr_MASK          0x00000001
+#define lpfc_mbx_rq_ftr_rsp_ashdr_WORD          word3
 };
 
 struct lpfc_mbx_memory_dump_type3 {
@@ -4219,6 +4228,9 @@ struct wqe_common {
 #define wqe_xchg_WORD         word10
 #define LPFC_SCSI_XCHG	      0x0
 #define LPFC_NVME_XCHG	      0x1
+#define wqe_appid_SHIFT       5
+#define wqe_appid_MASK        0x00000001
+#define wqe_appid_WORD        word10
 #define wqe_oas_SHIFT         6
 #define wqe_oas_MASK          0x00000001
 #define wqe_oas_WORD          word10
diff --git a/drivers/scsi/lpfc/lpfc_init.c b/drivers/scsi/lpfc/lpfc_init.c
index f81dfa3cb0a1..d6434e4ffa20 100644
--- a/drivers/scsi/lpfc/lpfc_init.c
+++ b/drivers/scsi/lpfc/lpfc_init.c
@@ -98,6 +98,7 @@ static struct scsi_transport_template *lpfc_transport_template = NULL;
 static struct scsi_transport_template *lpfc_vport_transport_template = NULL;
 static DEFINE_IDR(lpfc_hba_index);
 #define LPFC_NVMET_BUF_POST 254
+static int lpfc_vmid_res_alloc(struct lpfc_hba *phba, struct lpfc_vport *vport);
 
 /**
  * lpfc_config_port_prep - Perform lpfc initialization prior to config port
@@ -2888,6 +2889,10 @@ lpfc_cleanup(struct lpfc_vport *vport)
 	if (phba->link_state > LPFC_LINK_DOWN)
 		lpfc_port_link_failure(vport);
 
+	/* cleanup vmid resources */
+	if (lpfc_is_vmid_enabled(phba))
+		lpfc_vmid_vport_cleanup(vport);
+
 	list_for_each_entry_safe(ndlp, next_ndlp, &vport->fc_nodes, nlp_listp) {
 		if (vport->port_type != LPFC_PHYSICAL_PORT &&
 		    ndlp->nlp_DID == Fabric_DID) {
@@ -4307,6 +4312,57 @@ lpfc_get_wwpn(struct lpfc_hba *phba)
 		return rol64(wwn, 32);
 }
 
+/**
+ * lpfc_vmid_res_alloc - Allocates resources for VMID
+ * @phba: pointer to lpfc hba data structure.
+ * @vport: pointer to vport data structure
+ *
+ * This routine allocated the resources needed for the vmid.
+ *
+ * Return codes
+ *	0 on Succeess
+ *	Non-0 on Failure
+ */
+static int
+lpfc_vmid_res_alloc(struct lpfc_hba *phba, struct lpfc_vport *vport)
+{
+	/* vmid feature is supported only on SLI4 */
+	if (phba->sli_rev == LPFC_SLI_REV3) {
+		phba->cfg_vmid_app_header = 0;
+		phba->cfg_vmid_priority_tagging = 0;
+	}
+
+	/* if enabled, then allocated the resources */
+	if (lpfc_is_vmid_enabled(phba)) {
+		vport->vmid =
+		    kcalloc(phba->cfg_max_vmid, sizeof(struct lpfc_vmid),
+			    GFP_KERNEL);
+		if (!vport->vmid)
+			return -ENOMEM;
+
+		rwlock_init(&vport->vmid_lock);
+
+		/* setting the VMID parameters for the vport */
+		vport->vmid_priority_tagging = phba->cfg_vmid_priority_tagging;
+		vport->vmid_inactivity_timeout =
+		    phba->cfg_vmid_inactivity_timeout;
+		vport->max_vmid = phba->cfg_max_vmid;
+		vport->cur_vmid_cnt = 0;
+
+		vport->vmid_priority_range = bitmap_zalloc
+			(LPFC_VMID_MAX_PRIORITY_RANGE, GFP_KERNEL);
+
+		if (!vport->vmid_priority_range) {
+			kfree(vport->vmid);
+			return -ENOMEM;
+		}
+
+		/* Initialize the hashtable */
+		hash_init(vport->hash_table);
+	}
+	return 0;
+}
+
 /**
  * lpfc_create_port - Create an FC port
  * @phba: pointer to lpfc hba data structure.
@@ -4459,6 +4515,12 @@ lpfc_create_port(struct lpfc_hba *phba, int instance, struct device *dev)
 			vport->port_type, shost->sg_tablesize,
 			phba->cfg_scsi_seg_cnt, phba->cfg_sg_seg_cnt);
 
+	/* allocate the resources for vmid */
+	rc = lpfc_vmid_res_alloc(phba, vport);
+
+	if (rc)
+		goto out;
+
 	/* Initialize all internally managed lists. */
 	INIT_LIST_HEAD(&vport->fc_nodes);
 	INIT_LIST_HEAD(&vport->rcv_buffer_list);
@@ -4483,6 +4545,8 @@ lpfc_create_port(struct lpfc_hba *phba, int instance, struct device *dev)
 	return vport;
 
 out_put_shost:
+	kfree(vport->vmid);
+	bitmap_free(vport->vmid_priority_range);
 	scsi_host_put(shost);
 out:
 	return NULL;
diff --git a/drivers/scsi/lpfc/lpfc_mbox.c b/drivers/scsi/lpfc/lpfc_mbox.c
index eab991739e37..698243ca8f09 100644
--- a/drivers/scsi/lpfc/lpfc_mbox.c
+++ b/drivers/scsi/lpfc/lpfc_mbox.c
@@ -2101,6 +2101,12 @@ lpfc_request_features(struct lpfc_hba *phba, struct lpfcMboxq *mboxq)
 		bf_set(lpfc_mbx_rq_ftr_rq_iaab, &mboxq->u.mqe.un.req_ftrs, 0);
 		bf_set(lpfc_mbx_rq_ftr_rq_iaar, &mboxq->u.mqe.un.req_ftrs, 0);
 	}
+
+	/* Enable Application Services Header for apphedr VMID */
+	if (phba->cfg_vmid_app_header) {
+		bf_set(lpfc_mbx_rq_ftr_rq_ashdr, &mboxq->u.mqe.un.req_ftrs, 1);
+		bf_set(lpfc_ftr_ashdr, &phba->sli4_hba.sli4_flags, 1);
+	}
 	return;
 }
 
diff --git a/drivers/scsi/lpfc/lpfc_scsi.c b/drivers/scsi/lpfc/lpfc_scsi.c
index b8bb012abb33..37188a6c5c27 100644
--- a/drivers/scsi/lpfc/lpfc_scsi.c
+++ b/drivers/scsi/lpfc/lpfc_scsi.c
@@ -5417,6 +5417,35 @@ lpfc_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *cmnd)
 	return 0;
 }
 
+/*
+ * lpfc_vmid_vport_cleanup - cleans up the resources associated with a vports
+ * @vport: The virtual port for which this call is being executed.
+ */
+void lpfc_vmid_vport_cleanup(struct lpfc_vport *vport)
+{
+	u32 bucket;
+	struct lpfc_vmid *cur;
+
+	/* delete the timer */
+	if (vport->port_type == LPFC_PHYSICAL_PORT)
+		del_timer_sync(&vport->phba->inactive_vmid_poll);
+
+	/* free the resources */
+	kfree(vport->qfpa_res);
+	kfree(vport->vmid_priority.vmid_range);
+	kfree(vport->vmid);
+
+	/* for all elements in the hash table */
+	if (!hash_empty(vport->hash_table))
+		hash_for_each(vport->hash_table, bucket, cur, hnode)
+			hash_del(&cur->hnode);
+
+	/* reset variables */
+	vport->qfpa_res = NULL;
+	vport->vmid_priority.vmid_range = NULL;
+	vport->vmid = NULL;
+	vport->cur_vmid_cnt = 0;
+}
 
 /**
  * lpfc_abort_handler - scsi_host_template eh_abort_handler entry point
diff --git a/drivers/scsi/lpfc/lpfc_sli.c b/drivers/scsi/lpfc/lpfc_sli.c
index 1ad1beb2a8a8..c7d0acd6c8ef 100644
--- a/drivers/scsi/lpfc/lpfc_sli.c
+++ b/drivers/scsi/lpfc/lpfc_sli.c
@@ -7699,6 +7699,15 @@ lpfc_sli4_hba_setup(struct lpfc_hba *phba)
 		goto out_free_mbox;
 	}
 
+	/* Disable vmid if app header is not supported */
+	if (phba->cfg_vmid_app_header && !(bf_get(lpfc_mbx_rq_ftr_rsp_ashdr,
+						  &mqe->un.req_ftrs))) {
+		bf_set(lpfc_ftr_ashdr, &phba->sli4_hba.sli4_flags, 0);
+		phba->cfg_vmid_app_header = 0;
+		lpfc_printf_log(phba, KERN_DEBUG, LOG_SLI,
+				"1242 vmid feature not supported\n");
+	}
+
 	/*
 	 * The port must support FCP initiator mode as this is the
 	 * only mode running in the host.
-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:51 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:51 +0530
Subject: [PATCH v11 08/13] lpfc: vmid: Functions to manage vmids
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-9-muneendra.kumar@broadcom.com>

From: Gaurav Srivastava <gaurav.srivastava at broadcom.com>

This patch contains the routines to save, retrieve and remove the vmids
from the data structure. A hash table is used to save the vmids and
the corresponding UUIDs associated with the application/VMs.

Signed-off-by: Gaurav Srivastava  <gaurav.srivastava at broadcom.com>
Signed-off-by: James Smart <jsmart2021 at gmail.com>

---
v11:
No change

v10:
Fixed the Function name correction

v9:
Added changes for implementing kernel based hashtable

v8:
Added correct return value and error codes

v7:
No change

v6:
Added Forward declarations and functions to static

v5:
Changed Return code to non-numeric/Symbol

v4:
No change

v3:
No change

v2:
Ported the patch on top of 5.10/scsi-queue
---
 drivers/scsi/lpfc/lpfc_scsi.c | 118 ++++++++++++++++++++++++++++++++++
 1 file changed, 118 insertions(+)

diff --git a/drivers/scsi/lpfc/lpfc_scsi.c b/drivers/scsi/lpfc/lpfc_scsi.c
index 37188a6c5c27..65637d85f9bb 100644
--- a/drivers/scsi/lpfc/lpfc_scsi.c
+++ b/drivers/scsi/lpfc/lpfc_scsi.c
@@ -28,6 +28,7 @@
 #include <asm/unaligned.h>
 #include <linux/t10-pi.h>
 #include <linux/crc-t10dif.h>
+#include <linux/blk-cgroup.h>
 #include <net/checksum.h>
 
 #include <scsi/scsi.h>
@@ -86,6 +87,14 @@ static void
 lpfc_release_scsi_buf_s3(struct lpfc_hba *phba, struct lpfc_io_buf *psb);
 static int
 lpfc_prot_group_type(struct lpfc_hba *phba, struct scsi_cmnd *sc);
+static void
+lpfc_put_vmid_in_hashtable(struct lpfc_vport *vport, u32 hash,
+			   struct lpfc_vmid *vmp);
+static void lpfc_vmid_update_entry(struct lpfc_vport *vport, struct scsi_cmnd
+				   *cmd, struct lpfc_vmid *vmp,
+				   union lpfc_vmid_io_tag *tag);
+static void lpfc_vmid_assign_cs_ctl(struct lpfc_vport *vport,
+				    struct lpfc_vmid *vmid);
 
 static inline unsigned
 lpfc_cmd_blksize(struct scsi_cmnd *sc)
@@ -5178,6 +5187,115 @@ void lpfc_poll_timeout(struct timer_list *t)
 	}
 }
 
+/*
+ * lpfc_get_vmid_from_hashtable - search the UUID in the hash table
+ * @vport: The virtual port for which this call is being executed.
+ * @hash: calculated hash value
+ * @buf: uuid associated with the VE
+ * Returns the vmid entry associated with the UUID
+ * Make sure to acquire the appropriate lock before invoking this routine.
+ */
+struct lpfc_vmid *lpfc_get_vmid_from_hashtable(struct lpfc_vport *vport,
+					      u32 hash, u8 *buf)
+{
+	struct lpfc_vmid *vmp;
+
+	hash_for_each_possible(vport->hash_table, vmp, hnode, hash) {
+		if (memcmp(&vmp->host_vmid[0], buf, 16) == 0)
+			return vmp;
+	}
+	return NULL;
+}
+
+/*
+ * lpfc_put_vmid_in_hashtable - put the VMID in the hash table
+ * @vport: The virtual port for which this call is being executed.
+ * @hash - calculated hash value
+ * @vmp: Pointer to a VMID entry representing a VM sending IO
+ *
+ * This routine will insert the newly acquired vmid entity in the hash table.
+ * Make sure to acquire the appropriate lock before invoking this routine.
+ */
+static void
+lpfc_put_vmid_in_hashtable(struct lpfc_vport *vport, u32 hash,
+			   struct lpfc_vmid *vmp)
+{
+	hash_add(vport->hash_table, &vmp->hnode, hash);
+}
+
+/*
+ * lpfc_vmid_hash_fn - creates a hash value of the UUID
+ * @vmid: uuid associated with the VE
+ * @len: length of the vmid string
+ * Returns the calculated hash value
+ */
+int lpfc_vmid_hash_fn(const char *vmid, int len)
+{
+	int c;
+	int hash = 0;
+
+	if (len == 0)
+		return 0;
+	while (len--) {
+		c = *vmid++;
+		if (c >= 'A' && c <= 'Z')
+			c += 'a' - 'A';
+
+		hash = (hash + (c << LPFC_VMID_HASH_SHIFT) +
+			(c >> LPFC_VMID_HASH_SHIFT)) * 19;
+	}
+
+	return hash & LPFC_VMID_HASH_MASK;
+}
+
+/*
+ * lpfc_vmid_update_entry - update the vmid entry in the hash table
+ * @vport: The virtual port for which this call is being executed.
+ * @cmd: address of scsi cmmd descriptor
+ * @vmp: Pointer to a VMID entry representing a VM sending IO
+ * @tag: VMID tag
+ */
+static void lpfc_vmid_update_entry(struct lpfc_vport *vport, struct scsi_cmnd
+				   *cmd, struct lpfc_vmid *vmp,
+				   union lpfc_vmid_io_tag *tag)
+{
+	u64 *lta;
+
+	if (vport->vmid_priority_tagging)
+		tag->cs_ctl_vmid = vmp->un.cs_ctl_vmid;
+	else
+		tag->app_id = vmp->un.app_id;
+
+	if (cmd->sc_data_direction == DMA_TO_DEVICE)
+		vmp->io_wr_cnt++;
+	else
+		vmp->io_rd_cnt++;
+
+	/* update the last access timestamp in the table */
+	lta = per_cpu_ptr(vmp->last_io_time, raw_smp_processor_id());
+	*lta = jiffies;
+}
+
+static void lpfc_vmid_assign_cs_ctl(struct lpfc_vport *vport,
+				    struct lpfc_vmid *vmid)
+{
+	u32 hash;
+	struct lpfc_vmid *pvmid;
+
+	if (vport->port_type == LPFC_PHYSICAL_PORT) {
+		vmid->un.cs_ctl_vmid = lpfc_vmid_get_cs_ctl(vport);
+	} else {
+		hash = lpfc_vmid_hash_fn(vmid->host_vmid, vmid->vmid_len);
+		pvmid =
+		    lpfc_get_vmid_from_hashtable(vport->phba->pport, hash,
+						vmid->host_vmid);
+		if (pvmid)
+			vmid->un.cs_ctl_vmid = pvmid->un.cs_ctl_vmid;
+		else
+			vmid->un.cs_ctl_vmid = lpfc_vmid_get_cs_ctl(vport);
+	}
+}
+
 /**
  * lpfc_queuecommand - scsi_host_template queuecommand entry point
  * @shost: kernel scsi host pointer.
-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:50 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:50 +0530
Subject: [PATCH v11 07/13] lpfc: vmid: Implements ELS commands for appid patch
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-8-muneendra.kumar@broadcom.com>

From: Gaurav Srivastava <gaurav.srivastava at broadcom.com>

This patch implements ELS command like QFPA and UVEM for the priority
tagging appid support. Other supporting functions are also part of this
patch.

Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Gaurav Srivastava  <gaurav.srivastava at broadcom.com>
Signed-off-by: James Smart <jsmart2021 at gmail.com>

---
v11:
No change

v10:
Fixed the Spelling correction

v9:
Added a lock while accessing a flag

v8:
Added log messages modifications, memory allocation API changes,
return error codes

v7:
No change

v6:
Added Forward declarations, static functions and
removed unused variables

v5:
Changed Return code to non-numeric/Symbol.
Addressed the review comments by Hannes

v4:
No change

v3:
No change

v2:
Ported the patch on top of 5.10/scsi-queue
---
 drivers/scsi/lpfc/lpfc_els.c | 366 ++++++++++++++++++++++++++++++++++-
 1 file changed, 362 insertions(+), 4 deletions(-)

diff --git a/drivers/scsi/lpfc/lpfc_els.c b/drivers/scsi/lpfc/lpfc_els.c
index be9c92d97d63..d71782d0dc27 100644
--- a/drivers/scsi/lpfc/lpfc_els.c
+++ b/drivers/scsi/lpfc/lpfc_els.c
@@ -25,6 +25,7 @@
 #include <linux/pci.h>
 #include <linux/slab.h>
 #include <linux/interrupt.h>
+#include <linux/delay.h>
 
 #include <scsi/scsi.h>
 #include <scsi/scsi_device.h>
@@ -55,9 +56,15 @@ static int lpfc_issue_els_fdisc(struct lpfc_vport *vport,
 				struct lpfc_nodelist *ndlp, uint8_t retry);
 static int lpfc_issue_fabric_iocb(struct lpfc_hba *phba,
 				  struct lpfc_iocbq *iocb);
+static void lpfc_cmpl_els_uvem(struct lpfc_hba *, struct lpfc_iocbq *,
+			       struct lpfc_iocbq *);
 
 static int lpfc_max_els_tries = 3;
 
+static void lpfc_init_cs_ctl_bitmap(struct lpfc_vport *vport);
+static void lpfc_vmid_set_cs_ctl_range(struct lpfc_vport *vport, u32 min, u32 max);
+static void lpfc_vmid_put_cs_ctl(struct lpfc_vport *vport, u32 ctcl_vmid);
+
 /**
  * lpfc_els_chk_latt - Check host link attention event for a vport
  * @vport: pointer to a host virtual N_Port data structure.
@@ -314,10 +321,10 @@ lpfc_prep_els_iocb(struct lpfc_vport *vport, uint8_t expectRsp,
 		lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS,
 				 "0116 Xmit ELS command x%x to remote "
 				 "NPORT x%x I/O tag: x%x, port state:x%x "
-				 "rpi x%x fc_flag:x%x\n",
+				 "rpi x%x fc_flag:x%x nlp_flag:x%x vport:x%p\n",
 				 elscmd, did, elsiocb->iotag,
 				 vport->port_state, ndlp->nlp_rpi,
-				 vport->fc_flag);
+				 vport->fc_flag, ndlp->nlp_flag, vport);
 	} else {
 		/* Xmit ELS response <elsCmd> to remote NPORT <did> */
 		lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS,
@@ -1112,11 +1119,15 @@ lpfc_cmpl_els_flogi(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb,
 	/* FLOGI completes successfully */
 	lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS,
 			 "0101 FLOGI completes successfully, I/O tag:x%x, "
-			 "xri x%x Data: x%x x%x x%x x%x x%x %x\n",
+			 "xri x%x Data: x%x x%x x%x x%x x%x x%x x%x\n",
 			 cmdiocb->iotag, cmdiocb->sli4_xritag,
 			 irsp->un.ulpWord[4], sp->cmn.e_d_tov,
 			 sp->cmn.w2.r_a_tov, sp->cmn.edtovResolution,
-			 vport->port_state, vport->fc_flag);
+			 vport->port_state, vport->fc_flag,
+			 sp->cmn.priority_tagging);
+
+	if (sp->cmn.priority_tagging)
+		vport->vmid_flag |= LPFC_VMID_ISSUE_QFPA;
 
 	if (vport->port_state == LPFC_FLOGI) {
 		/*
@@ -1308,6 +1319,18 @@ lpfc_issue_els_flogi(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp,
 	if (sp->cmn.fcphHigh < FC_PH3)
 		sp->cmn.fcphHigh = FC_PH3;
 
+	/* to determine if switch supports priority tagging */
+	if (phba->cfg_vmid_priority_tagging) {
+		sp->cmn.priority_tagging = 1;
+		/* lpfc_vmid_host_uuid is combination of wwpn and wwnn */
+		if (uuid_is_null((uuid_t *)vport->lpfc_vmid_host_uuid)) {
+			memcpy(vport->lpfc_vmid_host_uuid, phba->wwpn,
+			       sizeof(phba->wwpn));
+			memcpy(&vport->lpfc_vmid_host_uuid[8], phba->wwnn,
+			       sizeof(phba->wwnn));
+		}
+	}
+
 	if  (phba->sli_rev == LPFC_SLI_REV4) {
 		if (bf_get(lpfc_sli_intf_if_type, &phba->sli4_hba.sli_intf) ==
 		    LPFC_SLI_INTF_IF_TYPE_0) {
@@ -1934,6 +1957,7 @@ lpfc_cmpl_els_plogi(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb,
 	struct lpfc_nodelist *ndlp, *free_ndlp;
 	struct lpfc_dmabuf *prsp;
 	int disc;
+	struct serv_parm *sp = NULL;
 
 	/* we pass cmdiocb to state machine which needs rspiocb as well */
 	cmdiocb->context_un.rsp_iocb = rspiocb;
@@ -2035,6 +2059,23 @@ lpfc_cmpl_els_plogi(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb,
 				   cmdiocb->context2)->list.next,
 				  struct lpfc_dmabuf, list);
 		ndlp = lpfc_plogi_confirm_nport(phba, prsp->virt, ndlp);
+
+		sp = (struct serv_parm *)((u8 *)prsp->virt +
+					  sizeof(u32));
+
+		ndlp->vmid_support = 0;
+		if ((phba->cfg_vmid_app_header && sp->cmn.app_hdr_support) ||
+		    (phba->cfg_vmid_priority_tagging &&
+		     sp->cmn.priority_tagging)) {
+			lpfc_printf_log(phba, KERN_DEBUG, LOG_ELS,
+					"4018 app_hdr_support %d tagging %d DID x%x\n",
+					sp->cmn.app_hdr_support,
+					sp->cmn.priority_tagging,
+					ndlp->nlp_DID);
+			/* if the dest port supports VMID, mark it in ndlp */
+			ndlp->vmid_support = 1;
+		}
+
 		lpfc_disc_state_machine(vport, ndlp, cmdiocb,
 					NLP_EVT_CMPL_PLOGI);
 	}
@@ -2157,6 +2198,14 @@ lpfc_issue_els_plogi(struct lpfc_vport *vport, uint32_t did, uint8_t retry)
 	memset(sp->un.vendorVersion, 0, sizeof(sp->un.vendorVersion));
 	sp->cmn.bbRcvSizeMsb &= 0xF;
 
+	/* check if the destination port supports VMID */
+	ndlp->vmid_support = 0;
+	if (vport->vmid_priority_tagging)
+		sp->cmn.priority_tagging = 1;
+	else if (phba->cfg_vmid_app_header &&
+		 bf_get(lpfc_ftr_ashdr, &phba->sli4_hba.sli4_flags))
+		sp->cmn.app_hdr_support = 1;
+
 	lpfc_debugfs_disc_trc(vport, LPFC_DISC_TRC_ELS_CMD,
 		"Issue PLOGI:     did:x%x",
 		did, 0, 0);
@@ -10465,3 +10514,312 @@ lpfc_sli_abts_recover_port(struct lpfc_vport *vport,
 	lpfc_unreg_rpi(vport, ndlp);
 }
 
+static void lpfc_init_cs_ctl_bitmap(struct lpfc_vport *vport)
+{
+	bitmap_zero(vport->vmid_priority_range, LPFC_VMID_MAX_PRIORITY_RANGE);
+}
+
+static void
+lpfc_vmid_set_cs_ctl_range(struct lpfc_vport *vport, u32 min, u32 max)
+{
+	u32 i;
+
+	if ((min > max) || (max > LPFC_VMID_MAX_PRIORITY_RANGE))
+		return;
+
+	for (i = min; i <= max; i++)
+		set_bit(i, vport->vmid_priority_range);
+}
+
+static void lpfc_vmid_put_cs_ctl(struct lpfc_vport *vport, u32 ctcl_vmid)
+{
+	set_bit(ctcl_vmid, vport->vmid_priority_range);
+}
+
+u32 lpfc_vmid_get_cs_ctl(struct lpfc_vport *vport)
+{
+	u32 i;
+
+	i = find_first_bit(vport->vmid_priority_range,
+			   LPFC_VMID_MAX_PRIORITY_RANGE);
+
+	if (i == LPFC_VMID_MAX_PRIORITY_RANGE)
+		return 0;
+
+	clear_bit(i, vport->vmid_priority_range);
+	return i;
+}
+
+#define MAX_PRIORITY_DESC	255
+
+static void
+lpfc_cmpl_els_qfpa(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb,
+		   struct lpfc_iocbq *rspiocb)
+{
+	struct lpfc_vport *vport = cmdiocb->vport;
+	struct priority_range_desc *desc;
+	struct lpfc_dmabuf *prsp = NULL;
+	struct lpfc_vmid_priority_range *vmid_range = NULL;
+	u32 *data;
+	struct lpfc_dmabuf *dmabuf = cmdiocb->context2;
+	IOCB_t *irsp = &rspiocb->iocb;
+	u8 *pcmd, max_desc;
+	u32 len, i;
+	struct lpfc_nodelist *ndlp = (struct lpfc_nodelist *)cmdiocb->context1;
+
+	prsp = list_get_first(&dmabuf->list, struct lpfc_dmabuf, list);
+	if (!prsp)
+		goto out;
+
+	pcmd = prsp->virt;
+	data = (u32 *)pcmd;
+	if (data[0] == ELS_CMD_LS_RJT) {
+		lpfc_printf_vlog(vport, KERN_WARNING, LOG_SLI,
+				 "3277 QFPA LS_RJT x%x  x%x\n",
+				 data[0], data[1]);
+		goto out;
+	}
+	if (irsp->ulpStatus) {
+		lpfc_printf_vlog(vport, KERN_ERR, LOG_SLI,
+				 "6529 QFPA failed with status x%x  x%x\n",
+				 irsp->ulpStatus, irsp->un.ulpWord[4]);
+		goto out;
+	}
+
+	if (!vport->qfpa_res) {
+		max_desc = FCELSSIZE / sizeof(*vport->qfpa_res);
+		vport->qfpa_res = kcalloc(max_desc, sizeof(*vport->qfpa_res),
+					  GFP_KERNEL);
+		if (!vport->qfpa_res)
+			goto out;
+	}
+
+	len = *((u32 *)(pcmd + 4));
+	len = be32_to_cpu(len);
+	memcpy(vport->qfpa_res, pcmd, len + 8);
+	len = len / LPFC_PRIORITY_RANGE_DESC_SIZE;
+
+	desc = (struct priority_range_desc *)(pcmd + 8);
+	vmid_range = vport->vmid_priority.vmid_range;
+	if (!vmid_range) {
+		vmid_range = kcalloc(MAX_PRIORITY_DESC, sizeof(*vmid_range),
+				     GFP_KERNEL);
+		if (!vmid_range) {
+			kfree(vport->qfpa_res);
+			goto out;
+		}
+		vport->vmid_priority.vmid_range = vmid_range;
+	}
+	vport->vmid_priority.num_descriptors = len;
+
+	for (i = 0; i < len; i++, vmid_range++, desc++) {
+		lpfc_printf_vlog(vport, KERN_DEBUG, LOG_ELS,
+				 "6539 vmid values low=%d, high=%d, qos=%d, "
+				 "local ve id=%d\n", desc->lo_range,
+				 desc->hi_range, desc->qos_priority,
+				 desc->local_ve_id);
+
+		vmid_range->low = desc->lo_range << 1;
+		if (desc->local_ve_id == QFPA_ODD_ONLY)
+			vmid_range->low++;
+		if (desc->qos_priority)
+			vport->vmid_flag |= LPFC_VMID_QOS_ENABLED;
+		vmid_range->qos = desc->qos_priority;
+
+		vmid_range->high = desc->hi_range << 1;
+		if ((desc->local_ve_id == QFPA_ODD_ONLY) ||
+		    (desc->local_ve_id == QFPA_EVEN_ODD))
+			vmid_range->high++;
+	}
+	lpfc_init_cs_ctl_bitmap(vport);
+	for (i = 0; i < vport->vmid_priority.num_descriptors; i++) {
+		lpfc_vmid_set_cs_ctl_range(vport,
+				vport->vmid_priority.vmid_range[i].low,
+				vport->vmid_priority.vmid_range[i].high);
+	}
+
+	vport->vmid_flag |= LPFC_VMID_QFPA_CMPL;
+ out:
+	lpfc_els_free_iocb(phba, cmdiocb);
+	lpfc_nlp_put(ndlp);
+}
+
+int lpfc_issue_els_qfpa(struct lpfc_vport *vport)
+{
+	struct lpfc_hba *phba = vport->phba;
+	struct lpfc_nodelist *ndlp;
+	struct lpfc_iocbq *elsiocb;
+	u8 *pcmd;
+	int ret;
+
+	ndlp = lpfc_findnode_did(phba->pport, Fabric_DID);
+	if (!ndlp || ndlp->nlp_state != NLP_STE_UNMAPPED_NODE)
+		return -ENXIO;
+
+	elsiocb = lpfc_prep_els_iocb(vport, 1, LPFC_QFPA_SIZE, 2, ndlp,
+				     ndlp->nlp_DID, ELS_CMD_QFPA);
+	if (!elsiocb)
+		return -ENOMEM;
+
+	pcmd = (u8 *)(((struct lpfc_dmabuf *)elsiocb->context2)->virt);
+
+	*((u32 *)(pcmd)) = ELS_CMD_QFPA;
+	pcmd += 4;
+
+	elsiocb->iocb_cmpl = lpfc_cmpl_els_qfpa;
+
+	elsiocb->context1 = lpfc_nlp_get(ndlp);
+	if (!elsiocb->context1) {
+		lpfc_els_free_iocb(vport->phba, elsiocb);
+		return -ENXIO;
+	}
+
+	ret = lpfc_sli_issue_iocb(phba, LPFC_ELS_RING, elsiocb, 2);
+	if (ret != IOCB_SUCCESS) {
+		lpfc_els_free_iocb(phba, elsiocb);
+		lpfc_nlp_put(ndlp);
+		return -EIO;
+	}
+	vport->vmid_flag &= ~LPFC_VMID_QOS_ENABLED;
+	return 0;
+}
+
+int
+lpfc_vmid_uvem(struct lpfc_vport *vport,
+	       struct lpfc_vmid *vmid, bool instantiated)
+{
+	struct lpfc_vem_id_desc *vem_id_desc;
+	struct lpfc_nodelist *ndlp;
+	struct lpfc_iocbq *elsiocb;
+	struct instantiated_ve_desc *inst_desc;
+	struct lpfc_vmid_context *vmid_context;
+	u8 *pcmd;
+	u32 *len;
+	int ret = 0;
+
+	ndlp = lpfc_findnode_did(vport, Fabric_DID);
+	if (!ndlp || ndlp->nlp_state != NLP_STE_UNMAPPED_NODE)
+		return -ENXIO;
+
+	vmid_context = kmalloc(sizeof(*vmid_context), GFP_KERNEL);
+	if (!vmid_context)
+		return -ENOMEM;
+	elsiocb = lpfc_prep_els_iocb(vport, 1, LPFC_UVEM_SIZE, 2,
+				     ndlp, Fabric_DID, ELS_CMD_UVEM);
+	if (!elsiocb)
+		goto out;
+
+	lpfc_printf_vlog(vport, KERN_DEBUG, LOG_ELS,
+			 "3427 Host vmid %s %d\n",
+			 vmid->host_vmid, instantiated);
+	vmid_context->vmp = vmid;
+	vmid_context->nlp = ndlp;
+	vmid_context->instantiated = instantiated;
+	elsiocb->vmid_tag.vmid_context = vmid_context;
+	pcmd = (u8 *)(((struct lpfc_dmabuf *)elsiocb->context2)->virt);
+
+	if (uuid_is_null((uuid_t *)vport->lpfc_vmid_host_uuid))
+		memcpy(vport->lpfc_vmid_host_uuid, vmid->host_vmid,
+		       LPFC_COMPRESS_VMID_SIZE);
+
+	*((u32 *)(pcmd)) = ELS_CMD_UVEM;
+	len = (u32 *)(pcmd + 4);
+	*len = cpu_to_be32(LPFC_UVEM_SIZE - 8);
+
+	vem_id_desc = (struct lpfc_vem_id_desc *)(pcmd + 8);
+	vem_id_desc->tag = be32_to_cpu(VEM_ID_DESC_TAG);
+	vem_id_desc->length = be32_to_cpu(LPFC_UVEM_VEM_ID_DESC_SIZE);
+	memcpy(vem_id_desc->vem_id, vport->lpfc_vmid_host_uuid,
+	       LPFC_COMPRESS_VMID_SIZE);
+
+	inst_desc = (struct instantiated_ve_desc *)(pcmd + 32);
+	inst_desc->tag = be32_to_cpu(INSTANTIATED_VE_DESC_TAG);
+	inst_desc->length = be32_to_cpu(LPFC_UVEM_VE_MAP_DESC_SIZE);
+	memcpy(inst_desc->global_vem_id, vmid->host_vmid,
+	       LPFC_COMPRESS_VMID_SIZE);
+
+	bf_set(lpfc_instantiated_nport_id, inst_desc, vport->fc_myDID);
+	bf_set(lpfc_instantiated_local_id, inst_desc,
+	       vmid->un.cs_ctl_vmid);
+	if (instantiated) {
+		inst_desc->tag = be32_to_cpu(INSTANTIATED_VE_DESC_TAG);
+	} else {
+		inst_desc->tag = be32_to_cpu(DEINSTANTIATED_VE_DESC_TAG);
+		lpfc_vmid_put_cs_ctl(vport, vmid->un.cs_ctl_vmid);
+	}
+	inst_desc->word6 = cpu_to_be32(inst_desc->word6);
+
+	elsiocb->iocb_cmpl = lpfc_cmpl_els_uvem;
+
+	elsiocb->context1 = lpfc_nlp_get(ndlp);
+	if (!elsiocb->context1) {
+		lpfc_els_free_iocb(vport->phba, elsiocb);
+		goto out;
+	}
+
+	ret = lpfc_sli_issue_iocb(vport->phba, LPFC_ELS_RING, elsiocb, 0);
+	if (ret != IOCB_SUCCESS) {
+		lpfc_els_free_iocb(vport->phba, elsiocb);
+		lpfc_nlp_put(ndlp);
+		goto out;
+	}
+
+	return 0;
+ out:
+	kfree(vmid_context);
+	return -EIO;
+}
+
+static void
+lpfc_cmpl_els_uvem(struct lpfc_hba *phba, struct lpfc_iocbq *icmdiocb,
+		   struct lpfc_iocbq *rspiocb)
+{
+	struct lpfc_vport *vport = icmdiocb->vport;
+	struct lpfc_dmabuf *prsp = NULL;
+	struct lpfc_vmid_context *vmid_context =
+	    icmdiocb->vmid_tag.vmid_context;
+	struct lpfc_nodelist *ndlp = icmdiocb->context1;
+	u8 *pcmd;
+	u32 *data;
+	IOCB_t *irsp = &rspiocb->iocb;
+	struct lpfc_dmabuf *dmabuf = icmdiocb->context2;
+	struct lpfc_vmid *vmid;
+
+	vmid = vmid_context->vmp;
+	if (!ndlp || ndlp->nlp_state != NLP_STE_UNMAPPED_NODE)
+		ndlp = NULL;
+
+	prsp = list_get_first(&dmabuf->list, struct lpfc_dmabuf, list);
+	if (!prsp)
+		goto out;
+	pcmd = prsp->virt;
+	data = (u32 *)pcmd;
+	if (data[0] == ELS_CMD_LS_RJT) {
+		lpfc_printf_vlog(vport, KERN_WARNING, LOG_SLI,
+				 "4532 UVEM LS_RJT %x %x\n", data[0], data[1]);
+		goto out;
+	}
+	if (irsp->ulpStatus) {
+		lpfc_printf_vlog(vport, KERN_WARNING, LOG_SLI,
+				 "4533 UVEM error status %x: %x\n",
+				 irsp->ulpStatus, irsp->un.ulpWord[4]);
+		goto out;
+	}
+	spin_lock(&phba->hbalock);
+	/* Set IN USE flag */
+	vport->vmid_flag |= LPFC_VMID_IN_USE;
+	phba->pport->vmid_flag |= LPFC_VMID_IN_USE;
+	spin_unlock(&phba->hbalock);
+
+	if (vmid_context->instantiated) {
+		write_lock(&vport->vmid_lock);
+		vmid->flag |= LPFC_VMID_REGISTERED;
+		vmid->flag &= ~LPFC_VMID_REQ_REGISTER;
+		write_unlock(&vport->vmid_lock);
+	}
+
+ out:
+	kfree(vmid_context);
+	lpfc_els_free_iocb(phba, icmdiocb);
+	lpfc_nlp_put(ndlp);
+}
-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:52 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:52 +0530
Subject: [PATCH v11 09/13] lpfc: vmid: Implements CT commands for appid.
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-10-muneendra.kumar@broadcom.com>

From: Gaurav Srivastava <gaurav.srivastava at broadcom.com>

The patch implements CT commands for registering and deregistering the
appid for the application. Also, a small change in decrementing the ndlp
ref counter has been added.

Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Gaurav Srivastava  <gaurav.srivastava at broadcom.com>
Signed-off-by: James Smart <jsmart2021 at gmail.com>

---
v11:
No change

v10:
Function name correction

v9:
Added Changes related to locking and new hashtable implementation

v8:
modified log messages and fixed the refcount

v7:
No Change

v6:
Added Forward declarations and functions to static

v5:
No change

v4:
No change

v3:
No change

v2:
Ported the patch on top of 5.10/scsi-queue
Removed redundant code due to changes since last submit
---
 drivers/scsi/lpfc/lpfc_ct.c | 255 ++++++++++++++++++++++++++++++++++++
 1 file changed, 255 insertions(+)

diff --git a/drivers/scsi/lpfc/lpfc_ct.c b/drivers/scsi/lpfc/lpfc_ct.c
index cbf1be56b1e7..610b6dabb3b5 100644
--- a/drivers/scsi/lpfc/lpfc_ct.c
+++ b/drivers/scsi/lpfc/lpfc_ct.c
@@ -75,6 +75,9 @@
 
 
 static char *lpfc_release_version = LPFC_DRIVER_VERSION;
+static void
+lpfc_cmpl_ct_cmd_vmid(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb,
+		      struct lpfc_iocbq *rspiocb);
 
 static void
 lpfc_ct_ignore_hbq_buffer(struct lpfc_hba *phba, struct lpfc_iocbq *piocbq,
@@ -3777,3 +3780,255 @@ lpfc_decode_firmware_rev(struct lpfc_hba *phba, char *fwrevision, int flag)
 	}
 	return;
 }
+
+static void
+lpfc_cmpl_ct_cmd_vmid(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb,
+		      struct lpfc_iocbq *rspiocb)
+{
+	struct lpfc_vport *vport = cmdiocb->vport;
+	struct lpfc_dmabuf *inp = cmdiocb->context1;
+	struct lpfc_dmabuf *outp = cmdiocb->context2;
+	struct lpfc_sli_ct_request *ctcmd = inp->virt;
+	struct lpfc_sli_ct_request *ctrsp = outp->virt;
+	u16 rsp = ctrsp->CommandResponse.bits.CmdRsp;
+	struct app_id_object *app;
+	u32 cmd, hash, bucket;
+	struct lpfc_vmid *vmp, *cur;
+	u8 *data = outp->virt;
+	int i;
+
+	cmd = be16_to_cpu(ctcmd->CommandResponse.bits.CmdRsp);
+	if (cmd == SLI_CTAS_DALLAPP_ID)
+		lpfc_ct_free_iocb(phba, cmdiocb);
+
+	if (lpfc_els_chk_latt(vport) || rspiocb->iocb.ulpStatus) {
+		if (cmd != SLI_CTAS_DALLAPP_ID)
+			return;
+	}
+	/* Check for a CT LS_RJT response */
+	if (rsp == be16_to_cpu(SLI_CT_RESPONSE_FS_RJT)) {
+		if (cmd != SLI_CTAS_DALLAPP_ID)
+			lpfc_printf_vlog(vport, KERN_DEBUG, LOG_DISCOVERY,
+					 "3306 VMID FS_RJT Data: x%x x%x x%x\n",
+					 cmd, ctrsp->ReasonCode,
+					 ctrsp->Explanation);
+		if ((cmd != SLI_CTAS_DALLAPP_ID) ||
+		    (ctrsp->ReasonCode != SLI_CT_UNABLE_TO_PERFORM_REQ) ||
+		    (ctrsp->Explanation != SLI_CT_APP_ID_NOT_AVAILABLE)) {
+			/* If DALLAPP_ID failed retry later */
+			if (cmd == SLI_CTAS_DALLAPP_ID)
+				vport->load_flag |= FC_DEREGISTER_ALL_APP_ID;
+			return;
+		}
+	}
+
+	switch (cmd) {
+	case SLI_CTAS_RAPP_IDENT:
+		app = (struct app_id_object *)(RAPP_IDENT_OFFSET + data);
+		lpfc_printf_vlog(vport, KERN_DEBUG, LOG_DISCOVERY,
+				 "6712 RAPP_IDENT app id %d  port id x%x id "
+				 "len %d\n", be32_to_cpu(app->app_id),
+				 be32_to_cpu(app->port_id),
+				 app->obj.entity_id_len);
+
+		if (app->obj.entity_id_len == 0 || app->port_id == 0)
+			return;
+
+		hash = lpfc_vmid_hash_fn(app->obj.entity_id,
+					 app->obj.entity_id_len);
+		vmp = lpfc_get_vmid_from_hashtable(vport, hash,
+						  app->obj.entity_id);
+		if (vmp) {
+			write_lock(&vport->vmid_lock);
+			vmp->un.app_id = be32_to_cpu(app->app_id);
+			vmp->flag |= LPFC_VMID_REGISTERED;
+			vmp->flag &= ~LPFC_VMID_REQ_REGISTER;
+			write_unlock(&vport->vmid_lock);
+			/* Set IN USE flag */
+			vport->vmid_flag |= LPFC_VMID_IN_USE;
+		} else {
+			lpfc_printf_vlog(vport, KERN_DEBUG, LOG_DISCOVERY,
+					 "6901 No entry found %s hash %d\n",
+					 app->obj.entity_id, hash);
+		}
+		break;
+	case SLI_CTAS_DAPP_IDENT:
+		app = (struct app_id_object *)(DAPP_IDENT_OFFSET + data);
+		lpfc_printf_vlog(vport, KERN_DEBUG, LOG_DISCOVERY,
+				 "6713 DAPP_IDENT app id %d  port id x%x\n",
+				 be32_to_cpu(app->app_id),
+				 be32_to_cpu(app->port_id));
+		break;
+	case SLI_CTAS_DALLAPP_ID:
+		lpfc_printf_vlog(vport, KERN_DEBUG, LOG_DISCOVERY,
+				 "8856 Deregistered all app ids\n");
+		read_lock(&vport->vmid_lock);
+		for (i = 0; i < phba->cfg_max_vmid; i++) {
+			vmp = &vport->vmid[i];
+			if (vmp->flag != LPFC_VMID_SLOT_FREE)
+				memset(vmp, 0, sizeof(struct lpfc_vmid));
+		}
+		read_unlock(&vport->vmid_lock);
+		/* for all elements in the hash table */
+		if (!hash_empty(vport->hash_table))
+			hash_for_each(vport->hash_table, bucket, cur, hnode)
+				hash_del(&cur->hnode);
+		vport->load_flag |= FC_ALLOW_VMID;
+		break;
+	default:
+		lpfc_printf_vlog(vport, KERN_DEBUG, LOG_DISCOVERY,
+				 "8857 Invalid command code\n");
+	}
+}
+
+/**
+ * lpfc_vmid_cmd - Build and send a FDMI cmd to the specified NPort
+ * @vport: pointer to a host virtual N_Port data structure.
+ * @ndlp: ndlp to send FDMI cmd to (if NULL use FDMI_DID)
+ * cmdcode: FDMI command to send
+ * mask: Mask of HBA or PORT Attributes to send
+ *
+ * Builds and sends a FDMI command using the CT subsystem.
+ */
+int
+lpfc_vmid_cmd(struct lpfc_vport *vport,
+	      int cmdcode, struct lpfc_vmid *vmid)
+{
+	struct lpfc_hba *phba = vport->phba;
+	struct lpfc_dmabuf *mp, *bmp;
+	struct lpfc_sli_ct_request *ctreq;
+	struct ulp_bde64 *bpl;
+	u32 size;
+	u32 rsp_size;
+	u8 *data;
+	struct lpfc_vmid_rapp_ident_list *rap;
+	struct lpfc_vmid_dapp_ident_list *dap;
+	u8 retry = 0;
+	struct lpfc_nodelist *ndlp;
+
+	void (*cmpl)(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb,
+		     struct lpfc_iocbq *rspiocb);
+
+	ndlp = lpfc_findnode_did(vport, FDMI_DID);
+	if (!ndlp || ndlp->nlp_state != NLP_STE_UNMAPPED_NODE)
+		return 0;
+
+	cmpl = lpfc_cmpl_ct_cmd_vmid;
+
+	/* fill in BDEs for command */
+	/* Allocate buffer for command payload */
+	mp = kmalloc(sizeof(*mp), GFP_KERNEL);
+	if (!mp)
+		goto vmid_free_mp_exit;
+
+	mp->virt = lpfc_mbuf_alloc(phba, 0, &mp->phys);
+	if (!mp->virt)
+		goto vmid_free_mp_virt_exit;
+
+	/* Allocate buffer for Buffer ptr list */
+	bmp = kmalloc(sizeof(*bmp), GFP_KERNEL);
+	if (!bmp)
+		goto vmid_free_bmp_exit;
+
+	bmp->virt = lpfc_mbuf_alloc(phba, 0, &bmp->phys);
+	if (!bmp->virt)
+		goto vmid_free_bmp_virt_exit;
+
+	INIT_LIST_HEAD(&mp->list);
+	INIT_LIST_HEAD(&bmp->list);
+
+	lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY,
+			 "3275 VMID Request Data: x%x x%x x%x\n",
+			 vport->fc_flag, vport->port_state, cmdcode);
+	ctreq = (struct lpfc_sli_ct_request *)mp->virt;
+	data = mp->virt;
+	/* First populate the CT_IU preamble */
+	memset(data, 0, LPFC_BPL_SIZE);
+	ctreq->RevisionId.bits.Revision = SLI_CT_REVISION;
+	ctreq->RevisionId.bits.InId = 0;
+
+	ctreq->FsType = SLI_CT_MANAGEMENT_SERVICE;
+	ctreq->FsSubType = SLI_CT_APP_SEV_Subtypes;
+
+	ctreq->CommandResponse.bits.CmdRsp = cpu_to_be16(cmdcode);
+	rsp_size = LPFC_BPL_SIZE;
+	size = 0;
+
+	switch (cmdcode) {
+	case SLI_CTAS_RAPP_IDENT:
+		lpfc_printf_vlog(vport, KERN_DEBUG, LOG_DISCOVERY,
+				 "1329 RAPP_IDENT for %s\n", vmid->host_vmid);
+		ctreq->un.PortID = cpu_to_be32(vport->fc_myDID);
+		rap = (struct lpfc_vmid_rapp_ident_list *)
+			(DAPP_IDENT_OFFSET + data);
+		rap->no_of_objects = cpu_to_be32(1);
+		rap->obj[0].entity_id_len = vmid->vmid_len;
+		memcpy(rap->obj[0].entity_id, vmid->host_vmid, vmid->vmid_len);
+		size = RAPP_IDENT_OFFSET +
+			sizeof(struct lpfc_vmid_rapp_ident_list);
+		retry = 1;
+		break;
+
+	case SLI_CTAS_GALLAPPIA_ID:
+		ctreq->un.PortID = cpu_to_be32(vport->fc_myDID);
+		size = GALLAPPIA_ID_SIZE;
+		break;
+
+	case SLI_CTAS_DAPP_IDENT:
+		lpfc_printf_vlog(vport, KERN_DEBUG, LOG_DISCOVERY,
+				 "1469 DAPP_IDENT for %s\n", vmid->host_vmid);
+		ctreq->un.PortID = cpu_to_be32(vport->fc_myDID);
+		dap = (struct lpfc_vmid_dapp_ident_list *)
+			(DAPP_IDENT_OFFSET + data);
+		dap->no_of_objects = cpu_to_be32(1);
+		dap->obj[0].entity_id_len = vmid->vmid_len;
+		memcpy(dap->obj[0].entity_id, vmid->host_vmid, vmid->vmid_len);
+		size = DAPP_IDENT_OFFSET +
+			sizeof(struct lpfc_vmid_dapp_ident_list);
+		write_lock(&vport->vmid_lock);
+		vmid->flag &= ~LPFC_VMID_REGISTERED;
+		write_unlock(&vport->vmid_lock);
+		retry = 1;
+		break;
+
+	case SLI_CTAS_DALLAPP_ID:
+		ctreq->un.PortID = cpu_to_be32(vport->fc_myDID);
+		size = DALLAPP_ID_SIZE;
+		break;
+
+	default:
+		lpfc_printf_vlog(vport, KERN_DEBUG, LOG_DISCOVERY,
+				 "7062 VMID cmdcode x%x not supported\n",
+				 cmdcode);
+		goto vmid_free_all_mem;
+	}
+
+	ctreq->CommandResponse.bits.Size = cpu_to_be16(rsp_size);
+
+	bpl = (struct ulp_bde64 *)bmp->virt;
+	bpl->addrHigh = putPaddrHigh(mp->phys);
+	bpl->addrLow = putPaddrLow(mp->phys);
+	bpl->tus.f.bdeFlags = 0;
+	bpl->tus.f.bdeSize = size;
+
+	/* The lpfc_ct_cmd/lpfc_get_req shall increment ndlp reference count
+	 * to hold ndlp reference for the corresponding callback function.
+	 */
+	if (!lpfc_ct_cmd(vport, mp, bmp, ndlp, cmpl, rsp_size, retry))
+		return 0;
+
+ vmid_free_all_mem:
+	lpfc_mbuf_free(phba, bmp->virt, bmp->phys);
+ vmid_free_bmp_virt_exit:
+	kfree(bmp);
+ vmid_free_bmp_exit:
+	lpfc_mbuf_free(phba, mp->virt, mp->phys);
+ vmid_free_mp_virt_exit:
+	kfree(mp);
+ vmid_free_mp_exit:
+
+	/* Issue CT request failed */
+	lpfc_printf_vlog(vport, KERN_DEBUG, LOG_DISCOVERY,
+			 "3276 VMID CT request failed Data: x%x\n", cmdcode);
+	return -EIO;
+}
-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:53 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:53 +0530
Subject: [PATCH v11 10/13] lpfc: vmid: Appends the vmid in the wqe before
 sending
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-11-muneendra.kumar@broadcom.com>

From: Gaurav Srivastava <gaurav.srivastava at broadcom.com>

This patch adds the vmid in wqe before sending out the request.
The type of vmid depends on the configured type and is checked before
being appended.

Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Gaurav Srivastava  <gaurav.srivastava at broadcom.com>
Signed-off-by: James Smart <jsmart2021 at gmail.com>

---
v11:
No change

v10:
Removed redudant code

v9:
No change

v8:
Modified the log messages

v7:
No change

v6:
No change

v5:
Modified the comments

v4:
No change

v3:
No change

v2:
Ported the patch on top of 5.10/scsi-queue
---
 drivers/scsi/lpfc/lpfc_sli.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/scsi/lpfc/lpfc_sli.c b/drivers/scsi/lpfc/lpfc_sli.c
index c7d0acd6c8ef..6555308cea7a 100644
--- a/drivers/scsi/lpfc/lpfc_sli.c
+++ b/drivers/scsi/lpfc/lpfc_sli.c
@@ -9766,6 +9766,8 @@ lpfc_sli4_iocb2wqe(struct lpfc_hba *phba, struct lpfc_iocbq *iocbq,
 				*pcmd == ELS_CMD_RSCN_XMT ||
 				*pcmd == ELS_CMD_FDISC ||
 				*pcmd == ELS_CMD_LOGO ||
+				*pcmd == ELS_CMD_QFPA ||
+				*pcmd == ELS_CMD_UVEM ||
 				*pcmd == ELS_CMD_PLOGI)) {
 				bf_set(els_req64_sp, &wqe->els_req, 1);
 				bf_set(els_req64_sid, &wqe->els_req,
@@ -10328,6 +10330,18 @@ __lpfc_sli_issue_fcp_io_s4(struct lpfc_hba *phba, uint32_t ring_number,
 		bf_set(wqe_wqes, &wqe->generic.wqe_com, 0);
 	}
 
+	/* add the VMID tags as per switch response */
+	if (unlikely(piocb->iocb_flag & LPFC_IO_VMID)) {
+		if (phba->pport->vmid_priority_tagging) {
+			bf_set(wqe_ccpe, &wqe->fcp_iwrite.wqe_com, 1);
+			bf_set(wqe_ccp, &wqe->fcp_iwrite.wqe_com,
+					(piocb->vmid_tag.cs_ctl_vmid));
+		} else {
+			bf_set(wqe_appid, &wqe->fcp_iwrite.wqe_com, 1);
+			bf_set(wqe_wqes, &wqe->fcp_iwrite.wqe_com, 1);
+			wqe->words[31] = piocb->vmid_tag.app_id;
+		}
+	}
 	rc = lpfc_sli4_issue_wqe(phba, lpfc_cmd->hdwq, piocb);
 	return rc;
 }
-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:54 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:54 +0530
Subject: [PATCH v11 11/13] lpfc: vmid: Timeout implementation for vmid
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-12-muneendra.kumar@broadcom.com>

From: Gaurav Srivastava <gaurav.srivastava at broadcom.com>

This patch implements the timeout functionality for the vmid. After the
set time period of inactivity, the vmid is deregistered from the switch.

Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Gaurav Srivastava  <gaurav.srivastava at broadcom.com>
Signed-off-by: James Smart <jsmart2021 at gmail.com>

---
v11:
No change

v10:
No change

v9:
Modified code for use of hashtable

v8:
Fixed the uninitialized variable

v7:
No change

v6:
Added Forward declarations

v5:
No change

v4:
No change

v3:
No change

v2:
Ported the patch on top of 5.10/scsi-queue
---
 drivers/scsi/lpfc/lpfc_hbadisc.c | 105 +++++++++++++++++++++++++++++++
 drivers/scsi/lpfc/lpfc_init.c    |  40 ++++++++++++
 2 files changed, 145 insertions(+)

diff --git a/drivers/scsi/lpfc/lpfc_hbadisc.c b/drivers/scsi/lpfc/lpfc_hbadisc.c
index 04b6e18c5342..41a388aab1cf 100644
--- a/drivers/scsi/lpfc/lpfc_hbadisc.c
+++ b/drivers/scsi/lpfc/lpfc_hbadisc.c
@@ -72,6 +72,7 @@ static void lpfc_disc_flush_list(struct lpfc_vport *vport);
 static void lpfc_unregister_fcfi_cmpl(struct lpfc_hba *, LPFC_MBOXQ_t *);
 static int lpfc_fcf_inuse(struct lpfc_hba *);
 static void lpfc_mbx_cmpl_read_sparam(struct lpfc_hba *, LPFC_MBOXQ_t *);
+static void lpfc_check_inactive_vmid(struct lpfc_hba *phba);
 
 static int
 lpfc_valid_xpt_node(struct lpfc_nodelist *ndlp)
@@ -234,6 +235,110 @@ lpfc_dev_loss_tmo_callbk(struct fc_rport *rport)
 	return;
 }
 
+/**
+ * lpfc_check_inactive_vmid_one - VMID inactivity checker for a vport
+ * @vport: Pointer to vport context object.
+ *
+ * This function checks for idle vmid entries related to a particular vport. If
+ * found unused/idle, it frees them accordingly.
+ **/
+static void lpfc_check_inactive_vmid_one(struct lpfc_vport *vport)
+{
+	u16 keep;
+	u32 difftime = 0, r, bucket;
+	u64 *lta;
+	int cpu;
+	struct lpfc_vmid *vmp;
+
+	write_lock(&vport->vmid_lock);
+
+	if (!vport->cur_vmid_cnt)
+		goto out;
+
+	/* iterate through the table */
+	hash_for_each(vport->hash_table, bucket, vmp, hnode) {
+		keep = 0;
+		if (vmp->flag & LPFC_VMID_REGISTERED) {
+			/* check if the particular vmid is in use */
+			/* for all available per cpu variable */
+			for_each_possible_cpu(cpu) {
+				/* if last access time is less than timeout */
+				lta = per_cpu_ptr(vmp->last_io_time, cpu);
+				if (!lta)
+					continue;
+				difftime = (jiffies) - (*lta);
+				if ((vport->vmid_inactivity_timeout *
+				     JIFFIES_PER_HR) > difftime) {
+					keep = 1;
+					break;
+				}
+			}
+
+			/* if none of the cpus have been used by the vm, */
+			/*  remove the entry if already registered */
+			if (!keep) {
+				/* mark the entry for deregistration */
+				vmp->flag = LPFC_VMID_DE_REGISTER;
+				write_unlock(&vport->vmid_lock);
+				if (vport->vmid_priority_tagging)
+					r = lpfc_vmid_uvem(vport, vmp, false);
+				else
+					r = lpfc_vmid_cmd(vport,
+							  SLI_CTAS_DAPP_IDENT,
+							  vmp);
+
+				/* decrement number of active vms and mark */
+				/* entry in slot as free */
+				write_lock(&vport->vmid_lock);
+				if (!r) {
+					struct lpfc_vmid *ht = vmp;
+
+					vport->cur_vmid_cnt--;
+					ht->flag = LPFC_VMID_SLOT_FREE;
+					free_percpu(ht->last_io_time);
+					ht->last_io_time = NULL;
+					hash_del(&ht->hnode);
+				}
+			}
+		}
+	}
+ out:
+	write_unlock(&vport->vmid_lock);
+}
+
+/**
+ * lpfc_check_inactive_vmid - VMID inactivity checker
+ * @phba: Pointer to hba context object.
+ *
+ * This function is called from the worker thread to determine if an entry in
+ * the vmid table can be released since there was no IO activity seen from that
+ * particular VM for the specified time. When this happens, the entry in the
+ * table is released and also the resources on the switch cleared.
+ **/
+
+static void lpfc_check_inactive_vmid(struct lpfc_hba *phba)
+{
+	struct lpfc_vport *vport;
+	struct lpfc_vport **vports;
+	int i;
+
+	vports = lpfc_create_vport_work_array(phba);
+	if (!vports)
+		return;
+
+	for (i = 0; i <= phba->max_vports; i++) {
+		if ((!vports[i]) && (i == 0))
+			vport = phba->pport;
+		else
+			vport = vports[i];
+		if (!vport)
+			break;
+
+		lpfc_check_inactive_vmid_one(vport);
+	}
+	lpfc_destroy_vport_work_array(phba, vports);
+}
+
 /**
  * lpfc_dev_loss_tmo_handler - Remote node devloss timeout handler
  * @ndlp: Pointer to remote node object.
diff --git a/drivers/scsi/lpfc/lpfc_init.c b/drivers/scsi/lpfc/lpfc_init.c
index d6434e4ffa20..c82b10f70361 100644
--- a/drivers/scsi/lpfc/lpfc_init.c
+++ b/drivers/scsi/lpfc/lpfc_init.c
@@ -4845,6 +4845,42 @@ lpfc_sli4_fcf_redisc_wait_tmo(struct timer_list *t)
 	lpfc_worker_wake_up(phba);
 }
 
+/**
+ * lpfc_vmid_poll - VMID timeout detection
+ * @ptr: Map to lpfc_hba data structure pointer.
+ *
+ * This routine is invoked when there is no IO on by a VM for the specified
+ * amount of time. When this situation is detected, the VMID has to be
+ * deregistered from the switch and all the local resources freed. The VMID
+ * will be reassigned to the VM once the IO begins.
+ **/
+static void
+lpfc_vmid_poll(struct timer_list *t)
+{
+	struct lpfc_hba *phba = from_timer(phba, t, inactive_vmid_poll);
+	u32 wake_up = 0;
+
+	/* check if there is a need to issue QFPA */
+	if (phba->pport->vmid_priority_tagging) {
+		wake_up = 1;
+		phba->pport->work_port_events |= WORKER_CHECK_VMID_ISSUE_QFPA;
+	}
+
+	/* Is the vmid inactivity timer enabled */
+	if (phba->pport->vmid_inactivity_timeout ||
+	    phba->pport->load_flag & FC_DEREGISTER_ALL_APP_ID) {
+		wake_up = 1;
+		phba->pport->work_port_events |= WORKER_CHECK_INACTIVE_VMID;
+	}
+
+	if (wake_up)
+		lpfc_worker_wake_up(phba);
+
+	/* restart the timer for the next iteration */
+	mod_timer(&phba->inactive_vmid_poll, jiffies + msecs_to_jiffies(1000 *
+							LPFC_VMID_TIMER));
+}
+
 /**
  * lpfc_sli4_parse_latt_fault - Parse sli4 link-attention link fault code
  * @phba: pointer to lpfc hba data structure.
@@ -6693,6 +6729,10 @@ lpfc_sli4_driver_resource_setup(struct lpfc_hba *phba)
 	phba->hbqs[LPFC_ELS_HBQ].hbq_alloc_buffer = lpfc_sli4_rb_alloc;
 	phba->hbqs[LPFC_ELS_HBQ].hbq_free_buffer = lpfc_sli4_rb_free;
 
+	/* for VMID idle timeout if VMID is enabled */
+	if (lpfc_is_vmid_enabled(phba))
+		timer_setup(&phba->inactive_vmid_poll, lpfc_vmid_poll, 0);
+
 	/*
 	 * Initialize the SLI Layer to run with lpfc SLI4 HBAs.
 	 */
-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:55 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:55 +0530
Subject: [PATCH v11 12/13] lpfc: vmid: Adding qfpa and vmid timeout check in
 worker thread
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-13-muneendra.kumar@broadcom.com>

From: Gaurav Srivastava <gaurav.srivastava at broadcom.com>

This patch add the periodic check for issuing of qfpa command and vmid
timeout in the worker thread. The inactivity timeout check is added via
the timer function.

Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Gaurav Srivastava  <gaurav.srivastava at broadcom.com>
Signed-off-by: James Smart <jsmart2021 at gmail.com>

---
v11:
No change

v10:
No change

v9:
No change

v8:
checked the function return value

v7:
No change

v6:
Added Forward declarations and removed unused variable

v5:
No change

v4:
No change

v3:
No change

v2:
Ported the patch on top of 5.10/scsi-queue
---
 drivers/scsi/lpfc/lpfc_hbadisc.c | 43 ++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/drivers/scsi/lpfc/lpfc_hbadisc.c b/drivers/scsi/lpfc/lpfc_hbadisc.c
index 41a388aab1cf..8ffd2727f47f 100644
--- a/drivers/scsi/lpfc/lpfc_hbadisc.c
+++ b/drivers/scsi/lpfc/lpfc_hbadisc.c
@@ -73,6 +73,7 @@ static void lpfc_unregister_fcfi_cmpl(struct lpfc_hba *, LPFC_MBOXQ_t *);
 static int lpfc_fcf_inuse(struct lpfc_hba *);
 static void lpfc_mbx_cmpl_read_sparam(struct lpfc_hba *, LPFC_MBOXQ_t *);
 static void lpfc_check_inactive_vmid(struct lpfc_hba *phba);
+static void lpfc_check_vmid_qfpa_issue(struct lpfc_hba *phba);
 
 static int
 lpfc_valid_xpt_node(struct lpfc_nodelist *ndlp)
@@ -428,6 +429,32 @@ lpfc_dev_loss_tmo_handler(struct lpfc_nodelist *ndlp)
 	return fcf_inuse;
 }
 
+static void lpfc_check_vmid_qfpa_issue(struct lpfc_hba *phba)
+{
+	struct lpfc_vport *vport;
+	struct lpfc_vport **vports;
+	int i;
+
+	vports = lpfc_create_vport_work_array(phba);
+	if (!vports)
+		return;
+
+	for (i = 0; i <= phba->max_vports; i++) {
+		if ((!vports[i]) && (i == 0))
+			vport = phba->pport;
+		else
+			vport = vports[i];
+		if (!vport)
+			break;
+
+		if (vport->vmid_flag & LPFC_VMID_ISSUE_QFPA) {
+			if (!lpfc_issue_els_qfpa(vport))
+				vport->vmid_flag &= ~LPFC_VMID_ISSUE_QFPA;
+		}
+	}
+	lpfc_destroy_vport_work_array(phba, vports);
+}
+
 /**
  * lpfc_sli4_post_dev_loss_tmo_handler - SLI4 post devloss timeout handler
  * @phba: Pointer to hba context object.
@@ -748,6 +775,22 @@ lpfc_work_done(struct lpfc_hba *phba)
 	if (ha_copy & HA_LATT)
 		lpfc_handle_latt(phba);
 
+	/* Handle VMID Events */
+	if (lpfc_is_vmid_enabled(phba)) {
+		if (phba->pport->work_port_events &
+		    WORKER_CHECK_VMID_ISSUE_QFPA) {
+			lpfc_check_vmid_qfpa_issue(phba);
+			phba->pport->work_port_events &=
+				~WORKER_CHECK_VMID_ISSUE_QFPA;
+		}
+		if (phba->pport->work_port_events &
+		    WORKER_CHECK_INACTIVE_VMID) {
+			lpfc_check_inactive_vmid(phba);
+			phba->pport->work_port_events &=
+			    ~WORKER_CHECK_INACTIVE_VMID;
+		}
+	}
+
 	/* Process SLI4 events */
 	if (phba->pci_dev_grp == LPFC_PCI_DEV_OC) {
 		if (phba->hba_flag & HBA_RRQ_ACTIVE)
-- 
2.26.2



From muneendra.kumar at broadcom.com  Mon Jun  7 21:35:56 2021
From: muneendra.kumar at broadcom.com (Muneendra Kumar)
Date: Tue,  8 Jun 2021 10:05:56 +0530
Subject: [PATCH v11 13/13] lpfc: vmid: Introducing vmid in io path.
In-Reply-To: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
Message-ID: <20210608043556.274139-14-muneendra.kumar@broadcom.com>

From: Gaurav Srivastava <gaurav.srivastava at broadcom.com>

The patch introduces the vmid in the io path. It checks if the vmid is
enabled and if io belongs to a vm or not and acts accordingly. Other
supporing APIs are also included in the patch.

Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Gaurav Srivastava  <gaurav.srivastava at broadcom.com>
Signed-off-by: James Smart <jsmart2021 at gmail.com>

---
v11:
No change

v10:
Remove redundant code and function name correction

v9:
Added changes related to locking and new hashtable implementation

v8:
Added proper error codes
updated logic while handling vmid

v7:
No change

v6:
No change

v5:
No change

v4:
No change

v3:
Replaced blkcg_get_app_identifier with blkcg_get_fc_appid

v2:
Ported the patch on top of 5.10/scsi-queue
Added a fix for issuing QFPA command which was not included in the
last submit
---
 drivers/scsi/lpfc/lpfc_scsi.c | 174 ++++++++++++++++++++++++++++++++++
 1 file changed, 174 insertions(+)

diff --git a/drivers/scsi/lpfc/lpfc_scsi.c b/drivers/scsi/lpfc/lpfc_scsi.c
index 65637d85f9bb..1cf271bee126 100644
--- a/drivers/scsi/lpfc/lpfc_scsi.c
+++ b/drivers/scsi/lpfc/lpfc_scsi.c
@@ -5296,6 +5296,160 @@ static void lpfc_vmid_assign_cs_ctl(struct lpfc_vport *vport,
 	}
 }
 
+/*
+ * lpfc_vmid_get_appid- get the vmid associated with the uuid
+ * @vport: The virtual port for which this call is being executed.
+ * @uuid: uuid associated with the VE
+ * @cmd: address of scsi cmmd descriptor
+ * @tag: VMID tag
+ * Returns status of the function
+ */
+static int lpfc_vmid_get_appid(struct lpfc_vport *vport, char *uuid, struct
+			       scsi_cmnd * cmd, union lpfc_vmid_io_tag *tag)
+{
+	struct lpfc_vmid *vmp = NULL;
+	int hash, len, rc, i;
+
+	/* check if QFPA is complete */
+	if (lpfc_vmid_is_type_priority_tag(vport) && !(vport->vmid_flag &
+	      LPFC_VMID_QFPA_CMPL)) {
+		vport->work_port_events |= WORKER_CHECK_VMID_ISSUE_QFPA;
+		return -EAGAIN;
+	}
+
+	/* search if the uuid has already been mapped to the vmid */
+	len = strlen(uuid);
+	hash = lpfc_vmid_hash_fn(uuid, len);
+
+	/* search for the VMID in the table */
+	read_lock(&vport->vmid_lock);
+	vmp = lpfc_get_vmid_from_hashtable(vport, hash, uuid);
+
+	/* if found, check if its already registered  */
+	if (vmp  && vmp->flag & LPFC_VMID_REGISTERED) {
+		read_unlock(&vport->vmid_lock);
+		lpfc_vmid_update_entry(vport, cmd, vmp, tag);
+		rc = 0;
+	} else if (vmp && (vmp->flag & LPFC_VMID_REQ_REGISTER ||
+			   vmp->flag & LPFC_VMID_DE_REGISTER)) {
+		/* else if register or dereg request has already been sent */
+		/* Hence vmid tag will not be added for this IO */
+		read_unlock(&vport->vmid_lock);
+		rc = -EBUSY;
+	} else {
+		/* The vmid was not found in the hashtable. At this point, */
+		/* drop the read lock first before proceeding further */
+		read_unlock(&vport->vmid_lock);
+		/* start the process to obtain one as per the */
+		/* type of the vmid indicated */
+		write_lock(&vport->vmid_lock);
+		vmp = lpfc_get_vmid_from_hashtable(vport, hash, uuid);
+
+		/* while the read lock was released, in case the entry was */
+		/* added by other context or is in process of being added */
+		if (vmp && vmp->flag & LPFC_VMID_REGISTERED) {
+			lpfc_vmid_update_entry(vport, cmd, vmp, tag);
+			write_unlock(&vport->vmid_lock);
+			return 0;
+		} else if (vmp && vmp->flag & LPFC_VMID_REQ_REGISTER) {
+			write_unlock(&vport->vmid_lock);
+			return -EBUSY;
+		}
+
+		/* else search and allocate a free slot in the hash table */
+		if (vport->cur_vmid_cnt < vport->max_vmid) {
+			for (i = 0; i < vport->max_vmid; i++) {
+				vmp = vport->vmid + i;
+				if (vmp->flag == LPFC_VMID_SLOT_FREE)
+					break;
+			}
+			if (i == vport->max_vmid)
+				vmp = NULL;
+		} else {
+			vmp = NULL;
+		}
+
+		if (!vmp) {
+			write_unlock(&vport->vmid_lock);
+			return -ENOMEM;
+		}
+
+		/* Add the vmid and register  */
+		lpfc_put_vmid_in_hashtable(vport, hash, vmp);
+		vmp->vmid_len = len;
+		memcpy(vmp->host_vmid, uuid, vmp->vmid_len);
+		vmp->io_rd_cnt = 0;
+		vmp->io_wr_cnt = 0;
+		vmp->flag = LPFC_VMID_SLOT_USED;
+
+		vmp->delete_inactive =
+			vport->vmid_inactivity_timeout ? 1 : 0;
+
+		/* if type priority tag, get next available vmid */
+		if (lpfc_vmid_is_type_priority_tag(vport))
+			lpfc_vmid_assign_cs_ctl(vport, vmp);
+
+		/* allocate the per cpu variable for holding */
+		/* the last access time stamp only if vmid is enabled */
+		if (!vmp->last_io_time)
+			vmp->last_io_time = __alloc_percpu(sizeof(u64),
+							   __alignof__(struct
+							   lpfc_vmid));
+		if (!vmp->last_io_time) {
+			hash_del(&vmp->hnode);
+			vmp->flag = LPFC_VMID_SLOT_FREE;
+			write_unlock(&vport->vmid_lock);
+			return -EIO;
+		}
+
+		write_unlock(&vport->vmid_lock);
+
+		/* complete transaction with switch */
+		if (lpfc_vmid_is_type_priority_tag(vport))
+			rc = lpfc_vmid_uvem(vport, vmp, true);
+		else
+			rc = lpfc_vmid_cmd(vport, SLI_CTAS_RAPP_IDENT, vmp);
+		if (!rc) {
+			write_lock(&vport->vmid_lock);
+			vport->cur_vmid_cnt++;
+			vmp->flag |= LPFC_VMID_REQ_REGISTER;
+			write_unlock(&vport->vmid_lock);
+		} else {
+			write_lock(&vport->vmid_lock);
+			hash_del(&vmp->hnode);
+			vmp->flag = LPFC_VMID_SLOT_FREE;
+			free_percpu(vmp->last_io_time);
+			write_unlock(&vport->vmid_lock);
+			return -EIO;
+		}
+
+		/* finally, enable the idle timer once */
+		if (!(vport->phba->pport->vmid_flag & LPFC_VMID_TIMER_ENBLD)) {
+			mod_timer(&vport->phba->inactive_vmid_poll,
+				  jiffies +
+				  msecs_to_jiffies(1000 * LPFC_VMID_TIMER));
+			vport->phba->pport->vmid_flag |= LPFC_VMID_TIMER_ENBLD;
+		}
+	}
+	return rc;
+}
+
+/*
+ * lpfc_is_command_vm_io - get the uuid from blk cgroup
+ * @cmd:Pointer to scsi_cmnd data structure
+ * Returns uuid if present if not null
+ */
+static char *lpfc_is_command_vm_io(struct scsi_cmnd *cmd)
+{
+	char *uuid = NULL;
+
+	if (cmd->request) {
+		if (cmd->request->bio)
+			uuid = blkcg_get_fc_appid(cmd->request->bio);
+	}
+	return uuid;
+}
+
 /**
  * lpfc_queuecommand - scsi_host_template queuecommand entry point
  * @shost: kernel scsi host pointer.
@@ -5321,6 +5475,7 @@ lpfc_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *cmnd)
 	int err, idx;
 #ifdef CONFIG_SCSI_LPFC_DEBUG_FS
 	uint64_t start = 0L;
+	u8 *uuid = NULL;
 
 	if (phba->ktime_on)
 		start = ktime_get_ns();
@@ -5448,6 +5603,25 @@ lpfc_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *cmnd)
 	}
 
 
+	/* check the necessary and sufficient condition to support VMID */
+	if (lpfc_is_vmid_enabled(phba) &&
+	    (ndlp->vmid_support ||
+	     phba->pport->vmid_priority_tagging ==
+	     LPFC_VMID_PRIO_TAG_ALL_TARGETS)) {
+		/* is the IO generated by a VM, get the associated virtual */
+		/* entity id */
+		uuid = lpfc_is_command_vm_io(cmnd);
+
+		if (uuid) {
+			err = lpfc_vmid_get_appid(vport, uuid, cmnd,
+				(union lpfc_vmid_io_tag *)
+					&lpfc_cmd->cur_iocbq.vmid_tag);
+			if (!err)
+				lpfc_cmd->cur_iocbq.iocb_flag |= LPFC_IO_VMID;
+		}
+	}
+
+	atomic_inc(&ndlp->cmd_pending);
 #ifdef CONFIG_SCSI_LPFC_DEBUG_FS
 	if (unlikely(phba->hdwqstat_on & LPFC_CHECK_SCSI_IO))
 		this_cpu_inc(phba->sli4_hba.c_stat->xmt_io);
-- 
2.26.2



From mgurtovoy at nvidia.com  Tue Jun  8 04:36:33 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Tue, 8 Jun 2021 14:36:33 +0300
Subject: [PATCH v2 2/4] nvmet: make sn stable once connection was
 established
In-Reply-To: <BYAPR04MB4965C4BD68DD41755F8CA15386389@BYAPR04MB4965.namprd04.prod.outlook.com>
References: <20210607092324.1031046-1-mgurtovoy@nvidia.com>
 <20210607092324.1031046-2-mgurtovoy@nvidia.com>
 <BYAPR04MB4965C4BD68DD41755F8CA15386389@BYAPR04MB4965.namprd04.prod.outlook.com>
Message-ID: <39a2ef7e-d511-8237-c298-cf32763a4880@nvidia.com>


On 6/7/2021 11:37 PM, Chaitanya Kulkarni wrote:
> Max,
>
> On 6/7/21 02:23, Max Gurtovoy wrote:
>> diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
>> index 0ae809ca428c..bd0a0b91d843 100644
>> --- a/drivers/nvme/target/nvmet.h
>> +++ b/drivers/nvme/target/nvmet.h
>> @@ -231,6 +231,7 @@ struct nvmet_subsys {
>>   
>>   	u64			ver;
>>   	char			serial[NVMET_SN_MAX_SIZE];
>> +	bool			subsys_discovered;
> When subsystem is discovered (connected to the host) we will have non empty
> controller list, instead of adding a new bool variable why not use
> something like following tested patch on the top of this one ?

No it's not the same. The controller list can be empty if initiator_A 
performed connect + disconnect.

In your solution, it is allowed to change the sn after the disconnection 
but in the original solution we don't allow this.

Once *any* initiator connected to this subsystem, the serial nmber will 
not be changed anymore.


> (that also fixes the indentation of
> nvmet_subsys_attr_serial_store_locked()).
>
> diff --git a/drivers/nvme/target/admin-cmd.c
> b/drivers/nvme/target/admin-cmd.c
> index bec84c231d37..897f419be618 100644
> --- a/drivers/nvme/target/admin-cmd.c
> +++ b/drivers/nvme/target/admin-cmd.c
> @@ -366,12 +366,6 @@ static void nvmet_execute_identify_ctrl(struct
> nvmet_req *req)
>          u32 cmd_capsule_size;
>          u16 status = 0;
>   
> -       if (!subsys->subsys_discovered) {
> -               mutex_lock(&subsys->lock);
> -               subsys->subsys_discovered = true;
> -               mutex_unlock(&subsys->lock);
> -       }
> -
>          /*
>           * If there is no model number yet, set it now.  It will then remain
>           * stable for the life time of the subsystem.
> diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
> index a13da86fb374..94e23dca763f 100644
> --- a/drivers/nvme/target/configfs.c
> +++ b/drivers/nvme/target/configfs.c
> @@ -1046,11 +1046,11 @@ static ssize_t
> nvmet_subsys_attr_serial_show(struct config_item *item,
>   
>   static ssize_t
>   nvmet_subsys_attr_serial_store_locked(struct nvmet_subsys *subsys,
> -               const char *page, size_t count)
> +                                     const char *page, size_t count)
>   {
>          int pos, len = strcspn(page, "\n");
>   
> -       if (subsys->subsys_discovered) {
> +       if (!list_empty(&subsys->ctrls)) {
>                  pr_err("Can't set serial number. %s is already assigned\n",
>                         subsys->serial);
>                  return -EINVAL;
> diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
> index f7fbf9690ce8..73c6c4dc8997 100644
> --- a/drivers/nvme/target/nvmet.h
> +++ b/drivers/nvme/target/nvmet.h
> @@ -232,7 +232,6 @@ struct nvmet_subsys {
>   
>          u64                     ver;
>          char                    serial[NVMET_SN_MAX_SIZE];
> -       bool                    subsys_discovered;
>          char                    *subsysnqn;
>          bool                    pi_support;
>   
>
> # nvme list
> Node                  SN
> Model                                    Namespace
> Usage                      Format           FW Rev
> --------------------- --------------------
> ---------------------------------------- ---------
> -------------------------- ---------------- --------
> /dev/nvme0n1          foo                  QEMU NVMe
> Ctrl                           1           1.07  GB /   1.07  GB
> 512   B +  0 B   1.0
> /dev/nvme1n1          bf854da611a264d448c2
> Linux                                    1           2.15  GB /   2.15
> GB      4 KiB +  0 B   5.13.0-r
> # echo 1 > /sys/kernel/config/nvmet/subsystems/nqn/
> allowed_hosts/       attr_cntlid_min      attr_serial          passthru/
> attr_allow_any_host  attr_model           attr_version
> attr_cntlid_max      attr_pi_enable       namespaces/
>   # echo "1234567890" > /sys/kernel/config/nvmet/subsystems/nqn/attr_serial
> bash: echo: write error: Invalid argument
> # dmesg  -c
> [311081.155148] nvmet: Can't set serial number. bf854da611a264d448c2 is
> already assigned
>
>


From hch at lst.de  Tue Jun  8 05:19:25 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 14:19:25 +0200
Subject: [PATCH] nvme: NVME_TCP_OFFLOAD should not default to m
In-Reply-To: <39b1a3684880e1d85ef76e34403886e8f1d22508.1623149635.git.geert+renesas@glider.be>
References: <39b1a3684880e1d85ef76e34403886e8f1d22508.1623149635.git.geert+renesas@glider.be>
Message-ID: <20210608121925.GA24201@lst.de>

On Tue, Jun 08, 2021 at 12:56:09PM +0200, Geert Uytterhoeven wrote:
> The help text for the symbol controlling support for the NVM Express
> over Fabrics TCP offload common layer suggests to not enable this
> support when unsure.
> 
> Hence drop the "default m", which actually means "default y" if
> CONFIG_MODULES is not enabled.
> 
> Fixes: f0e8cb6106da2703 ("nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW offload ULP")
> Signed-off-by: Geert Uytterhoeven <geert+renesas at glider.be>

Err, where did this appear from?  This code has not been accepted into
the NVMe tree and is indeed not acceptable in this form.


From wubo40 at huawei.com  Tue Jun  8 05:30:14 2021
From: wubo40 at huawei.com (Wu Bo)
Date: Tue, 8 Jun 2021 20:30:14 +0800
Subject: [PATCH] nvme-multipath: combine grpid and ANA state checks in
 nvme_parse_ana_log()
In-Reply-To: <20210608052320.GA13828@lst.de>
References: <1623125616-629270-1-git-send-email-wubo40@huawei.com>
 <20210608052320.GA13828@lst.de>
Message-ID: <3a243d2c-f9b9-edcf-1762-9b08cf21f152@huawei.com>

On 2021/6/8 13:23, Christoph Hellwig wrote:
> On Tue, Jun 08, 2021 at 12:13:36PM +0800, Wu Bo wrote:
>> -		if (WARN_ON_ONCE(desc->grpid == 0))
>> +		if (WARN_ON_ONCE(desc->grpid == 0 ||
>> +			le32_to_cpu(desc->grpid) > ctrl->anagrpmax))
>>   			return -EINVAL;
>> -		if (WARN_ON_ONCE(le32_to_cpu(desc->grpid) > ctrl->anagrpmax))
>> -			return -EINVAL;
>> -		if (WARN_ON_ONCE(desc->state == 0))
>> -			return -EINVAL;
>> -		if (WARN_ON_ONCE(desc->state > NVME_ANA_CHANGE))
>> +		if (WARN_ON_ONCE(desc->state == 0 ||
>> +			desc->state > NVME_ANA_CHANGE))
> 
> So besides making the code impossibl to read due to the incorrect
> indentation this also makes each WARN_ON_ONCE cover multiple conditions.
> Not very useful for debugging.
> Indeed, not very useful for debugging, please ignore this patch.

Thanks,
Wu Bo
> 



From wubo40 at huawei.com  Tue Jun  8 05:32:15 2021
From: wubo40 at huawei.com (Wu Bo)
Date: Tue, 8 Jun 2021 20:32:15 +0800
Subject: [PATCH] nvme-multipath: combine grpid and ANA state checks in
 nvme_parse_ana_log()
In-Reply-To: <20210608052320.GA13828@lst.de>
References: <1623125616-629270-1-git-send-email-wubo40@huawei.com>
 <20210608052320.GA13828@lst.de>
Message-ID: <0f672080-671b-c1b8-18d5-3ddad90f53c4@huawei.com>

On 2021/6/8 13:23, Christoph Hellwig wrote:
> On Tue, Jun 08, 2021 at 12:13:36PM +0800, Wu Bo wrote:
>> -		if (WARN_ON_ONCE(desc->grpid == 0))
>> +		if (WARN_ON_ONCE(desc->grpid == 0 ||
>> +			le32_to_cpu(desc->grpid) > ctrl->anagrpmax))
>>   			return -EINVAL;
>> -		if (WARN_ON_ONCE(le32_to_cpu(desc->grpid) > ctrl->anagrpmax))
>> -			return -EINVAL;
>> -		if (WARN_ON_ONCE(desc->state == 0))
>> -			return -EINVAL;
>> -		if (WARN_ON_ONCE(desc->state > NVME_ANA_CHANGE))
>> +		if (WARN_ON_ONCE(desc->state == 0 ||
>> +			desc->state > NVME_ANA_CHANGE))
> 
> So besides making the code impossibl to read due to the incorrect
> indentation this also makes each WARN_ON_ONCE cover multiple conditions.
> Not very useful for debugging.
> .
> 

Indeed, not very useful for debugging, please ignore this patch.

Thanks,
Wu Bo



From hare at suse.de  Tue Jun  8 06:09:32 2021
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 8 Jun 2021 15:09:32 +0200
Subject: [PATCH v11 08/13] lpfc: vmid: Functions to manage vmids
In-Reply-To: <20210608043556.274139-9-muneendra.kumar@broadcom.com>
References: <20210608043556.274139-1-muneendra.kumar@broadcom.com>
 <20210608043556.274139-9-muneendra.kumar@broadcom.com>
Message-ID: <6f3e5eb9-22de-ba43-9591-1f3610172738@suse.de>

On 6/8/21 6:35 AM, Muneendra Kumar wrote:
> From: Gaurav Srivastava <gaurav.srivastava at broadcom.com>
> 
> This patch contains the routines to save, retrieve and remove the vmids
> from the data structure. A hash table is used to save the vmids and
> the corresponding UUIDs associated with the application/VMs.
> 
> Signed-off-by: Gaurav Srivastava  <gaurav.srivastava at broadcom.com>
> Signed-off-by: James Smart <jsmart2021 at gmail.com>
> 
Reviewed-by: Hannes Reinecke <hare at suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare at suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), GF: Felix Imend?rffer


From geert at linux-m68k.org  Tue Jun  8 06:37:47 2021
From: geert at linux-m68k.org (Geert Uytterhoeven)
Date: Tue, 8 Jun 2021 15:37:47 +0200
Subject: [PATCH] nvme: NVME_TCP_OFFLOAD should not default to m
In-Reply-To: <20210608121925.GA24201@lst.de>
References: <39b1a3684880e1d85ef76e34403886e8f1d22508.1623149635.git.geert+renesas@glider.be>
 <20210608121925.GA24201@lst.de>
Message-ID: <CAMuHMdWgLf8GJfOaRUyx=AvOTnuOs8FS-2=C+OCk02OLDCyrDg@mail.gmail.com>

Hi Christoph,

On Tue, Jun 8, 2021 at 2:19 PM Christoph Hellwig <hch at lst.de> wrote:
> On Tue, Jun 08, 2021 at 12:56:09PM +0200, Geert Uytterhoeven wrote:
> > The help text for the symbol controlling support for the NVM Express
> > over Fabrics TCP offload common layer suggests to not enable this
> > support when unsure.
> >
> > Hence drop the "default m", which actually means "default y" if
> > CONFIG_MODULES is not enabled.
> >
> > Fixes: f0e8cb6106da2703 ("nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW offload ULP")
> > Signed-off-by: Geert Uytterhoeven <geert+renesas at glider.be>
>
> Err, where did this appear from?  This code has not been accepted into
> the NVMe tree and is indeed not acceptable in this form.

It was applied to net-next.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds


From mario.limonciello at amd.com  Tue Jun  8 07:18:53 2021
From: mario.limonciello at amd.com (Limonciello, Mario)
Date: Tue, 8 Jun 2021 09:18:53 -0500
Subject: [PATCH v6 1/2] ACPI: Move check for _DSD StorageD3Enable property
 to acpi
In-Reply-To: <CAJZ5v0gMgwJ9kq_mo_dYoTUAqQzhXmrgp9wnY4TRPU1pJp=TyQ@mail.gmail.com>
References: <20210607173156.5548-1-mario.limonciello@amd.com>
 <20210608053546.GA14116@lst.de>
 <CAJZ5v0gMgwJ9kq_mo_dYoTUAqQzhXmrgp9wnY4TRPU1pJp=TyQ@mail.gmail.com>
Message-ID: <556a22c7-58af-b9d4-d50c-25d7d16bd344@amd.com>

On 6/8/2021 06:20, Rafael J. Wysocki wrote:
> On Tue, Jun 8, 2021 at 7:35 AM Christoph Hellwig <hch at lst.de> wrote:
>>
>> On Mon, Jun 07, 2021 at 12:31:55PM -0500, Mario Limonciello wrote:
>>> +/**
>>> + * acpi_storage_d3 - Check if a storage device should use D3.
> 
> Let's be specific about what D3 means here in the first place and
> that's D3hot AFAICS.
> 
> And the comment should be something like "Check whether or not to use
> D3hot in the suspend path".

Actually it can be D3hot or D3cold.  Microsoft's documentation doesn't 
indicate it's D3hot.  On the AMD platforms that prompted some of these 
changes it's D3cold.

> 
>>> + * @dev: Device to check
>>> + *
>>> + * Returns %true if @dev should be put into D3 when the ->suspend method is
>>> + * called, else %false.  The name of this function is somewhat misleading
>>> + * as it has nothing to do with storage except for the name of the ACPI
>>> + * property.  On some platforms resume will not work if this hint is ignored.
> 
> I would write it this way:
> 
> "Return %true if the platform firmware wants @dev to be programmed
> into D3hot in the suspend path, or %false when there is no specific
> preference. On some platforms, if this hint is ignored, @dev may
> remain unresponsive after suspending the platform as a whole."
> 
> And I'm not sure if it is necessary to mention "storage" in this comment at all.
> 

Is your thought here in not mentioning "storage" that this symbol may be 
overloaded in the future to look at more than just the StorageD3Enable 
property and used for other things too?

>>> + *
>>> + */
>>
>> I still do not like this comment.  There is nothing about only using this
>> for storage devices.  It is specific to a PCIe slot, and if I plug
>> something that is not a storage device into it the same restrictions
>> still apply.
> 
> Originally, it was about storage devices IIUC (and that's why the
> property name is what it is).
>  > IIRC PCIe link management was broken in some NVMe drives and putting
> them into D3hot would allow the link to go down later and it would
> never go up again.  So you-know-who decided to avoid using D3hot for
> all NVMe drives and they were generally characterized as "storage".
> Alas, that turned out to break other things depending on the devices
> being put into D3hot (like PS_ON) so the property was invented to
> allow OEMs to mark the configurations including the devices that
> actually worked (and Linux was/is on the receiving end). And
> originally the property applied to the endpoint device under the root
> port it was present for (my personal opinion about this doesn't really
> matter, so let me avoid wasting time on expressing it).
> 
> Now, it evidently has been re-purposed to mark PCIe root ports that
> require devices under them to be put into D3hot in the suspend path
> (and it now means any device under the root port with this property,
> at least in some cases).
> 
>> Also no need for the empty-ish line at the end.
> 
> Right.
> 



From weiyongjun1 at huawei.com  Tue Jun  8 07:38:54 2021
From: weiyongjun1 at huawei.com (Wei Yongjun)
Date: Tue, 8 Jun 2021 14:38:54 +0000
Subject: [PATCH -next] nvme-tcp-offload: Fix error return code in
 nvme_tcp_ofld_setup_ctrl()
Message-ID: <20210608143854.4154005-1-weiyongjun1@huawei.com>

Fix to return negative error code -EINVAL if mandatory sgls
are not supported.

Fixes: 5aadd5f9311e ("nvme-tcp-offload: Add controller level implementation")
Reported-by: Hulk Robot <hulkci at huawei.com>
Signed-off-by: Wei Yongjun <weiyongjun1 at huawei.com>
---
 drivers/nvme/host/tcp-offload.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index c76822e5ada7..3879d5ed7497 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -622,6 +622,7 @@ static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 
 	if (!(nctrl->sgls & ((1 << 0) | (1 << 1)))) {
 		dev_err(nctrl->device, "Mandatory sgls are not supported!\n");
+		rc = -EINVAL;
 		goto destroy_admin;
 	}
 



From hch at infradead.org  Tue Jun  8 07:31:04 2021
From: hch at infradead.org (Christoph Hellwig)
Date: Tue, 8 Jun 2021 15:31:04 +0100
Subject: [GIT PULL] first round of nvme updates for Linux 5.14
Message-ID: <YL9/KDQ2uGRVAh6F@infradead.org>


The following changes since commit 8184035805dc87dd826101b930d3dce97758f7b1:

  rsxx: Use struct_size() in vmalloc() (2021-05-24 06:47:30 -0600)

are available in the Git repository at:

  git://git.infradead.org/nvme.git tags/nvme-5.14-2021-06-08

for you to fetch changes up to 346ac785badf66120d8b4c7b48f87b0a536f691e:

  nvmet: remove a superfluous variable (2021-06-03 10:29:26 +0300)

----------------------------------------------------------------
nvme updates for Linux 5.14

 - improve the APST configuration algorithm (Alexey Bogoslavsky)
 - look for StorageD3Enable on companion ACPI device (Mario Limonciello)
 - allow selecting the network interface for TCP connections
   (Martin Belanger)
 - misc cleanups (Amit Engel, Chaitanya Kulkarni, Colin Ian King, me)

----------------------------------------------------------------
Alexey Bogoslavsky (1):
      nvme: extend and modify the APST configuration algorithm

Amit Engel (1):
      nvmet: move ka_work initialization to nvmet_alloc_ctrl

Chaitanya Kulkarni (5):
      nvme-fabrics: fix the kerneldco comment for nvmf_log_connect_error()
      nvme-fabrics: remove extra new lines in the switch
      nvme-fabrics: remove an extra comment
      nvme-fabrics: remove extra braces
      nvmet: remove a superfluous variable

Christoph Hellwig (7):
      nvme: open code nvme_put_ns_from_disk in nvme_ns_head_chr_ioctl
      nvme: open code nvme_{get,put}_ns_from_disk in nvme_ns_head_ioctl
      nvme: open code nvme_put_ns_from_disk in nvme_ns_head_ctrl_ioctl
      nvme: add a sparse annotation to nvme_ns_head_ctrl_ioctl
      nvme: move the CSI sanity check into nvme_ns_report_zones
      nvme: split nvme_report_zones
      nvme: remove nvme_{get,put}_ns_from_disk

Colin Ian King (1):
      nvme: remove redundant initialization of variable ret

Mario Limonciello (1):
      nvme-pci: look for StorageD3Enable on companion ACPI device instead

Martin Belanger (1):
      nvme-tcp: allow selecting the network interface for connections

 drivers/nvme/host/core.c          | 173 ++++++++++++++++++++++++++------------
 drivers/nvme/host/fabrics.c       |  49 ++++++-----
 drivers/nvme/host/fabrics.h       |   6 +-
 drivers/nvme/host/ioctl.c         |  35 ++++----
 drivers/nvme/host/multipath.c     |  21 ++++-
 drivers/nvme/host/nvme.h          |  12 +--
 drivers/nvme/host/pci.c           |  24 +-----
 drivers/nvme/host/rdma.c          |   2 +-
 drivers/nvme/host/tcp.c           |  27 +++++-
 drivers/nvme/host/zns.c           |  27 ++----
 drivers/nvme/target/core.c        |   2 +-
 drivers/nvme/target/io-cmd-bdev.c |   3 +-
 12 files changed, 230 insertions(+), 151 deletions(-)


From rafael at kernel.org  Tue Jun  8 08:27:32 2021
From: rafael at kernel.org (Rafael J. Wysocki)
Date: Tue, 8 Jun 2021 17:27:32 +0200
Subject: [PATCH v6 1/2] ACPI: Move check for _DSD StorageD3Enable property
 to acpi
In-Reply-To: <556a22c7-58af-b9d4-d50c-25d7d16bd344@amd.com>
References: <20210607173156.5548-1-mario.limonciello@amd.com>
 <20210608053546.GA14116@lst.de>
 <CAJZ5v0gMgwJ9kq_mo_dYoTUAqQzhXmrgp9wnY4TRPU1pJp=TyQ@mail.gmail.com>
 <556a22c7-58af-b9d4-d50c-25d7d16bd344@amd.com>
Message-ID: <CAJZ5v0jwjSruQ1OYNyuJ=3UJA5LJW1-fGSw3Jy8jhRFfXasTuQ@mail.gmail.com>

On Tue, Jun 8, 2021 at 4:18 PM Limonciello, Mario
<mario.limonciello at amd.com> wrote:
>
> On 6/8/2021 06:20, Rafael J. Wysocki wrote:
> > On Tue, Jun 8, 2021 at 7:35 AM Christoph Hellwig <hch at lst.de> wrote:
> >>
> >> On Mon, Jun 07, 2021 at 12:31:55PM -0500, Mario Limonciello wrote:
> >>> +/**
> >>> + * acpi_storage_d3 - Check if a storage device should use D3.
> >
> > Let's be specific about what D3 means here in the first place and
> > that's D3hot AFAICS.
> >
> > And the comment should be something like "Check whether or not to use
> > D3hot in the suspend path".
>
> Actually it can be D3hot or D3cold.  Microsoft's documentation doesn't
> indicate it's D3hot.  On the AMD platforms that prompted some of these
> changes it's D3cold.

So say "D3" in the one-line description above and "D3hot or D3cold (if
supported)" in the more detailed comment below.

> >
> >>> + * @dev: Device to check
> >>> + *
> >>> + * Returns %true if @dev should be put into D3 when the ->suspend method is
> >>> + * called, else %false.  The name of this function is somewhat misleading
> >>> + * as it has nothing to do with storage except for the name of the ACPI
> >>> + * property.  On some platforms resume will not work if this hint is ignored.
> >
> > I would write it this way:
> >
> > "Return %true if the platform firmware wants @dev to be programmed
> > into D3hot in the suspend path, or %false when there is no specific
> > preference. On some platforms, if this hint is ignored, @dev may
> > remain unresponsive after suspending the platform as a whole."
> >
> > And I'm not sure if it is necessary to mention "storage" in this comment at all.
> >
>
> Is your thought here in not mentioning "storage" that this symbol may be
> overloaded in the future to look at more than just the StorageD3Enable
> property and used for other things too?

Well, the property itself is not about storage any more anyway.


From jsmart2021 at gmail.com  Tue Jun  8 08:39:18 2021
From: jsmart2021 at gmail.com (James Smart)
Date: Tue, 8 Jun 2021 08:39:18 -0700
Subject: [PATCH V2 2/5] nvme-fc: add a helper to check ctrl sgl support
In-Reply-To: <20210607195454.16829-3-chaitanya.kulkarni@wdc.com>
References: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
 <20210607195454.16829-3-chaitanya.kulkarni@wdc.com>
Message-ID: <4ba3579f-7223-c665-6cb0-c00f0a4c773e@gmail.com>

On 6/7/2021 12:54 PM, Chaitanya Kulkarni wrote:
> For transports it is common to check if NVMe SGLs are supported or not
> by the controller.
> 
> Add a helper instead of open coding controller SGL support and use it
> in fc transport.
> 
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> ---
>   drivers/nvme/host/fc.c   | 2 +-
>   drivers/nvme/host/nvme.h | 8 ++++++++
>   2 files changed, 9 insertions(+), 1 deletion(-)
> 

looks good

Reviewed-by: James Smart <jsmart2021 at gmail.com>

-- james



From mario.limonciello at amd.com  Tue Jun  8 08:42:54 2021
From: mario.limonciello at amd.com (Mario Limonciello)
Date: Tue,  8 Jun 2021 10:42:54 -0500
Subject: [PATCH v7 1/2] ACPI: Move check for _DSD StorageD3Enable property to
 acpi
Message-ID: <20210608154255.8693-1-mario.limonciello@amd.com>

Although first implemented for NVME, this check may be usable by
other drivers as well. Microsoft's specification explicitly mentions
that is may be usable by SATA and AHCI devices.  Google also indicates
that they have used this with SDHCI in a downstream kernel tree that
a user can plug a storage device into.

Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
Suggested-by: Keith Busch <kbusch at kernel.org>
CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
CC: Alexander Deucher <Alexander.Deucher at amd.com>
CC: Rafael J. Wysocki <rjw at rjwysocki.net>
CC: Prike Liang <prike.liang at amd.com>
Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
---
 drivers/acpi/device_pm.c | 25 +++++++++++++++++++++++++
 drivers/nvme/host/pci.c  | 28 +---------------------------
 include/linux/acpi.h     |  5 +++++
 3 files changed, 31 insertions(+), 27 deletions(-)

Changes from v4->v5:
 * Correct extra "Link:" word in commit message
Changes from v5->v6:
 * Add in commit message tags from Raul, Rafael and Keith
Changes from v6->v7:
 * Update kerneldoc comments per suggestsions.
 * Drop all Acked/Reviewed tags

diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index d260bc1f3e6e..c45f2d76b67e 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -1340,4 +1340,29 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on)
 	return 1;
 }
 EXPORT_SYMBOL_GPL(acpi_dev_pm_attach);
+
+/**
+ * acpi_storage_d3 - Check if a storage device should use D3 in the suspend path
+ * @dev: Device to check
+ *
+ * Return %true if the platform firmware wants @dev to be programmed
+ * into D3hot or D3cold (if supported) in the suspend path, or %false
+ * when there is no specific preference. On some platforms, if this
+ * hint is ignored, @dev may remain unresponsive after suspending the
+ * platform as a whole.
+ */
+bool acpi_storage_d3(struct device *dev)
+{
+	struct acpi_device *adev = ACPI_COMPANION(dev);
+	u8 val;
+
+	if (!adev)
+		return false;
+	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
+			&val))
+		return false;
+	return val == 1;
+}
+EXPORT_SYMBOL_GPL(acpi_storage_d3);
+
 #endif /* CONFIG_PM */
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..8fbc4c87a0d8 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2828,32 +2828,6 @@ static unsigned long check_vendor_combination_bug(struct pci_dev *pdev)
 	return 0;
 }
 
-#ifdef CONFIG_ACPI
-static bool nvme_acpi_storage_d3(struct pci_dev *dev)
-{
-	struct acpi_device *adev = ACPI_COMPANION(&dev->dev);
-	u8 val;
-
-	/*
-	 * Look for _DSD property specifying that the storage device on the port
-	 * must use D3 to support deep platform power savings during
-	 * suspend-to-idle.
-	 */
-
-	if (!adev)
-		return false;
-	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
-			&val))
-		return false;
-	return val == 1;
-}
-#else
-static inline bool nvme_acpi_storage_d3(struct pci_dev *dev)
-{
-	return false;
-}
-#endif /* CONFIG_ACPI */
-
 static void nvme_async_probe(void *data, async_cookie_t cookie)
 {
 	struct nvme_dev *dev = data;
@@ -2903,7 +2877,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	quirks |= check_vendor_combination_bug(pdev);
 
-	if (!noacpi && nvme_acpi_storage_d3(pdev)) {
+	if (!noacpi && acpi_storage_d3(&pdev->dev)) {
 		/*
 		 * Some systems use a bios work around to ask for D3 on
 		 * platforms that support kernel managed suspend.
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index c60745f657e9..dd0dafd21e33 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -1004,6 +1004,7 @@ int acpi_dev_resume(struct device *dev);
 int acpi_subsys_runtime_suspend(struct device *dev);
 int acpi_subsys_runtime_resume(struct device *dev);
 int acpi_dev_pm_attach(struct device *dev, bool power_on);
+bool acpi_storage_d3(struct device *dev);
 #else
 static inline int acpi_subsys_runtime_suspend(struct device *dev) { return 0; }
 static inline int acpi_subsys_runtime_resume(struct device *dev) { return 0; }
@@ -1011,6 +1012,10 @@ static inline int acpi_dev_pm_attach(struct device *dev, bool power_on)
 {
 	return 0;
 }
+static inline bool acpi_storage_d3(struct device *dev)
+{
+	return false;
+}
 #endif
 
 #if defined(CONFIG_ACPI) && defined(CONFIG_PM_SLEEP)
-- 
2.25.1



From mario.limonciello at amd.com  Tue Jun  8 08:42:55 2021
From: mario.limonciello at amd.com (Mario Limonciello)
Date: Tue,  8 Jun 2021 10:42:55 -0500
Subject: [PATCH v7 2/2] ACPI: Add quirks for AMD Renoir/Lucienne CPUs to force
 the D3 hint
In-Reply-To: <20210608154255.8693-1-mario.limonciello@amd.com>
References: <20210608154255.8693-1-mario.limonciello@amd.com>
Message-ID: <20210608154255.8693-2-mario.limonciello@amd.com>

AMD systems from Renoir and Lucienne require that the NVME controller
is put into D3 over a Modern Standby / suspend-to-idle
cycle.  This is "typically" accomplished using the `StorageD3Enable`
property in the _DSD, but this property was introduced after many
of these systems launched and most OEM systems don't have it in
their BIOS.

On AMD Renoir without these drives going into D3 over suspend-to-idle
the resume will fail with the NVME controller being reset and a trace
like this in the kernel logs:
```
[   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
[   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
[   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
[   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
[   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
[   95.332843] nvme nvme0: Abort status: 0x371
[   95.332852] nvme nvme0: Abort status: 0x371
[   95.332856] nvme nvme0: Abort status: 0x371
[   95.332859] nvme nvme0: Abort status: 0x371
[   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
[   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
```

The Microsoft documentation for StorageD3Enable mentioned that Windows has
a hardcoded allowlist for D3 support, which was used for these platforms.
Introduce quirks to hardcode them for Linux as well.

As this property is now "standardized", OEM systems using AMD Cezanne and
newer APU's have adopted this property, and quirks like this should not be
necessary.

CC: Julian Sikorski <belegdol at gmail.com>
CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
CC: Alexander Deucher <Alexander.Deucher at amd.com>
CC: Rafael J. Wysocki <rjw at rjwysocki.net>
CC: Prike Liang <prike.liang at amd.com>
Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
---
 drivers/acpi/device_pm.c |  3 +++
 drivers/acpi/internal.h  |  9 +++++++++
 drivers/acpi/x86/utils.c | 25 +++++++++++++++++++++++++
 3 files changed, 37 insertions(+)

Changes from v4->v5:
 * Add this patch back in as it's been made apparent that the
   system needs to be hardcoded for these.
   Changes:
   - Drop Cezanne - it's now covered by StorageD3Enable
   - Rebase ontop of acpi_storage_d3 outside of NVME
Changes from v5->v6:
 * Move the quirk check into drivers/acpi/x86/ as suggested by
   Rafael.
Changes from v6->v7:
 * Move header location
 * Optimization of force function

diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index c45f2d76b67e..31e0025a913e 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -1356,6 +1356,9 @@ bool acpi_storage_d3(struct device *dev)
 	struct acpi_device *adev = ACPI_COMPANION(dev);
 	u8 val;
 
+	if (force_storage_d3())
+		return true;
+
 	if (!adev)
 		return false;
 	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
diff --git a/drivers/acpi/internal.h b/drivers/acpi/internal.h
index f973bbe90e5e..e29ec463bb07 100644
--- a/drivers/acpi/internal.h
+++ b/drivers/acpi/internal.h
@@ -236,6 +236,15 @@ static inline int suspend_nvs_save(void) { return 0; }
 static inline void suspend_nvs_restore(void) {}
 #endif
 
+#ifdef CONFIG_X86
+bool force_storage_d3(void);
+#else
+static inline bool force_storage_d3(void)
+{
+	return false;
+}
+#endif
+
 /*--------------------------------------------------------------------------
 				Device properties
   -------------------------------------------------------------------------- */
diff --git a/drivers/acpi/x86/utils.c b/drivers/acpi/x86/utils.c
index bdc1ba00aee9..5298bb4d81fe 100644
--- a/drivers/acpi/x86/utils.c
+++ b/drivers/acpi/x86/utils.c
@@ -135,3 +135,28 @@ bool acpi_device_always_present(struct acpi_device *adev)
 
 	return ret;
 }
+
+/*
+ * AMD systems from Renoir and Lucienne *require* that the NVME controller
+ * is put into D3 over a Modern Standby / suspend-to-idle cycle.
+ *
+ * This is "typically" accomplished using the `StorageD3Enable`
+ * property in the _DSD that is checked via the `acpi_storage_d3` function
+ * but this property was introduced after many of these systems launched
+ * and most OEM systems don't have it in their BIOS.
+ *
+ * The Microsoft documentation for StorageD3Enable mentioned that Windows has
+ * a hardcoded allowlist for D3 support, which was used for these platforms.
+ *
+ * This allows quirking on Linux in a similar fashion.
+ */
+const struct x86_cpu_id storage_d3_cpu_ids[] = {
+	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),	/* Renoir */
+	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL),	/* Lucienne */
+	{}
+};
+
+bool force_storage_d3(void)
+{
+	return x86_match_cpu(storage_d3_cpu_ids);
+}
-- 
2.25.1



From belegdol at gmail.com  Tue Jun  8 09:20:23 2021
From: belegdol at gmail.com (Julian Sikorski)
Date: Tue, 8 Jun 2021 18:20:23 +0200
Subject: [PATCH v7 2/2] ACPI: Add quirks for AMD Renoir/Lucienne CPUs to
 force the D3 hint
In-Reply-To: <20210608154255.8693-2-mario.limonciello@amd.com>
References: <20210608154255.8693-1-mario.limonciello@amd.com>
 <20210608154255.8693-2-mario.limonciello@amd.com>
Message-ID: <0825a2d6-be28-3a4d-54b3-c45cc18fbab9@gmail.com>

W dniu 08.06.2021 o?17:42, Mario Limonciello pisze:
> AMD systems from Renoir and Lucienne require that the NVME controller
> is put into D3 over a Modern Standby / suspend-to-idle
> cycle.  This is "typically" accomplished using the `StorageD3Enable`
> property in the _DSD, but this property was introduced after many
> of these systems launched and most OEM systems don't have it in
> their BIOS.
> 
> On AMD Renoir without these drives going into D3 over suspend-to-idle
> the resume will fail with the NVME controller being reset and a trace
> like this in the kernel logs:
> ```
> [   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
> [   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
> [   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
> [   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
> [   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
> [   95.332843] nvme nvme0: Abort status: 0x371
> [   95.332852] nvme nvme0: Abort status: 0x371
> [   95.332856] nvme nvme0: Abort status: 0x371
> [   95.332859] nvme nvme0: Abort status: 0x371
> [   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
> [   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
> ```
> 
> The Microsoft documentation for StorageD3Enable mentioned that Windows has
> a hardcoded allowlist for D3 support, which was used for these platforms.
> Introduce quirks to hardcode them for Linux as well.
> 
> As this property is now "standardized", OEM systems using AMD Cezanne and
> newer APU's have adopted this property, and quirks like this should not be
> necessary.
> 
> CC: Julian Sikorski <belegdol at gmail.com>
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> ---
>   drivers/acpi/device_pm.c |  3 +++
>   drivers/acpi/internal.h  |  9 +++++++++
>   drivers/acpi/x86/utils.c | 25 +++++++++++++++++++++++++
>   3 files changed, 37 insertions(+)
> 
> Changes from v4->v5:
>   * Add this patch back in as it's been made apparent that the
>     system needs to be hardcoded for these.
>     Changes:
>     - Drop Cezanne - it's now covered by StorageD3Enable
>     - Rebase ontop of acpi_storage_d3 outside of NVME
> Changes from v5->v6:
>   * Move the quirk check into drivers/acpi/x86/ as suggested by
>     Rafael.
> Changes from v6->v7:
>   * Move header location
>   * Optimization of force function
> 
> diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
> index c45f2d76b67e..31e0025a913e 100644
> --- a/drivers/acpi/device_pm.c
> +++ b/drivers/acpi/device_pm.c
> @@ -1356,6 +1356,9 @@ bool acpi_storage_d3(struct device *dev)
>   	struct acpi_device *adev = ACPI_COMPANION(dev);
>   	u8 val;
>   
> +	if (force_storage_d3())
> +		return true;
> +
>   	if (!adev)
>   		return false;
>   	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> diff --git a/drivers/acpi/internal.h b/drivers/acpi/internal.h
> index f973bbe90e5e..e29ec463bb07 100644
> --- a/drivers/acpi/internal.h
> +++ b/drivers/acpi/internal.h
> @@ -236,6 +236,15 @@ static inline int suspend_nvs_save(void) { return 0; }
>   static inline void suspend_nvs_restore(void) {}
>   #endif
>   
> +#ifdef CONFIG_X86
> +bool force_storage_d3(void);
> +#else
> +static inline bool force_storage_d3(void)
> +{
> +	return false;
> +}
> +#endif
> +
>   /*--------------------------------------------------------------------------
>   				Device properties
>     -------------------------------------------------------------------------- */
> diff --git a/drivers/acpi/x86/utils.c b/drivers/acpi/x86/utils.c
> index bdc1ba00aee9..5298bb4d81fe 100644
> --- a/drivers/acpi/x86/utils.c
> +++ b/drivers/acpi/x86/utils.c
> @@ -135,3 +135,28 @@ bool acpi_device_always_present(struct acpi_device *adev)
>   
>   	return ret;
>   }
> +
> +/*
> + * AMD systems from Renoir and Lucienne *require* that the NVME controller
> + * is put into D3 over a Modern Standby / suspend-to-idle cycle.
> + *
> + * This is "typically" accomplished using the `StorageD3Enable`
> + * property in the _DSD that is checked via the `acpi_storage_d3` function
> + * but this property was introduced after many of these systems launched
> + * and most OEM systems don't have it in their BIOS.
> + *
> + * The Microsoft documentation for StorageD3Enable mentioned that Windows has
> + * a hardcoded allowlist for D3 support, which was used for these platforms.
> + *
> + * This allows quirking on Linux in a similar fashion.
> + */
> +const struct x86_cpu_id storage_d3_cpu_ids[] = {
> +	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),	/* Renoir */
> +	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL),	/* Lucienne */
> +	{}
> +};
> +
> +bool force_storage_d3(void)
> +{
> +	return x86_match_cpu(storage_d3_cpu_ids);
> +}
> 

Tested-by: Julian Sikorski <belegdol at gmail.com>


From hch at lst.de  Tue Jun  8 09:42:47 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 18:42:47 +0200
Subject: [PATCHv5] nvme: allow to re-attach namespaces after all paths
 are down
In-Reply-To: <20210607133435.138525-1-hare@suse.de>
References: <20210607133435.138525-1-hare@suse.de>
Message-ID: <20210608164247.GA12430@lst.de>

While I do like the look, I don't see how this could actually work:

>  	} else {
>  		ret = -EINVAL;
> -		if (!is_shared || !head->shared) {
> +		/*
> +		 * If multipath is enabled we might hit an ns head with no
> +		 * paths, but that doesn't indicate it's a shared namespace.
> +		 */
> +		if (!list_empty(&head->list) && (!is_shared || !head->shared)) {
>  			dev_err(ctrl->device,
>  				"Duplicate unshared namespace %d\n", nsid);
>  			goto out_put_ns_head;

When we get here with an existing ns_head with no namespaces attached
to it, we can assume it once had namespaces attached to it.  Because
of that NVME_NSHEAD_DISK_LIVE will be set, and we'll never add the
disk for it again.  Even if we did add the disk again what prevents
adding a disk and removing one from racing with each other?


From hch at lst.de  Tue Jun  8 09:43:17 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 18:43:17 +0200
Subject: [PATCH] nvmet/fc: do not check for invalid target port in
 nvmet_fc_handle_fcp_rqst()
In-Reply-To: <20210525125414.68118-1-hare@suse.de>
References: <20210525125414.68118-1-hare@suse.de>
Message-ID: <20210608164317.GA12661@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Tue Jun  8 09:43:28 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 18:43:28 +0200
Subject: [PATCH] nvme-tcp: fix error codes in nvme_tcp_setup_ctrl()
In-Reply-To: <YLtykIn+Or0l08oV@mwanda>
References: <YLtykIn+Or0l08oV@mwanda>
Message-ID: <20210608164328.GB12661@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Tue Jun  8 09:43:42 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 18:43:42 +0200
Subject: [PATCH v3] nvme: remove superfluous bio_set_dev in
 nvme_requeue_work
In-Reply-To: <20210607084651.99122-1-dwagner@suse.de>
References: <20210607084651.99122-1-dwagner@suse.de>
Message-ID: <20210608164342.GC12661@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Tue Jun  8 09:43:51 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 18:43:51 +0200
Subject: [PATCH v2] nvme: verify MNAN value if ANA is enabled
In-Reply-To: <20210607085656.101772-1-dwagner@suse.de>
References: <20210607085656.101772-1-dwagner@suse.de>
Message-ID: <20210608164351.GD12661@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Tue Jun  8 09:44:21 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 18:44:21 +0200
Subject: [PATCH v2 1/4] nvmet: change sn size and check validity
In-Reply-To: <20210607092324.1031046-1-mgurtovoy@nvidia.com>
References: <20210607092324.1031046-1-mgurtovoy@nvidia.com>
Message-ID: <20210608164421.GA12806@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Tue Jun  8 09:48:41 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 18:48:41 +0200
Subject: [PATCH V2 0/5] nvme: small cleanups
In-Reply-To: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
References: <20210607195454.16829-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210608164841.GA13206@lst.de>

Thanks,

I've applied patches 1, 4 and 5 to nvme-5.14.  Please resend 2 and 3
as they don't apply to the current tree.


From hch at lst.de  Tue Jun  8 09:50:16 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 8 Jun 2021 18:50:16 +0200
Subject: BUG: scheduling while atomic when nvmet_rdma_queue_response
 fails in posting a request
In-Reply-To: <CH0PR18MB4129247C2B9AE092E3C59740A1209@CH0PR18MB4129.namprd18.prod.outlook.com>
References: <CH0PR18MB4129247C2B9AE092E3C59740A1209@CH0PR18MB4129.namprd18.prod.outlook.com>
Message-ID: <20210608165016.GA13293@lst.de>

What kernel version is this?

On Sun, May 30, 2021 at 07:33:18AM +0000, Michal Kalderon wrote:
> 
> this leads to nvmet_rdma_release_rsp being called from softirq eventually
> reaching the blk_mq_delay_run_hw_queue which tries to schedule in softirq. (full stack below)
> 
> could you please advise what the correct solution should be in this case ? 
> 
> thanks,
> Michal
> 
> [ 8790.082863] nvmet_rdma: post_recv cmd failed
> [ 8790.083484] nvmet_rdma: sending cmd response failed
> [ 8790.084131] ------------[ cut here ]------------
> [ 8790.084140] WARNING: CPU: 7 PID: 46 at block/blk-mq.c:1422 __blk_mq_run_hw_queue+0xb7/0x100
> [ 8790.084619] Modules linked in: null_blk nvmet_rdma nvmet nvme_rdma nvme_fabrics nvme_core netconsole qedr(OE) qede(OE) qed(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_CHECKSUM nft_chain_nat xt_MASQUERADE nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nft_counter nft_compat tun bridge stp llc nf_tables nfnetlink ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib ib_umad rpcrdma rdma_ucm ib_iser rdma_cm iw_cm intel_rapl_msr intel_rapl_common ib_cm sb_edac libiscsi scsi_transport_iscsi kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sunrpc rapl ib_uverbs ib_core cirrus drm_kms_helper drm virtio_balloon i2c_piix4 pcspkr crc32c_intel virtio_net serio_raw net_failover failover floppy crc8 ata_generic pata_acpi qemu_fw_cfg [last unloaded: qedr]
> [ 8790.084748] CPU: 7 PID: 46 Comm: ksoftirqd/7 Tainted: G?????????? OE???? 5.8.10 #1
> [ 8790.084749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180724_192412-buildhw-07.phx2.fedoraproject.org-1.fc29 04/01/2014
> [ 8790.084752] RIP: 0010:__blk_mq_run_hw_queue+0xb7/0x100
> [ 8790.084753] Code: 00 48 89 ef e8 ea 34 c8 ff 48 89 df 41 89 c4 e8 1f 7f 00 00 f6 83 a8 00 00 00 20 74 b1 41 f7 c4 fe ff ff ff 74 b7 0f 0b eb b3 <0f> 0b eb 86 48 83 bf 98 00 00 00 00 48 c7 c0 df 81 3f 82 48 c7 c2
> [ 8790.084754] RSP: 0018:ffffc9000020ba60 EFLAGS: 00010206
> [ 8790.084755] RAX: 0000000000000100 RBX: ffff88809fe8c400 RCX: 00000000ffffffff
> [ 8790.084756] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88809fe8c400
> [ 8790.084756] RBP: ffff888137b81a50 R08: ffffffffffffffff R09: 0000000000000020
> [ 8790.084757] R10: 0000000000000001 R11: ffff8881365d4968 R12: 0000000000000000
> [ 8790.084758] R13: ffff888137b81a40 R14: ffff88811e2b9e80 R15: ffff8880b3d964f0
> [ 8790.084759] FS:? 0000000000000000(0000) GS:ffff88813bbc0000(0000) knlGS:0000000000000000
> [ 8790.084759] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 8790.084760] CR2: 000055ca53900da8 CR3: 000000012b83e006 CR4: 0000000000360ee0
> [ 8790.084763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 8790.084763] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 8790.084764] Call Trace:
> [ 8790.084767]? __blk_mq_delay_run_hw_queue+0x140/0x160
> [ 8790.084768]? blk_mq_get_tag+0x1d1/0x270
> [ 8790.084771]? ? finish_wait+0x80/0x80
> [ 8790.084773]? __blk_mq_alloc_request+0xb1/0x100
> [ 8790.084774]? blk_mq_make_request+0x144/0x5d0
> [ 8790.084778]? generic_make_request+0x2db/0x340
> [ 8790.084779]? ? bvec_alloc+0x82/0xe0
> [ 8790.084781]? submit_bio+0x43/0x160
> [ 8790.084781]? ? bio_add_page+0x39/0x90
> [ 8790.084794]? nvmet_bdev_execute_rw+0x28c/0x360 [nvmet]
> [ 8790.084800]? nvmet_rdma_execute_command+0x72/0x110 [nvmet_rdma]
> [ 8790.084802]? nvmet_rdma_release_rsp+0xc1/0x1e0 [nvmet_rdma]
> [ 8790.084804]? nvmet_rdma_queue_response.cold.63+0x14/0x19 [nvmet_rdma]
> [ 8790.084806]? nvmet_req_complete+0x11/0x40 [nvmet]
> [ 8790.084809]? nvmet_bio_done+0x27/0x100 [nvmet]
> [ 8790.084811]? blk_update_request+0x23e/0x3b0
> [ 8790.084812]? blk_mq_end_request+0x1a/0x120
> [ 8790.084814]? blk_done_softirq+0xa1/0xd0
> [ 8790.084818]? __do_softirq+0xe4/0x2f8
> [ 8790.084821]? ? sort_range+0x20/0x20
> [ 8790.084824]? run_ksoftirqd+0x26/0x40
> [ 8790.084825]? smpboot_thread_fn+0xc5/0x160
> [ 8790.084827]? kthread+0x116/0x130
> [ 8790.084828]? ? kthread_park+0x80/0x80
> [ 8790.084832]? ret_from_fork+0x22/0x30
> [ 8790.084833] ---[ end trace 16ec813ee3f82b56 ]---
> [ 8790.085314] BUG: scheduling while atomic: ksoftirqd/7/46/0x00000100
---end quoted text---


From sagi at grimberg.me  Tue Jun  8 10:43:45 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 8 Jun 2021 10:43:45 -0700
Subject: BUG: scheduling while atomic when nvmet_rdma_queue_response fails
 in posting a request
In-Reply-To: <CH0PR18MB4129247C2B9AE092E3C59740A1209@CH0PR18MB4129.namprd18.prod.outlook.com>
References: <CH0PR18MB4129247C2B9AE092E3C59740A1209@CH0PR18MB4129.namprd18.prod.outlook.com>
Message-ID: <e059fc90-81e7-dbe8-86f4-87b4bdbd5bb0@grimberg.me>


> Hi Christoph, Sagi,
> 
> We're testing some device error recovery scenarios and hit the following BUG, stack trace below.
> In the error scenario, nvmet_rdma_queue_response receives an error from the device when trying to post a wr,
> 
> this leads to nvmet_rdma_release_rsp being called from softirq eventually
> reaching the blk_mq_delay_run_hw_queue which tries to schedule in softirq. (full stack below)
> 
> could you please advise what the correct solution should be in this case ?

Hey Michal,

I agree this can happen and requires correction. Does the below resolve
the issue?

--
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 7d607f435e36..6d2eea322779 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -16,6 +16,7 @@
  #include <linux/wait.h>
  #include <linux/inet.h>
  #include <asm/unaligned.h>
+#include <linux/async.h>

  #include <rdma/ib_verbs.h>
  #include <rdma/rdma_cm.h>
@@ -712,6 +713,12 @@ static void nvmet_rdma_send_done(struct ib_cq *cq, 
struct ib_wc *wc)
         }
  }

+static void nvmet_rdma_async_release_rsp(void *data, async_cookie_t cookie)
+{
+       struct nvmet_rdma_rsp *rsp = data;
+       nvmet_rdma_release_rsp(rsp);
+}
+
  static void nvmet_rdma_queue_response(struct nvmet_req *req)
  {
         struct nvmet_rdma_rsp *rsp =
@@ -745,7 +752,12 @@ static void nvmet_rdma_queue_response(struct 
nvmet_req *req)

         if (unlikely(ib_post_send(cm_id->qp, first_wr, NULL))) {
                 pr_err("sending cmd response failed\n");
-               nvmet_rdma_release_rsp(rsp);
+               /*
+                * We might be in atomic context, hence release
+                * the rsp in async context in case we need to
+                * process the wr_wait_list.
+                */
+               async_schedule(nvmet_rdma_async_release_rsp, rsp);
         }
  }
--

> 
> thanks,
> Michal
> 
> [ 8790.082863] nvmet_rdma: post_recv cmd failed
> [ 8790.083484] nvmet_rdma: sending cmd response failed
> [ 8790.084131] ------------[ cut here ]------------
> [ 8790.084140] WARNING: CPU: 7 PID: 46 at block/blk-mq.c:1422 __blk_mq_run_hw_queue+0xb7/0x100
> [ 8790.084619] Modules linked in: null_blk nvmet_rdma nvmet nvme_rdma nvme_fabrics nvme_core netconsole qedr(OE) qede(OE) qed(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_CHECKSUM nft_chain_nat xt_MASQUERADE nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nft_counter nft_compat tun bridge stp llc nf_tables nfnetlink ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib ib_umad rpcrdma rdma_ucm ib_iser rdma_cm iw_cm intel_rapl_msr intel_rapl_common ib_cm sb_edac libiscsi scsi_transport_iscsi kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sunrpc rapl ib_uverbs ib_core cirrus drm_kms_helper drm virtio_balloon i2c_piix4 pcspkr crc32c_intel virtio_net serio_raw net_failover failover floppy crc8 ata_generic pata_acpi qemu_fw_cfg [last unloaded: qedr]
> [ 8790.084748] CPU: 7 PID: 46 Comm: ksoftirqd/7 Tainted: G?????????? OE???? 5.8.10 #1
> [ 8790.084749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180724_192412-buildhw-07.phx2.fedoraproject.org-1.fc29 04/01/2014
> [ 8790.084752] RIP: 0010:__blk_mq_run_hw_queue+0xb7/0x100
> [ 8790.084753] Code: 00 48 89 ef e8 ea 34 c8 ff 48 89 df 41 89 c4 e8 1f 7f 00 00 f6 83 a8 00 00 00 20 74 b1 41 f7 c4 fe ff ff ff 74 b7 0f 0b eb b3 <0f> 0b eb 86 48 83 bf 98 00 00 00 00 48 c7 c0 df 81 3f 82 48 c7 c2
> [ 8790.084754] RSP: 0018:ffffc9000020ba60 EFLAGS: 00010206
> [ 8790.084755] RAX: 0000000000000100 RBX: ffff88809fe8c400 RCX: 00000000ffffffff
> [ 8790.084756] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88809fe8c400
> [ 8790.084756] RBP: ffff888137b81a50 R08: ffffffffffffffff R09: 0000000000000020
> [ 8790.084757] R10: 0000000000000001 R11: ffff8881365d4968 R12: 0000000000000000
> [ 8790.084758] R13: ffff888137b81a40 R14: ffff88811e2b9e80 R15: ffff8880b3d964f0
> [ 8790.084759] FS:? 0000000000000000(0000) GS:ffff88813bbc0000(0000) knlGS:0000000000000000
> [ 8790.084759] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 8790.084760] CR2: 000055ca53900da8 CR3: 000000012b83e006 CR4: 0000000000360ee0
> [ 8790.084763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 8790.084763] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 8790.084764] Call Trace:
> [ 8790.084767]? __blk_mq_delay_run_hw_queue+0x140/0x160
> [ 8790.084768]? blk_mq_get_tag+0x1d1/0x270
> [ 8790.084771]? ? finish_wait+0x80/0x80
> [ 8790.084773]? __blk_mq_alloc_request+0xb1/0x100
> [ 8790.084774]? blk_mq_make_request+0x144/0x5d0
> [ 8790.084778]? generic_make_request+0x2db/0x340
> [ 8790.084779]? ? bvec_alloc+0x82/0xe0
> [ 8790.084781]? submit_bio+0x43/0x160
> [ 8790.084781]? ? bio_add_page+0x39/0x90
> [ 8790.084794]? nvmet_bdev_execute_rw+0x28c/0x360 [nvmet]
> [ 8790.084800]? nvmet_rdma_execute_command+0x72/0x110 [nvmet_rdma]
> [ 8790.084802]? nvmet_rdma_release_rsp+0xc1/0x1e0 [nvmet_rdma]
> [ 8790.084804]? nvmet_rdma_queue_response.cold.63+0x14/0x19 [nvmet_rdma]
> [ 8790.084806]? nvmet_req_complete+0x11/0x40 [nvmet]
> [ 8790.084809]? nvmet_bio_done+0x27/0x100 [nvmet]
> [ 8790.084811]? blk_update_request+0x23e/0x3b0
> [ 8790.084812]? blk_mq_end_request+0x1a/0x120
> [ 8790.084814]? blk_done_softirq+0xa1/0xd0
> [ 8790.084818]? __do_softirq+0xe4/0x2f8
> [ 8790.084821]? ? sort_range+0x20/0x20
> [ 8790.084824]? run_ksoftirqd+0x26/0x40
> [ 8790.084825]? smpboot_thread_fn+0xc5/0x160
> [ 8790.084827]? kthread+0x116/0x130
> [ 8790.084828]? ? kthread_park+0x80/0x80
> [ 8790.084832]? ret_from_fork+0x22/0x30
> [ 8790.084833] ---[ end trace 16ec813ee3f82b56 ]---
> [ 8790.085314] BUG: scheduling while atomic: ksoftirqd/7/46/0x00000100
> 


From hare at suse.de  Tue Jun  8 10:44:04 2021
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 8 Jun 2021 19:44:04 +0200
Subject: [PATCHv5] nvme: allow to re-attach namespaces after all paths are
 down
In-Reply-To: <20210608164247.GA12430@lst.de>
References: <20210607133435.138525-1-hare@suse.de>
 <20210608164247.GA12430@lst.de>
Message-ID: <02a8ca3c-3f6d-7f1d-0afd-655ab1b78f9a@suse.de>

On 6/8/21 6:42 PM, Christoph Hellwig wrote:
> While I do like the look, I don't see how this could actually work:
> 
>>   	} else {
>>   		ret = -EINVAL;
>> -		if (!is_shared || !head->shared) {
>> +		/*
>> +		 * If multipath is enabled we might hit an ns head with no
>> +		 * paths, but that doesn't indicate it's a shared namespace.
>> +		 */
>> +		if (!list_empty(&head->list) && (!is_shared || !head->shared)) {
>>   			dev_err(ctrl->device,
>>   				"Duplicate unshared namespace %d\n", nsid);
>>   			goto out_put_ns_head;
> 
> When we get here with an existing ns_head with no namespaces attached
> to it, we can assume it once had namespaces attached to it.  Because
> of that NVME_NSHEAD_DISK_LIVE will be set, and we'll never add the
> disk for it again.  Even if we did add the disk again what prevents
> adding a disk and removing one from racing with each other?
> 
Hmm. I thought it was resolved by refcount magic.
Will have a closer look to check what needs to be done.

Thanks for the review.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From kbusch at kernel.org  Tue Jun  8 11:41:34 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 8 Jun 2021 11:41:34 -0700
Subject: BUG: scheduling while atomic when nvmet_rdma_queue_response
 fails in posting a request
In-Reply-To: <e059fc90-81e7-dbe8-86f4-87b4bdbd5bb0@grimberg.me>
References: <CH0PR18MB4129247C2B9AE092E3C59740A1209@CH0PR18MB4129.namprd18.prod.outlook.com>
 <e059fc90-81e7-dbe8-86f4-87b4bdbd5bb0@grimberg.me>
Message-ID: <20210608184134.GA339600@dhcp-10-100-145-180.wdc.com>

On Tue, Jun 08, 2021 at 10:43:45AM -0700, Sagi Grimberg wrote:
> diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
> index 7d607f435e36..6d2eea322779 100644
> --- a/drivers/nvme/target/rdma.c
> +++ b/drivers/nvme/target/rdma.c
> @@ -16,6 +16,7 @@
>  #include <linux/wait.h>
>  #include <linux/inet.h>
>  #include <asm/unaligned.h>
> +#include <linux/async.h>
> 
>  #include <rdma/ib_verbs.h>
>  #include <rdma/rdma_cm.h>
> @@ -712,6 +713,12 @@ static void nvmet_rdma_send_done(struct ib_cq *cq,
> struct ib_wc *wc)
>         }
>  }
> 
> +static void nvmet_rdma_async_release_rsp(void *data, async_cookie_t cookie)
> +{
> +       struct nvmet_rdma_rsp *rsp = data;
> +       nvmet_rdma_release_rsp(rsp);
> +}
> +
>  static void nvmet_rdma_queue_response(struct nvmet_req *req)
>  {
>         struct nvmet_rdma_rsp *rsp =
> @@ -745,7 +752,12 @@ static void nvmet_rdma_queue_response(struct nvmet_req
> *req)
> 
>         if (unlikely(ib_post_send(cm_id->qp, first_wr, NULL))) {
>                 pr_err("sending cmd response failed\n");
> -               nvmet_rdma_release_rsp(rsp);
> +               /*
> +                * We might be in atomic context, hence release
> +                * the rsp in async context in case we need to
> +                * process the wr_wait_list.
> +                */
> +               async_schedule(nvmet_rdma_async_release_rsp, rsp);
>         }
>  }

Just FYI, async_schedule() has conditions where it may execute your
callback synchronously. Your suggestion is probably fine for testing,
but it sounds like you require something that can guarantee a non-atomic
context for nvmet_rdma_release_rsp().


From davem at davemloft.net  Tue Jun  8 12:08:39 2021
From: davem at davemloft.net (David Miller)
Date: Tue, 08 Jun 2021 12:08:39 -0700 (PDT)
Subject: please drop the nvme code from net-next
In-Reply-To: <20210608134303.GA30977@lst.de>
References: <20210608121925.GA24201@lst.de>
 <CAMuHMdWgLf8GJfOaRUyx=AvOTnuOs8FS-2=C+OCk02OLDCyrDg@mail.gmail.com>
 <20210608134303.GA30977@lst.de>
Message-ID: <20210608.120839.259439633590059559.davem@davemloft.net>

From: Christoph Hellwig <hch at lst.de>
Date: Tue, 8 Jun 2021 15:43:03 +0200

> please drop the nvme-offload code from net-next.  Code for drivers/nvme/
> needs ACKs from us nvme maintainers and for something this significant
> also needs to go through the NVMe tree.  And this code is not ready yet.

Please send me a revert, and I will apply it, thank you.

It's tricky because at least one driver uses the new interfaces.

Thank you.


From patchwork-bot+netdevbpf at kernel.org  Tue Jun  8 12:10:05 2021
From: patchwork-bot+netdevbpf at kernel.org (patchwork-bot+netdevbpf at kernel.org)
Date: Tue, 08 Jun 2021 19:10:05 +0000
Subject: [PATCH] nvme: NVME_TCP_OFFLOAD should not default to m
In-Reply-To: <39b1a3684880e1d85ef76e34403886e8f1d22508.1623149635.git.geert+renesas@glider.be>
References: <39b1a3684880e1d85ef76e34403886e8f1d22508.1623149635.git.geert+renesas@glider.be>
Message-ID: <162317940597.2276.17237091802998795532.git-patchwork-notify@kernel.org>

Hello:

This patch was applied to netdev/net-next.git (refs/heads/master):

On Tue,  8 Jun 2021 12:56:09 +0200 you wrote:
> The help text for the symbol controlling support for the NVM Express
> over Fabrics TCP offload common layer suggests to not enable this
> support when unsure.
> 
> Hence drop the "default m", which actually means "default y" if
> CONFIG_MODULES is not enabled.
> 
> [...]

Here is the summary with links:
  - nvme: NVME_TCP_OFFLOAD should not default to m
    https://git.kernel.org/netdev/net-next/c/762411542050

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html




From chaitanya.kulkarni at wdc.com  Tue Jun  8 12:10:36 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue,  8 Jun 2021 12:10:36 -0700
Subject: [PATCH RESEND 1/2] nvme-fc: add a helper to check ctrl sgl support
Message-ID: <20210608191037.44251-1-chaitanya.kulkarni@wdc.com>

For transports it is common to check if NVMe SGLs are supported or not
by the controller.

Add a helper instead of open coding controller SGL support and use it
in fc transport.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/fc.c   | 2 +-
 drivers/nvme/host/nvme.h | 8 ++++++++
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index f183f9fa03d0..7600863f7752 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -3112,7 +3112,7 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	}
 
 	/* FC-NVME supports normal SGL Data Block Descriptors */
-	if (!(ctrl->ctrl.sgls & ((1 << 0) | (1 << 1)))) {
+	if (!nvme_ctrl_sgl_supported(&ctrl->ctrl)) {
 		dev_err(ctrl->ctrl.device,
 			"Mandatory sgls are not supported!\n");
 		ret = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1f397ecba16c..1aab74128d40 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -869,6 +869,14 @@ static inline void nvme_hwmon_exit(struct nvme_ctrl *ctrl)
 }
 #endif
 
+static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
+{
+	if (!(ctrl->sgls & ((1 << 0) | (1 << 1))))
+		return true;
+
+	return false;
+}
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 void nvme_execute_passthru_rq(struct request *rq);
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Tue Jun  8 12:10:37 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue,  8 Jun 2021 12:10:37 -0700
Subject: [PATCH RESEND 2/2] nvme-tcp: use helper for ctrl sgl check
In-Reply-To: <20210608191037.44251-1-chaitanya.kulkarni@wdc.com>
References: <20210608191037.44251-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210608191037.44251-2-chaitanya.kulkarni@wdc.com>

Use helper to check the controller's SGL support instead of open coding.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/tcp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 6a65b0516180..c7bd37103cf4 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1993,7 +1993,7 @@ static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new)
 		goto destroy_admin;
 	}
 
-	if (!(ctrl->sgls & ((1 << 0) | (1 << 1)))) {
+	if (!nvme_ctrl_sgl_supported(ctrl)) {
 		ret = -EOPNOTSUPP;
 		dev_err(ctrl->device, "Mandatory sgls are not supported!\n");
 		goto destroy_admin;
-- 
2.22.1



From kbusch at kernel.org  Tue Jun  8 12:40:42 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 8 Jun 2021 12:40:42 -0700
Subject: please drop the nvme code from net-next
In-Reply-To: <20210608.120839.259439633590059559.davem@davemloft.net>
References: <20210608121925.GA24201@lst.de>
 <CAMuHMdWgLf8GJfOaRUyx=AvOTnuOs8FS-2=C+OCk02OLDCyrDg@mail.gmail.com>
 <20210608134303.GA30977@lst.de>
 <20210608.120839.259439633590059559.davem@davemloft.net>
Message-ID: <20210608194042.GA339628@dhcp-10-100-145-180.wdc.com>

On Tue, Jun 08, 2021 at 12:08:39PM -0700, David Miller wrote:
> From: Christoph Hellwig <hch at lst.de>
> Date: Tue, 8 Jun 2021 15:43:03 +0200
> 
> > please drop the nvme-offload code from net-next.  Code for drivers/nvme/
> > needs ACKs from us nvme maintainers and for something this significant
> > also needs to go through the NVMe tree.  And this code is not ready yet.
> 
> Please send me a revert, and I will apply it, thank you.
> 
> It's tricky because at least one driver uses the new interfaces.

Shouldn't whoever merged un-ACK'ed patches from a different subsystem
get to own the tricky revert?


From kbusch at kernel.org  Tue Jun  8 13:01:01 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 8 Jun 2021 13:01:01 -0700
Subject: [PATCH v6 1/2] ACPI: Move check for _DSD StorageD3Enable
 property to acpi
In-Reply-To: <20210608053546.GA14116@lst.de>
References: <20210607173156.5548-1-mario.limonciello@amd.com>
 <20210608053546.GA14116@lst.de>
Message-ID: <20210608200101.GB339628@dhcp-10-100-145-180.wdc.com>

On Tue, Jun 08, 2021 at 07:35:46AM +0200, Christoph Hellwig wrote:
> On Mon, Jun 07, 2021 at 12:31:55PM -0500, Mario Limonciello wrote:
> > +/**
> > + * acpi_storage_d3 - Check if a storage device should use D3.
> > + * @dev: Device to check
> > + *
> > + * Returns %true if @dev should be put into D3 when the ->suspend method is
> > + * called, else %false.  The name of this function is somewhat misleading
> > + * as it has nothing to do with storage except for the name of the ACPI
> > + * property.  On some platforms resume will not work if this hint is ignored.
> > + *
> > + */
> 
> I still do not like this comment.  There is nothing about only using this
> for storage devices.  It is specific to a PCIe slot, and if I plug
> something that is not a storage device into it the same restrictions
> still apply.

Hm, I thought this comment was good. Despite the dupious connection to
storage, the function name matches the ACPI property, so an explanation
for this misnomer seems appropriate.


From axboe at kernel.dk  Tue Jun  8 14:03:30 2021
From: axboe at kernel.dk (Jens Axboe)
Date: Tue, 8 Jun 2021 15:03:30 -0600
Subject: [GIT PULL] first round of nvme updates for Linux 5.14
In-Reply-To: <YL9/KDQ2uGRVAh6F@infradead.org>
References: <YL9/KDQ2uGRVAh6F@infradead.org>
Message-ID: <b3534f5a-133e-85c9-8ec8-49972a228fa5@kernel.dk>

On 6/8/21 8:31 AM, Christoph Hellwig wrote:
> 
> The following changes since commit 8184035805dc87dd826101b930d3dce97758f7b1:
> 
>   rsxx: Use struct_size() in vmalloc() (2021-05-24 06:47:30 -0600)
> 
> are available in the Git repository at:
> 
>   git://git.infradead.org/nvme.git tags/nvme-5.14-2021-06-08

Pulled, thanks.

-- 
Jens Axboe



From lkp at intel.com  Tue Jun  8 14:30:02 2021
From: lkp at intel.com (kernel test robot)
Date: Wed, 9 Jun 2021 05:30:02 +0800
Subject: [linux-nvme:nvme-5.14 27/28] drivers/nvme/host/pci.c:539:7: error:
 implicit declaration of function 'nvme_ctrl_sgl_supported'
Message-ID: <202106090556.FQr8aEIa-lkp@intel.com>

tree:   git://git.infradead.org/nvme.git nvme-5.14
head:   b0386eb167134d958f1876a1a7e39e113385c516
commit: a854c5eb4e959f0dfc6e1c22b51128ac68bc051c [27/28] nvme-pci: use helper for ctrl sgl check
config: sparc-randconfig-r024-20210608 (attached as .config)
compiler: sparc-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        git remote add linux-nvme git://git.infradead.org/nvme.git
        git fetch --no-tags linux-nvme nvme-5.14
        git checkout a854c5eb4e959f0dfc6e1c22b51128ac68bc051c
        # save the attached .config to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=sparc 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp at intel.com>

All errors (new ones prefixed by >>):

   drivers/nvme/host/pci.c: In function 'nvme_pci_use_sgls':
>> drivers/nvme/host/pci.c:539:7: error: implicit declaration of function 'nvme_ctrl_sgl_supported' [-Werror=implicit-function-declaration]
     539 |  if (!nvme_ctrl_sgl_supported(&dev->ctrl))
         |       ^~~~~~~~~~~~~~~~~~~~~~~
   cc1: some warnings being treated as errors


vim +/nvme_ctrl_sgl_supported +539 drivers/nvme/host/pci.c

   530	
   531	static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
   532	{
   533		struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
   534		int nseg = blk_rq_nr_phys_segments(req);
   535		unsigned int avg_seg_size;
   536	
   537		avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req), nseg);
   538	
 > 539		if (!nvme_ctrl_sgl_supported(&dev->ctrl))
   540			return false;
   541		if (!iod->nvmeq->qid)
   542			return false;
   543		if (!sgl_threshold || avg_seg_size < sgl_threshold)
   544			return false;
   545		return true;
   546	}
   547	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 29349 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20210609/bbca56a8/attachment-0001.gz>

From lkp at intel.com  Tue Jun  8 15:11:26 2021
From: lkp at intel.com (kernel test robot)
Date: Wed, 9 Jun 2021 06:11:26 +0800
Subject: [linux-nvme:nvme-5.14 27/28] drivers/nvme/host/pci.c:539:7: error:
 implicit declaration of function 'nvme_ctrl_sgl_supported'
Message-ID: <202106090620.0DjzhCKh-lkp@intel.com>

tree:   git://git.infradead.org/nvme.git nvme-5.14
head:   b0386eb167134d958f1876a1a7e39e113385c516
commit: a854c5eb4e959f0dfc6e1c22b51128ac68bc051c [27/28] nvme-pci: use helper for ctrl sgl check
config: powerpc-randconfig-r013-20210608 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project d32cc150feb72f315a5bbd34f92e7beca21a50da)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # install powerpc cross compiling tool for clang build
        # apt-get install binutils-powerpc-linux-gnu
        git remote add linux-nvme git://git.infradead.org/nvme.git
        git fetch --no-tags linux-nvme nvme-5.14
        git checkout a854c5eb4e959f0dfc6e1c22b51128ac68bc051c
        # save the attached .config to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=powerpc 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp at intel.com>

All errors (new ones prefixed by >>):

   In file included from drivers/nvme/host/pci.c:7:
   In file included from include/linux/acpi.h:13:
   In file included from include/linux/irqdomain.h:35:
   In file included from include/linux/of.h:15:
   In file included from include/linux/bitops.h:32:
   In file included from arch/powerpc/include/asm/bitops.h:62:
   arch/powerpc/include/asm/barrier.h:49:9: warning: '__lwsync' macro redefined [-Wmacro-redefined]
   #define __lwsync()      __asm__ __volatile__ (stringify_in_c(LWSYNC) : : :"memory")
           ^
   <built-in>:308:9: note: previous definition is here
   #define __lwsync __builtin_ppc_lwsync
           ^
>> drivers/nvme/host/pci.c:539:7: error: implicit declaration of function 'nvme_ctrl_sgl_supported' [-Werror,-Wimplicit-function-declaration]
           if (!nvme_ctrl_sgl_supported(&dev->ctrl))
                ^
   drivers/nvme/host/pci.c:858:8: error: implicit declaration of function 'nvme_ctrl_sgl_supported' [-Werror,-Wimplicit-function-declaration]
                               nvme_ctrl_sgl_supported(&dev->ctrl))
                               ^
   1 warning and 2 errors generated.

Kconfig warnings: (for reference only)
   WARNING: unmet direct dependencies detected for HOTPLUG_CPU
   Depends on SMP && (PPC_PSERIES || PPC_PMAC || PPC_POWERNV || FSL_SOC_BOOKE
   Selected by
   - PM_SLEEP_SMP && SMP && (ARCH_SUSPEND_POSSIBLE || ARCH_HIBERNATION_POSSIBLE && PM_SLEEP


vim +/nvme_ctrl_sgl_supported +539 drivers/nvme/host/pci.c

   530	
   531	static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
   532	{
   533		struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
   534		int nseg = blk_rq_nr_phys_segments(req);
   535		unsigned int avg_seg_size;
   536	
   537		avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req), nseg);
   538	
 > 539		if (!nvme_ctrl_sgl_supported(&dev->ctrl))
   540			return false;
   541		if (!iod->nvmeq->qid)
   542			return false;
   543		if (!sgl_threshold || avg_seg_size < sgl_threshold)
   544			return false;
   545		return true;
   546	}
   547	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 36964 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20210609/28d08855/attachment-0001.gz>

From sagi at grimberg.me  Tue Jun  8 16:39:21 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 8 Jun 2021 16:39:21 -0700
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at
 0000000000000230
In-Reply-To: <CO1PR19MB4885314DCBFEB376C7305C98EE3D9@CO1PR19MB4885.namprd19.prod.outlook.com>
References: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <CO1PR19MB4885314DCBFEB376C7305C98EE3D9@CO1PR19MB4885.namprd19.prod.outlook.com>
Message-ID: <e40414a9-41ef-f82c-7d1c-2583d8251c26@grimberg.me>


> Hi Sagi,
> 
> A correction to the below analysis:
> It seems like sock->sk is NULL and not queue->sock
> 
> As part of _nvme_tcp_stop_queue
> kernel_sock_shutdown and nvme_tcp_restore_sock_calls are being called:
> kernel_sock_shutdown leads to nvme_tcp_state_change which will trigger err_work (nvme_tcp_error_recovery_work)
> 
> As part of nvme_tcp_error_recovery_work, nvme_tcp_free_queue is being called which releases the socket (sock_release)
> 
> In our case, based on the below bt:
> nvme_tcp_error_recovery_work is being triggered (and so sock_release) before nvme_tcp_restore_sock_calls , which end up with NULL dereference pointer at 'rwlock_t sk_callback_lock' ?
> 
> Can you please review and provide your inputs for this potential race ?

Seems that RH8.3 is missing the mutex protection on nvme_tcp_stop_queue.
I'm assuming it doesn't happen upstream?


From sagi at grimberg.me  Tue Jun  8 16:46:19 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 8 Jun 2021 16:46:19 -0700
Subject: [PATCH] nvmet-tcp: fix memory leak when having inflight commands
 on disconnect
In-Reply-To: <20210601160234.5200-1-elad.grupi@dell.com>
References: <20210601160234.5200-1-elad.grupi@dell.com>
Message-ID: <e1170f50-dc26-4eee-87cc-dff2e3690f2d@grimberg.me>


> From: Elad Grupi <elad.grupi at dell.com>
> 
> Some nvme commands might get completed during the execution of
> nvmet_tcp_release_queue_work. Those commands are being added to
> resp_list on the queue and never processed after cancel_work_sync
> is done, causing memory leak of the resources allocated for those commands.
> 
> Signed-off-by: Elad Grupi <elad.grupi at dell.com>
> ---
>   drivers/nvme/target/tcp.c | 26 ++++++++++++++++++++++++++
>   1 file changed, 26 insertions(+)
> 
> diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
> index f9f34f6caf5e..04d53b532ae2 100644
> --- a/drivers/nvme/target/tcp.c
> +++ b/drivers/nvme/target/tcp.c
> @@ -1426,6 +1426,31 @@ static void nvmet_tcp_uninit_data_in_cmds(struct nvmet_tcp_queue *queue)
>   	}
>   }
>   
> +static void nvmet_tcp_free_resp_list(struct nvmet_tcp_queue *queue)
> +{
> +	struct nvmet_tcp_cmd *cmd;
> +	int c = 0;
> +
> +	nvmet_tcp_process_resp_list(queue);
> +
> +	list_for_each_entry(cmd, &queue->resp_send_list, entry) {
> +		kfree(cmd->iov);
> +		sgl_free(cmd->req.sg);
> +		c++;
> +	}
> +
> +	WARN_ON(c != queue->send_list_len);
> +
> +	if (queue->snd_cmd) {
> +		kfree(queue->snd_cmd->iov);
> +		sgl_free(queue->snd_cmd->req.sg);
> +		c++;
> +	}
> +
> +	pr_debug("qid %u send_list_len %d, free %d unprocessed commands\n",
> +			queue->nvme_sq.qid, queue->send_list_len, c);
> +}
> +
>   static void nvmet_tcp_release_queue_work(struct work_struct *w)
>   {
>   	struct nvmet_tcp_queue *queue =
> @@ -1441,6 +1466,7 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)
>   	nvmet_tcp_uninit_data_in_cmds(queue);
>   	nvmet_sq_destroy(&queue->nvme_sq);
>   	cancel_work_sync(&queue->io_work);
> +	nvmet_tcp_free_resp_list(queue);

Can't this be combined with nvme_tcp_free_cmds? Maybe consolidate
the sgl and iov free to something like nvme_tcp_free_cmd_data that
will free these and make it null-free-safe?

>   	sock_release(queue->sock);
>   	nvmet_tcp_free_cmds(queue);
>   	if (queue->hdr_digest || queue->data_digest)
> 


From sagi at grimberg.me  Tue Jun  8 17:03:48 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 8 Jun 2021 17:03:48 -0700
Subject: BUG: scheduling while atomic when nvmet_rdma_queue_response fails
 in posting a request
In-Reply-To: <20210608184134.GA339600@dhcp-10-100-145-180.wdc.com>
References: <CH0PR18MB4129247C2B9AE092E3C59740A1209@CH0PR18MB4129.namprd18.prod.outlook.com>
 <e059fc90-81e7-dbe8-86f4-87b4bdbd5bb0@grimberg.me>
 <20210608184134.GA339600@dhcp-10-100-145-180.wdc.com>
Message-ID: <4a031bc0-fba3-12d6-428f-a378aba1897e@grimberg.me>


>> diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
>> index 7d607f435e36..6d2eea322779 100644
>> --- a/drivers/nvme/target/rdma.c
>> +++ b/drivers/nvme/target/rdma.c
>> @@ -16,6 +16,7 @@
>>   #include <linux/wait.h>
>>   #include <linux/inet.h>
>>   #include <asm/unaligned.h>
>> +#include <linux/async.h>
>>
>>   #include <rdma/ib_verbs.h>
>>   #include <rdma/rdma_cm.h>
>> @@ -712,6 +713,12 @@ static void nvmet_rdma_send_done(struct ib_cq *cq,
>> struct ib_wc *wc)
>>          }
>>   }
>>
>> +static void nvmet_rdma_async_release_rsp(void *data, async_cookie_t cookie)
>> +{
>> +       struct nvmet_rdma_rsp *rsp = data;
>> +       nvmet_rdma_release_rsp(rsp);
>> +}
>> +
>>   static void nvmet_rdma_queue_response(struct nvmet_req *req)
>>   {
>>          struct nvmet_rdma_rsp *rsp =
>> @@ -745,7 +752,12 @@ static void nvmet_rdma_queue_response(struct nvmet_req
>> *req)
>>
>>          if (unlikely(ib_post_send(cm_id->qp, first_wr, NULL))) {
>>                  pr_err("sending cmd response failed\n");
>> -               nvmet_rdma_release_rsp(rsp);
>> +               /*
>> +                * We might be in atomic context, hence release
>> +                * the rsp in async context in case we need to
>> +                * process the wr_wait_list.
>> +                */
>> +               async_schedule(nvmet_rdma_async_release_rsp, rsp);
>>          }
>>   }
> 
> Just FYI, async_schedule() has conditions where it may execute your
> callback synchronously. Your suggestion is probably fine for testing,
> but it sounds like you require something that can guarantee a non-atomic
> context for nvmet_rdma_release_rsp().

OK, it seems that the issue is that we are submitting I/O in atomic
context. This should be more appropriate...

--
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 7d607f435e36..16f2f5a84ae7 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -102,6 +102,7 @@ struct nvmet_rdma_queue {

         struct work_struct      release_work;
         struct list_head        rsp_wait_list;
+       struct work_struct      wr_wait_work;
         struct list_head        rsp_wr_wait_list;
         spinlock_t              rsp_wr_wait_lock;

@@ -517,8 +518,10 @@ static int nvmet_rdma_post_recv(struct 
nvmet_rdma_device *ndev,
         return ret;
  }

-static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
+static void nvmet_rdma_process_wr_wait_list(struct work_struct *w)
  {
+       struct nvmet_rdma_queue *queue =
+               container_of(w, struct nvmet_rdma_queue, wr_wait_work);
         spin_lock(&queue->rsp_wr_wait_lock);
         while (!list_empty(&queue->rsp_wr_wait_list)) {
                 struct nvmet_rdma_rsp *rsp;
@@ -677,7 +680,7 @@ static void nvmet_rdma_release_rsp(struct 
nvmet_rdma_rsp *rsp)
                 nvmet_req_free_sgls(&rsp->req);

         if (unlikely(!list_empty_careful(&queue->rsp_wr_wait_list)))
-               nvmet_rdma_process_wr_wait_list(queue);
+               schedule_work(&queue->wr_wait_work);

         nvmet_rdma_put_rsp(rsp);
  }
@@ -1446,6 +1449,7 @@ nvmet_rdma_alloc_queue(struct nvmet_rdma_device *ndev,
          * inside a CM callback would trigger a deadlock. (great API 
design..)
          */
         INIT_WORK(&queue->release_work, nvmet_rdma_release_queue_work);
+       INIT_WORK(&queue->wr_wait_work, nvmet_rdma_process_wr_wait_list);
         queue->dev = ndev;
         queue->cm_id = cm_id;
         queue->port = port->nport;
--


From Chaitanya.Kulkarni at wdc.com  Tue Jun  8 17:14:06 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed, 9 Jun 2021 00:14:06 +0000
Subject: [linux-nvme:nvme-5.14 27/28] drivers/nvme/host/pci.c:539:7:
 error: implicit declaration of function 'nvme_ctrl_sgl_supported'
In-Reply-To: <202106090556.FQr8aEIa-lkp@intel.com>
References: <202106090556.FQr8aEIa-lkp@intel.com>
Message-ID: <E9067936-7A06-4EFF-A4A9-9405612333B6@wdc.com>


> On Jun 8, 2021, at 2:30 PM, kernel test robot <lkp at intel.com> wrote:
> 
> ?tree:   git://git.infradead.org/nvme.git nvme-5.14
> head:   b0386eb167134d958f1876a1a7e39e113385c516
> commit: a854c5eb4e959f0dfc6e1c22b51128ac68bc051c [27/28] nvme-pci: use helper for ctrl sgl check
> config: sparc-randconfig-r024-20210608 (attached as .config)
> compiler: sparc-linux-gcc (GCC) 9.3.0
> reproduce (this is a W=1 build):
>        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>        chmod +x ~/bin/make.cross
>        git remote add linux-nvme git://git.infradead.org/nvme.git
>        git fetch --no-tags linux-nvme nvme-5.14
>        git checkout a854c5eb4e959f0dfc6e1c22b51128ac68bc051c
>        # save the attached .config to linux build tree
>        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=sparc 
> 
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp at intel.com>
> 
> All errors (new ones prefixed by >>):
> 
>   drivers/nvme/host/pci.c: In function 'nvme_pci_use_sgls':
>>> drivers/nvme/host/pci.c:539:7: error: implicit declaration of function 'nvme_ctrl_sgl_supported' [-Werror=implicit-function-declaration]
>     539 |  if (!nvme_ctrl_sgl_supported(&dev->ctrl))
>         |       ^~~~~~~~~~~~~~~~~~~~~~~
>   cc1: some warnings being treated as errors
> 
> 
> vim +/nvme_ctrl_sgl_supported +539 drivers/nvme/host/pci.c
> 
>   530    
>   531    static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
>   532    {
>   533        struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
>   534        int nseg = blk_rq_nr_phys_segments(req);
>   535        unsigned int avg_seg_size;
>   536    
>   537        avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req), nseg);
>   538    
>> 539        if (!nvme_ctrl_sgl_supported(&dev->ctrl))
>   540            return false;
>   541        if (!iod->nvmeq->qid)
>   542            return false;
>   543        if (!sgl_threshold || avg_seg_size < sgl_threshold)
>   544            return false;
>   545        return true;
>   546    }
>   547    
> 
> ---
> 0-DAY CI Kernel Test Service, Intel Corporation
> https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org
> <.config.gz>


The patches to fix this are already sent.



From sagi at grimberg.me  Tue Jun  8 17:23:49 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 8 Jun 2021 17:23:49 -0700
Subject: [PATCH RESEND 2/2] nvme-tcp: use helper for ctrl sgl check
In-Reply-To: <20210608191037.44251-2-chaitanya.kulkarni@wdc.com>
References: <20210608191037.44251-1-chaitanya.kulkarni@wdc.com>
 <20210608191037.44251-2-chaitanya.kulkarni@wdc.com>
Message-ID: <8f511cf9-ab30-ecca-5a93-ba9dbae387e2@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Tue Jun  8 17:23:54 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 8 Jun 2021 17:23:54 -0700
Subject: [PATCH RESEND 1/2] nvme-fc: add a helper to check ctrl sgl support
In-Reply-To: <20210608191037.44251-1-chaitanya.kulkarni@wdc.com>
References: <20210608191037.44251-1-chaitanya.kulkarni@wdc.com>
Message-ID: <2114a922-21db-09ef-76b3-3380913d927a@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From lkp at intel.com  Tue Jun  8 22:39:02 2021
From: lkp at intel.com (kernel test robot)
Date: Wed, 09 Jun 2021 13:39:02 +0800
Subject: [linux-nvme:nvme-5.14] BUILD REGRESSION
 b0386eb167134d958f1876a1a7e39e113385c516
Message-ID: <60c053f6.eA3BFs9/MLJgLMRl%lkp@intel.com>

tree/branch: git://git.infradead.org/nvme.git nvme-5.14
branch HEAD: b0386eb167134d958f1876a1a7e39e113385c516  nvme-pci: remove trailing lines for helpers

Error/Warning in current branch:

drivers/nvme/host/pci.c:539:7: error: implicit declaration of function 'nvme_ctrl_sgl_supported' [-Werror,-Wimplicit-function-declaration]
drivers/nvme/host/pci.c:539:7: error: implicit declaration of function 'nvme_ctrl_sgl_supported' [-Werror=implicit-function-declaration]

Error/Warning ids grouped by kconfigs:

gcc_recent_errors
|-- alpha-allmodconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- alpha-allyesconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- arm-allmodconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- arm-allyesconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- i386-randconfig-a011-20210608
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- ia64-allmodconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- ia64-allyesconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- mips-allmodconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- parisc-allyesconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- powerpc-allmodconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- powerpc-pseries_defconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- riscv-allmodconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- riscv-allyesconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- s390-allyesconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- s390-defconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- sparc-allyesconfig
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- sparc-randconfig-r024-20210608
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- x86_64-kexec
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- x86_64-randconfig-a003-20210608
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- x86_64-rhel-8.3
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
|-- x86_64-rhel-8.3-kbuiltin
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported
`-- xtensa-allyesconfig
    `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported

clang_recent_errors
|-- powerpc-randconfig-r013-20210608
|   `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported-Werror-Wimplicit-function-declaration
`-- x86_64-randconfig-a011-20210608
    `-- drivers-nvme-host-pci.c:error:implicit-declaration-of-function-nvme_ctrl_sgl_supported-Werror-Wimplicit-function-declaration

elapsed time: 722m

configs tested: 94
configs skipped: 2

gcc tested configs:
arm                                 defconfig
arm64                            allyesconfig
arm64                               defconfig
arm                              allyesconfig
arm                              allmodconfig
xtensa                           allyesconfig
sh                           se7722_defconfig
sh                           se7724_defconfig
arc                          axs103_defconfig
sh                           se7206_defconfig
sh                 kfr2r09-romimage_defconfig
powerpc                 linkstation_defconfig
sh                          sdk7786_defconfig
xtensa                  nommu_kc705_defconfig
um                           x86_64_defconfig
sh                             espt_defconfig
powerpc                     pseries_defconfig
powerpc                    gamecube_defconfig
mips                      bmips_stb_defconfig
um                               alldefconfig
arm                          pxa168_defconfig
mips                           ip27_defconfig
h8300                               defconfig
ia64                             allmodconfig
ia64                                defconfig
ia64                             allyesconfig
x86_64                            allnoconfig
m68k                             allmodconfig
m68k                                defconfig
m68k                             allyesconfig
nds32                               defconfig
nios2                            allyesconfig
csky                                defconfig
alpha                               defconfig
alpha                            allyesconfig
h8300                            allyesconfig
arc                                 defconfig
sh                               allmodconfig
parisc                              defconfig
s390                             allyesconfig
s390                             allmodconfig
parisc                           allyesconfig
s390                                defconfig
i386                             allyesconfig
sparc                            allyesconfig
sparc                               defconfig
i386                                defconfig
nios2                               defconfig
arc                              allyesconfig
nds32                             allnoconfig
mips                             allyesconfig
mips                             allmodconfig
powerpc                          allyesconfig
powerpc                          allmodconfig
powerpc                           allnoconfig
x86_64               randconfig-a004-20210608
x86_64               randconfig-a002-20210608
x86_64               randconfig-a003-20210608
x86_64               randconfig-a006-20210608
x86_64               randconfig-a005-20210608
x86_64               randconfig-a001-20210608
i386                 randconfig-a003-20210608
i386                 randconfig-a006-20210608
i386                 randconfig-a004-20210608
i386                 randconfig-a001-20210608
i386                 randconfig-a005-20210608
i386                 randconfig-a002-20210608
i386                 randconfig-a015-20210608
i386                 randconfig-a013-20210608
i386                 randconfig-a016-20210608
i386                 randconfig-a011-20210608
i386                 randconfig-a012-20210608
i386                 randconfig-a014-20210608
riscv                    nommu_k210_defconfig
riscv                            allyesconfig
riscv                    nommu_virt_defconfig
riscv                             allnoconfig
riscv                               defconfig
riscv                          rv32_defconfig
riscv                            allmodconfig
um                             i386_defconfig
um                            kunit_defconfig
x86_64                           allyesconfig
x86_64                    rhel-8.3-kselftests
x86_64                              defconfig
x86_64                               rhel-8.3
x86_64                      rhel-8.3-kbuiltin
x86_64                                  kexec

clang tested configs:
x86_64               randconfig-a015-20210608
x86_64               randconfig-a012-20210608
x86_64               randconfig-a014-20210608
x86_64               randconfig-a011-20210608
x86_64               randconfig-a016-20210608
x86_64               randconfig-a013-20210608

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org


From Amit.Engel at Dell.com  Wed Jun  9 00:48:59 2021
From: Amit.Engel at Dell.com (Engel, Amit)
Date: Wed, 9 Jun 2021 07:48:59 +0000
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at
 0000000000000230
In-Reply-To: <e40414a9-41ef-f82c-7d1c-2583d8251c26@grimberg.me>
References: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <CO1PR19MB4885314DCBFEB376C7305C98EE3D9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <e40414a9-41ef-f82c-7d1c-2583d8251c26@grimberg.me>
Message-ID: <CO1PR19MB48850B44ED23C179A4E4F541EE369@CO1PR19MB4885.namprd19.prod.outlook.com>

Hi Sagi,

Indeed RHEL8.3 does not have the mutex protection on nvme_tcp_stop_queue
However, in our case, based on the below back trace
We don't get to __nvme_tcp_stop_queue from nvme_tcp_stop_queue
We get to it from:
nvme_tcp_reconnect_ctrl_work --> nvme_tcp_setup_ctrl --> nvme_tcp_start_queue  --> __nvme_tcp_stop_queue

so I'm not sure how this mutex protection will help in this case

crash> bt -l
PID: 193053  TASK: ffff9491bdad17c0  CPU: 7   COMMAND: "kworker/u193:9"
 #0 [ffffb2e9cfdbbb70] machine_kexec at ffffffffb245bf3e
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/kernel/machine_kexec_64.c: 389
 #1 [ffffb2e9cfdbbbc8] __crash_kexec at ffffffffb256072d
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/kernel/kexec_core.c: 956
 #2 [ffffb2e9cfdbbc90] crash_kexec at ffffffffb256160d
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./include/linux/compiler.h: 219
 #3 [ffffb2e9cfdbbca8] oops_end at ffffffffb2422d4d
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/kernel/dumpstack.c: 334
 #4 [ffffb2e9cfdbbcc8] no_context at ffffffffb246ba9e
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/mm/fault.c: 773
 #5 [ffffb2e9cfdbbd20] do_page_fault at ffffffffb246c5c2
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./arch/x86/include/asm/jump_label.h: 38
 #6 [ffffb2e9cfdbbd50] page_fault at ffffffffb2e0122e
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/entry/entry_64.S: 1183
    [exception RIP: _raw_write_lock_bh+23]
    RIP: ffffffffb2cd6cc7  RSP: ffffb2e9cfdbbe00  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff94b2aefb4000  RCX: 0000000000000003
    RDX: 00000000000000ff  RSI: 00000000fffffe01  RDI: 0000000000000230
    RBP: ffff94923f793f40   R8: ffff9492ff1ea7f8   R9: 0000000000000000
    R10: 0000000000000000  R11: ffff9492ff1e8c64  R12: ffff94b2b7210338
    R13: 0000000000000000  R14: ffff94b27f7a4100  R15: ffff94b2b72110a0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./arch/x86/include/asm/atomic.h: 194
 #7 [ffffb2e9cfdbbe00] __nvme_tcp_stop_queue at ffffffffc02dc0aa [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1486
 #8 [ffffb2e9cfdbbe18] nvme_tcp_start_queue at ffffffffc02dcd18 [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1525
 #9 [ffffb2e9cfdbbe38] nvme_tcp_setup_ctrl at ffffffffc02df258 [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1814
#10 [ffffb2e9cfdbbe80] nvme_tcp_reconnect_ctrl_work at ffffffffc02df4bf [nvme_tcp]
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/drivers/nvme/host/tcp.c: 1962
#11 [ffffb2e9cfdbbe98] process_one_work at ffffffffb24d3477
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./arch/x86/include/asm/jump_label.h: 38
#12 [ffffb2e9cfdbbed8] worker_thread at ffffffffb24d3b40
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/./include/linux/compiler.h: 193
#13 [ffffb2e9cfdbbf10] kthread at ffffffffb24d9502
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/kernel/kthread.c: 280
#14 [ffffb2e9cfdbbf50] ret_from_fork at ffffffffb2e00255
    /usr/src/debug/kernel-4.18.0-240.el8/linux-4.18.0-240.el8.x86_64/arch/x86/entry/entry_64.S: 360

-----Original Message-----
From: Sagi Grimberg <sagi at grimberg.me> 
Sent: Wednesday, June 9, 2021 2:39 AM
To: Engel, Amit; linux-nvme at lists.infradead.org
Cc: Anner, Ran; Grupi, Elad
Subject: Re: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at 0000000000000230


[EXTERNAL EMAIL] 


> Hi Sagi,
> 
> A correction to the below analysis:
> It seems like sock->sk is NULL and not queue->sock
> 
> As part of _nvme_tcp_stop_queue
> kernel_sock_shutdown and nvme_tcp_restore_sock_calls are being called:
> kernel_sock_shutdown leads to nvme_tcp_state_change which will trigger err_work (nvme_tcp_error_recovery_work)
> 
> As part of nvme_tcp_error_recovery_work, nvme_tcp_free_queue is being called which releases the socket (sock_release)
> 
> In our case, based on the below bt:
> nvme_tcp_error_recovery_work is being triggered (and so sock_release) before nvme_tcp_restore_sock_calls , which end up with NULL dereference pointer at 'rwlock_t sk_callback_lock' ?
> 
> Can you please review and provide your inputs for this potential race ?

Seems that RH8.3 is missing the mutex protection on nvme_tcp_stop_queue.
I'm assuming it doesn't happen upstream?


From sagi at grimberg.me  Wed Jun  9 01:04:58 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 9 Jun 2021 01:04:58 -0700
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at
 0000000000000230
In-Reply-To: <CO1PR19MB48850B44ED23C179A4E4F541EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
References: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <CO1PR19MB4885314DCBFEB376C7305C98EE3D9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <e40414a9-41ef-f82c-7d1c-2583d8251c26@grimberg.me>
 <CO1PR19MB48850B44ED23C179A4E4F541EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
Message-ID: <b9c7e6cc-fea5-b8ce-af07-7d004116e3e3@grimberg.me>


> Hi Sagi,
> 
> Indeed RHEL8.3 does not have the mutex protection on nvme_tcp_stop_queue
> However, in our case, based on the below back trace
> We don't get to __nvme_tcp_stop_queue from nvme_tcp_stop_queue
> We get to it from:
> nvme_tcp_reconnect_ctrl_work --> nvme_tcp_setup_ctrl --> nvme_tcp_start_queue  --> __nvme_tcp_stop_queue
> 
> so I'm not sure how this mutex protection will help in this case


Oh, well iirc we probably need the same mutex protection in start
failure case then?
--
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 216d21a6a165..00dff3654e6f 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1548,6 +1548,7 @@ static void nvme_tcp_stop_queue(struct nvme_ctrl 
*nctrl, int qid)
  static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx)
  {
         struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
+       struct nvme_tcp_queue *queue = &ctrl->queues[idx];
         int ret;

         if (idx)
@@ -1556,10 +1557,12 @@ static int nvme_tcp_start_queue(struct nvme_ctrl 
*nctrl, int idx)
                 ret = nvmf_connect_admin_queue(nctrl);

         if (!ret) {
-               set_bit(NVME_TCP_Q_LIVE, &ctrl->queues[idx].flags);
+               set_bit(NVME_TCP_Q_LIVE, &queue->flags);
         } else {
-               if (test_bit(NVME_TCP_Q_ALLOCATED, 
&ctrl->queues[idx].flags))
-                       __nvme_tcp_stop_queue(&ctrl->queues[idx]);
+               mutex_lock(&queue->queue_lock);
+               if (test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags))
+                       __nvme_tcp_stop_queue(queue);
+               mutex_unlock(&queue->queue_lock);
                 dev_err(nctrl->device,
                         "failed to connect queue: %d ret=%d\n", idx, ret);
         }
--


From Amit.Engel at Dell.com  Wed Jun  9 01:39:42 2021
From: Amit.Engel at Dell.com (Engel, Amit)
Date: Wed, 9 Jun 2021 08:39:42 +0000
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at
 0000000000000230
In-Reply-To: <b9c7e6cc-fea5-b8ce-af07-7d004116e3e3@grimberg.me>
References: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <CO1PR19MB4885314DCBFEB376C7305C98EE3D9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <e40414a9-41ef-f82c-7d1c-2583d8251c26@grimberg.me>
 <CO1PR19MB48850B44ED23C179A4E4F541EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
 <b9c7e6cc-fea5-b8ce-af07-7d004116e3e3@grimberg.me>
Message-ID: <CO1PR19MB48859C3254E4F3EA3282EDC9EE369@CO1PR19MB4885.namprd19.prod.outlook.com>


Im not sure that using the queue_lock mutex ill help
The race in this case is between sock_release and nvme_tcp_restore_sock_calls
sock_release is being called as part of nvme_tcp_free_queue which is destroying the mutex

-----Original Message-----
From: Sagi Grimberg <sagi at grimberg.me> 
Sent: Wednesday, June 9, 2021 11:05 AM
To: Engel, Amit; linux-nvme at lists.infradead.org
Cc: Anner, Ran; Grupi, Elad
Subject: Re: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at 0000000000000230


[EXTERNAL EMAIL] 


> Hi Sagi,
> 
> Indeed RHEL8.3 does not have the mutex protection on 
> nvme_tcp_stop_queue However, in our case, based on the below back 
> trace We don't get to __nvme_tcp_stop_queue from nvme_tcp_stop_queue 
> We get to it from:
> nvme_tcp_reconnect_ctrl_work --> nvme_tcp_setup_ctrl --> 
> nvme_tcp_start_queue  --> __nvme_tcp_stop_queue
> 
> so I'm not sure how this mutex protection will help in this case


Oh, well iirc we probably need the same mutex protection in start failure case then?
--
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 216d21a6a165..00dff3654e6f 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1548,6 +1548,7 @@ static void nvme_tcp_stop_queue(struct nvme_ctrl *nctrl, int qid)
  static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx)
  {
         struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
+       struct nvme_tcp_queue *queue = &ctrl->queues[idx];
         int ret;

         if (idx)
@@ -1556,10 +1557,12 @@ static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx)
                 ret = nvmf_connect_admin_queue(nctrl);

         if (!ret) {
-               set_bit(NVME_TCP_Q_LIVE, &ctrl->queues[idx].flags);
+               set_bit(NVME_TCP_Q_LIVE, &queue->flags);
         } else {
-               if (test_bit(NVME_TCP_Q_ALLOCATED, 
&ctrl->queues[idx].flags))
-                       __nvme_tcp_stop_queue(&ctrl->queues[idx]);
+               mutex_lock(&queue->queue_lock);
+               if (test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags))
+                       __nvme_tcp_stop_queue(queue);
+               mutex_unlock(&queue->queue_lock);
                 dev_err(nctrl->device,
                         "failed to connect queue: %d ret=%d\n", idx, ret);
         }
--

From sagi at grimberg.me  Wed Jun  9 02:11:09 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 9 Jun 2021 02:11:09 -0700
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at
 0000000000000230
In-Reply-To: <CO1PR19MB48859C3254E4F3EA3282EDC9EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
References: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <CO1PR19MB4885314DCBFEB376C7305C98EE3D9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <e40414a9-41ef-f82c-7d1c-2583d8251c26@grimberg.me>
 <CO1PR19MB48850B44ED23C179A4E4F541EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
 <b9c7e6cc-fea5-b8ce-af07-7d004116e3e3@grimberg.me>
 <CO1PR19MB48859C3254E4F3EA3282EDC9EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
Message-ID: <a76ecf8e-15d0-1ebc-aaf2-c93fee591602@grimberg.me>


> Im not sure that using the queue_lock mutex ill help
> The race in this case is between sock_release and nvme_tcp_restore_sock_calls
> sock_release is being called as part of nvme_tcp_free_queue which is destroying the mutex

Maybe I'm not understanding the issue here. What is the scenario again?
stop_queue is called (ctx1), that triggers error_recovery (ctx2) which
then calls free_queue before ctx1 gets to restore sock callbacks?

err_work will first stop the queues before freeing them, so it will
serialize behind ctx1. What am I missing?


From dan.carpenter at oracle.com  Wed Jun  9 02:28:31 2021
From: dan.carpenter at oracle.com (Dan Carpenter)
Date: Wed, 9 Jun 2021 12:28:31 +0300
Subject: [PATCH] nvme-tcp-offload: Fix error code in
 nvme_tcp_ofld_create_ctrl()
Message-ID: <YMCJv0sYKm+Gcwr7@mwanda>

Return an error pointer if try_module_get() fails.  Currently it returns
NULL.

Fixes: 4b8178ec5794 ("nvme-tcp-offload: Add device scan implementation")
Signed-off-by: Dan Carpenter <dan.carpenter at oracle.com>
---
 drivers/nvme/host/tcp-offload.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index c76822e5ada7..c07f84a7e697 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -1215,7 +1215,7 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 	/* Increase driver refcnt */
 	if (!try_module_get(dev->ops->module)) {
 		pr_err("try_module_get failed\n");
-		dev = NULL;
+		rc = -ENODEV;
 		goto out_free_ctrl;
 	}
 
-- 
2.30.2



From Damien.LeMoal at wdc.com  Wed Jun  9 02:32:11 2021
From: Damien.LeMoal at wdc.com (Damien Le Moal)
Date: Wed, 9 Jun 2021 09:32:11 +0000
Subject: [PATCH 1/5] block: export blk_next_bio()
References: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
 <20210608054635.5661-2-chaitanya.kulkarni@wdc.com>
Message-ID: <DM6PR04MB70819F84C206B3AE2756C709E7369@DM6PR04MB7081.namprd04.prod.outlook.com>

On 2021/06/08 14:46, Chaitanya Kulkarni wrote:
> The block layer provides emulation of zone management operations
> targeting all zones of a zoned block device only for the zone reset
> operation (REQ_OP_ZONE_RESET). In order to correctly implement
> exporting of zoned block devices with NVMeOF, emulating zone management
> operations targeting all zones of a device is also necessary for the
> open, close and finish zone operations (REQ_OP_ZONE_OPEN,
> REQ_OP_ZONE_CLOSE and REQ_OP_ZONE_FINISH).
> 
> Instead of duplicating the code, export the existing helper from block
> layer so we can use a bio chaining pattern that is present in the block
> layer for REQ_OP_ZONE RESET all emulation in the NVMeOF zoned block
> device backend.
> 
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> ---
>  block/blk-lib.c     | 1 +
>  include/linux/bio.h | 2 ++
>  2 files changed, 3 insertions(+)
> 
> diff --git a/block/blk-lib.c b/block/blk-lib.c
> index 7b256131b20b..9f09beadcbe3 100644
> --- a/block/blk-lib.c
> +++ b/block/blk-lib.c
> @@ -21,6 +21,7 @@ struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp)
>  
>  	return new;
>  }
> +EXPORT_SYMBOL_GPL(blk_next_bio);
>  
>  int __blkdev_issue_discard(struct block_device *bdev, sector_t sector,
>  		sector_t nr_sects, gfp_t gfp_mask, int flags,
> diff --git a/include/linux/bio.h b/include/linux/bio.h
> index a0b4cfdf62a4..b2491ead22a0 100644
> --- a/include/linux/bio.h
> +++ b/include/linux/bio.h
> @@ -822,4 +822,6 @@ static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb)
>  		bio->bi_opf |= REQ_NOWAIT;
>  }
>  
> +struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp);
> +
>  #endif /* __LINUX_BIO_H */
> 

Note: this should be sent to Jens and linux-block too. This needs Jens Ack.

Looks good.

Reviewed-by: Damien Le Moal <damien.lemoal at wdc.com>

-- 
Damien Le Moal
Western Digital Research


From Damien.LeMoal at wdc.com  Wed Jun  9 02:32:28 2021
From: Damien.LeMoal at wdc.com (Damien Le Moal)
Date: Wed, 9 Jun 2021 09:32:28 +0000
Subject: [PATCH 2/5] nvmet: add req cns error complete helper
References: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
 <20210608054635.5661-3-chaitanya.kulkarni@wdc.com>
Message-ID: <DM6PR04MB7081145E11808E08288DFF3CE7369@DM6PR04MB7081.namprd04.prod.outlook.com>

On 2021/06/08 14:46, Chaitanya Kulkarni wrote:
> We report error and complete the request when identify cns value is not
> handled in nvmet_execute_identify(). This error reporting is also needed
> for Zone Block Device backend for NVMeOF target.
> 
> Add a helper nvmet_req_cns_error_compplete() to report an error and
> complete the request when idenitfy command cns not handled value.
> 
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> ---
>  drivers/nvme/target/admin-cmd.c | 5 +----
>  drivers/nvme/target/nvmet.h     | 8 ++++++++
>  2 files changed, 9 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
> index dcd49a72f2f3..94230803c75a 100644
> --- a/drivers/nvme/target/admin-cmd.c
> +++ b/drivers/nvme/target/admin-cmd.c
> @@ -659,10 +659,7 @@ static void nvmet_execute_identify(struct nvmet_req *req)
>  		return nvmet_execute_identify_desclist(req);
>  	}
>  
> -	pr_debug("unhandled identify cns %d on qid %d\n",
> -	       req->cmd->identify.cns, req->sq->qid);
> -	req->error_loc = offsetof(struct nvme_identify, cns);
> -	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
> +	nvmet_req_cns_error_complete(req);
>  }
>  
>  /*
> diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
> index d69a409515d6..058b6d3d48fa 100644
> --- a/drivers/nvme/target/nvmet.h
> +++ b/drivers/nvme/target/nvmet.h
> @@ -622,4 +622,12 @@ static inline bool nvmet_use_inline_bvec(struct nvmet_req *req)
>  	       req->sg_cnt <= NVMET_MAX_INLINE_BIOVEC;
>  }
>  
> +static inline void nvmet_req_cns_error_complete(struct nvmet_req *req)
> +{
> +	pr_debug("unhandled identify cns %d on qid %d\n",
> +	       req->cmd->identify.cns, req->sq->qid);
> +	req->error_loc = offsetof(struct nvme_identify, cns);
> +	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
> +}
> +
>  #endif /* _NVMET_H */
> 

Looks good.

Reviewed-by: Damien Le Moal <damien.lemoal at wdc.com>


-- 
Damien Le Moal
Western Digital Research


From Damien.LeMoal at wdc.com  Wed Jun  9 02:32:50 2021
From: Damien.LeMoal at wdc.com (Damien Le Moal)
Date: Wed, 9 Jun 2021 09:32:50 +0000
Subject: [PATCH 3/5] nvmet: add nvmet_req_bio put helper for backends
References: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
 <20210608054635.5661-4-chaitanya.kulkarni@wdc.com>
Message-ID: <DM6PR04MB7081F255F299A6359A3ECDDCE7369@DM6PR04MB7081.namprd04.prod.outlook.com>

On 2021/06/08 14:47, Chaitanya Kulkarni wrote:
> In current code there exists two backends which are using inline bio
> optimization, that adds a duplicate code for freeing the bio.
> 
> For Zoned Block Device backend we also use the same optimzation and it
> will lead to having duplicate code in the three backends: generic
> bdev, passsthru, and generic zns.
> 
> Add a helper function to avoid duplicate code and update the respective
> backends.
> 
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> ---
>  drivers/nvme/target/io-cmd-bdev.c | 3 +--
>  drivers/nvme/target/nvmet.h       | 6 ++++++
>  drivers/nvme/target/passthru.c    | 3 +--
>  3 files changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
> index f673679d258a..40534d9a86d1 100644
> --- a/drivers/nvme/target/io-cmd-bdev.c
> +++ b/drivers/nvme/target/io-cmd-bdev.c
> @@ -164,8 +164,7 @@ static void nvmet_bio_done(struct bio *bio)
>  	struct nvmet_req *req = bio->bi_private;
>  
>  	nvmet_req_complete(req, blk_to_nvme_status(req, bio->bi_status));
> -	if (bio != &req->b.inline_bio)
> -		bio_put(bio);
> +	nvmet_req_bio_put(req, bio);
>  }
>  
>  #ifdef CONFIG_BLK_DEV_INTEGRITY
> diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
> index 058b6d3d48fa..a7d035be9f13 100644
> --- a/drivers/nvme/target/nvmet.h
> +++ b/drivers/nvme/target/nvmet.h
> @@ -630,4 +630,10 @@ static inline void nvmet_req_cns_error_complete(struct nvmet_req *req)
>  	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
>  }
>  
> +static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio)
> +{
> +	if (bio != &req->b.inline_bio)
> +		bio_put(bio);
> +}
> +
>  #endif /* _NVMET_H */
> diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
> index 39b1473f7204..fced52de33ce 100644
> --- a/drivers/nvme/target/passthru.c
> +++ b/drivers/nvme/target/passthru.c
> @@ -206,8 +206,7 @@ static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq)
>  	for_each_sg(req->sg, sg, req->sg_cnt, i) {
>  		if (bio_add_pc_page(rq->q, bio, sg_page(sg), sg->length,
>  				    sg->offset) < sg->length) {
> -			if (bio != &req->p.inline_bio)
> -				bio_put(bio);
> +			nvmet_req_bio_put(req, bio);
>  			return -EINVAL;
>  		}
>  	}
> 

Looks good.

Reviewed-by: Damien Le Moal <damien.lemoal at wdc.com>


-- 
Damien Le Moal
Western Digital Research


From Damien.LeMoal at wdc.com  Wed Jun  9 02:44:21 2021
From: Damien.LeMoal at wdc.com (Damien Le Moal)
Date: Wed, 9 Jun 2021 09:44:21 +0000
Subject: [PATCH 5/5] nvmet: add ZBD over ZNS backend support
References: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
 <20210608054635.5661-6-chaitanya.kulkarni@wdc.com>
Message-ID: <DM6PR04MB7081372FC5AB2D825F3372DAE7369@DM6PR04MB7081.namprd04.prod.outlook.com>

On 2021/06/08 14:47, Chaitanya Kulkarni wrote:
> NVMe TP 4053 ? Zoned Namespaces (ZNS) allows host software to
> communicate with a non-volatile memory subsystem using zones for NVMe
> protocol-based controllers. NVMeOF already support the ZNS NVMe
> Protocol compliant devices on the target in the passthru mode. There
> are generic zoned block devices like  Shingled Magnetic Recording (SMR)
> HDDs that are not based on the NVMe protocol.
> 
> This patch adds ZNS backend support for non-ZNS zoned block devices as
> NVMeOF targets.
> 
> This support includes implementing the new command set NVME_CSI_ZNS,
> adding different command handlers for ZNS command set such as NVMe
> Identify Controller, NVMe Identify Namespace, NVMe Zone Append,
> NVMe Zone Management Send and NVMe Zone Management Receive.
> 
> With the new command set identifier, we also update the target command
> effects logs to reflect the ZNS compliant commands.
> 
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> ---
>  drivers/nvme/target/Makefile      |   1 +
>  drivers/nvme/target/admin-cmd.c   |  41 ++
>  drivers/nvme/target/core.c        |  15 +-
>  drivers/nvme/target/io-cmd-bdev.c |  26 +-
>  drivers/nvme/target/nvmet.h       |  20 +
>  drivers/nvme/target/zns.c         | 622 ++++++++++++++++++++++++++++++
>  include/linux/nvme.h              |   7 +
>  7 files changed, 721 insertions(+), 11 deletions(-)
>  create mode 100644 drivers/nvme/target/zns.c
> 
> diff --git a/drivers/nvme/target/Makefile b/drivers/nvme/target/Makefile
> index ebf91fc4c72e..9837e580fa7e 100644
> --- a/drivers/nvme/target/Makefile
> +++ b/drivers/nvme/target/Makefile
> @@ -12,6 +12,7 @@ obj-$(CONFIG_NVME_TARGET_TCP)		+= nvmet-tcp.o
>  nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
>  			discovery.o io-cmd-file.o io-cmd-bdev.o
>  nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
> +nvmet-$(CONFIG_BLK_DEV_ZONED)		+= zns.o
>  nvme-loop-y	+= loop.o
>  nvmet-rdma-y	+= rdma.o
>  nvmet-fc-y	+= fc.o
> diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
> index 118b81a34bbb..72470e815146 100644
> --- a/drivers/nvme/target/admin-cmd.c
> +++ b/drivers/nvme/target/admin-cmd.c
> @@ -179,6 +179,13 @@ static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log)
>  	log->iocs[nvme_cmd_write_zeroes]	= cpu_to_le32(1 << 0);
>  }
>  
> +static void nvmet_get_cmd_effects_zns(struct nvme_effects_log *log)
> +{
> +	log->iocs[nvme_cmd_zone_append]		= cpu_to_le32(1 << 0);
> +	log->iocs[nvme_cmd_zone_mgmt_send]	= cpu_to_le32(1 << 0);
> +	log->iocs[nvme_cmd_zone_mgmt_recv]	= cpu_to_le32(1 << 0);
> +}
> +
>  static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
>  {
>  	struct nvme_effects_log *log;
> @@ -194,6 +201,14 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
>  	case NVME_CSI_NVM:
>  		nvmet_get_cmd_effects_nvm(log);
>  		break;
> +	case NVME_CSI_ZNS:
> +		if (!IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
> +			status = NVME_SC_INVALID_IO_CMD_SET;
> +			goto free;
> +		}
> +		nvmet_get_cmd_effects_nvm(log);
> +		nvmet_get_cmd_effects_zns(log);
> +		break;
>  	default:
>  		status = NVME_SC_INVALID_LOG_PAGE;
>  		goto free;
> @@ -669,6 +684,12 @@ static bool nvmet_handle_identify_desclist(struct nvmet_req *req)
>  	case NVME_CSI_NVM:
>  		nvmet_execute_identify_desclist(req);
>  		return true;
> +	case NVME_CSI_ZNS:
> +		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
> +			nvmet_execute_identify_desclist(req);
> +			return true;
> +		}
> +		return false;
>  	default:
>  		return false;
>  	}
> @@ -688,12 +709,32 @@ static void nvmet_execute_identify(struct nvmet_req *req)
>  			break;
>  		}
>  		break;
> +	case NVME_ID_CNS_CS_NS:
> +		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
> +			switch (req->cmd->identify.csi) {
> +			case NVME_CSI_ZNS:
> +				return nvmet_execute_identify_cns_cs_ns(req);
> +			default:
> +				break;
> +			}
> +		}
> +		break;
>  	case NVME_ID_CNS_CTRL:
>  		switch (req->cmd->identify.csi) {
>  		case NVME_CSI_NVM:
>  			return nvmet_execute_identify_ctrl(req);
>  		}
>  		break;
> +	case NVME_ID_CNS_CS_CTRL:
> +		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
> +			switch (req->cmd->identify.csi) {
> +			case NVME_CSI_ZNS:
> +				return nvmet_execute_identify_cns_cs_ctrl(req);
> +			default:
> +				break;
> +			}
> +		}
> +		break;
>  	case NVME_ID_CNS_NS_ACTIVE_LIST:
>  		switch (req->cmd->identify.csi) {
>  		case NVME_CSI_NVM:
> diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
> index 7ee9418cd9b6..3ca85e948fc4 100644
> --- a/drivers/nvme/target/core.c
> +++ b/drivers/nvme/target/core.c
> @@ -16,6 +16,7 @@
>  #include "nvmet.h"
>  
>  struct workqueue_struct *buffered_io_wq;
> +struct workqueue_struct *zbd_wq;
>  static const struct nvmet_fabrics_ops *nvmet_transports[NVMF_TRTYPE_MAX];
>  static DEFINE_IDA(cntlid_ida);
>  
> @@ -893,6 +894,10 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
>  		if (req->ns->file)
>  			return nvmet_file_parse_io_cmd(req);
>  		return nvmet_bdev_parse_io_cmd(req);
> +	case NVME_CSI_ZNS:
> +		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED))
> +			return nvmet_bdev_zns_parse_io_cmd(req);
> +		return NVME_SC_INVALID_IO_CMD_SET;
>  	default:
>  		return NVME_SC_INVALID_IO_CMD_SET;
>  	}
> @@ -1587,11 +1592,15 @@ static int __init nvmet_init(void)
>  
>  	nvmet_ana_group_enabled[NVMET_DEFAULT_ANA_GRPID] = 1;
>  
> +	zbd_wq = alloc_workqueue("nvmet-zbd-wq", WQ_MEM_RECLAIM, 0);
> +	if (!zbd_wq)
> +		return -ENOMEM;
> +
>  	buffered_io_wq = alloc_workqueue("nvmet-buffered-io-wq",
>  			WQ_MEM_RECLAIM, 0);
>  	if (!buffered_io_wq) {
>  		error = -ENOMEM;
> -		goto out;
> +		goto out_free_zbd_work_queue;
>  	}
>  
>  	error = nvmet_init_discovery();
> @@ -1607,7 +1616,8 @@ static int __init nvmet_init(void)
>  	nvmet_exit_discovery();
>  out_free_work_queue:
>  	destroy_workqueue(buffered_io_wq);
> -out:
> +out_free_zbd_work_queue:
> +	destroy_workqueue(zbd_wq);
>  	return error;
>  }
>  
> @@ -1617,6 +1627,7 @@ static void __exit nvmet_exit(void)
>  	nvmet_exit_discovery();
>  	ida_destroy(&cntlid_ida);
>  	destroy_workqueue(buffered_io_wq);
> +	destroy_workqueue(zbd_wq);
>  
>  	BUILD_BUG_ON(sizeof(struct nvmf_disc_rsp_page_entry) != 1024);
>  	BUILD_BUG_ON(sizeof(struct nvmf_disc_rsp_page_hdr) != 1024);
> diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
> index 40534d9a86d1..57f619aa7cba 100644
> --- a/drivers/nvme/target/io-cmd-bdev.c
> +++ b/drivers/nvme/target/io-cmd-bdev.c
> @@ -47,6 +47,14 @@ void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id)
>  	id->nows = to0based(ql->io_opt / ql->logical_block_size);
>  }
>  
> +void nvmet_bdev_ns_disable(struct nvmet_ns *ns)
> +{
> +	if (ns->bdev) {
> +		blkdev_put(ns->bdev, FMODE_WRITE | FMODE_READ);
> +		ns->bdev = NULL;
> +	}
> +}
> +
>  static void nvmet_bdev_ns_enable_integrity(struct nvmet_ns *ns)
>  {
>  	struct blk_integrity *bi = bdev_get_integrity(ns->bdev);
> @@ -86,15 +94,15 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
>  	if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY_T10))
>  		nvmet_bdev_ns_enable_integrity(ns);
>  
> -	return 0;
> -}
> -
> -void nvmet_bdev_ns_disable(struct nvmet_ns *ns)
> -{
> -	if (ns->bdev) {
> -		blkdev_put(ns->bdev, FMODE_WRITE | FMODE_READ);
> -		ns->bdev = NULL;
> +	if (bdev_is_zoned(ns->bdev)) {
> +		if (!nvmet_bdev_zns_enable(ns)) {
> +			nvmet_bdev_ns_disable(ns);
> +			return -EINVAL;
> +		}
> +		ns->csi = NVME_CSI_ZNS;
>  	}
> +
> +	return 0;
>  }
>  
>  void nvmet_bdev_ns_revalidate(struct nvmet_ns *ns)
> @@ -102,7 +110,7 @@ void nvmet_bdev_ns_revalidate(struct nvmet_ns *ns)
>  	ns->size = i_size_read(ns->bdev->bd_inode);
>  }
>  
> -static u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts)
> +u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts)
>  {
>  	u16 status = NVME_SC_SUCCESS;
>  
> diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
> index f709238014f5..a105982e8f8a 100644
> --- a/drivers/nvme/target/nvmet.h
> +++ b/drivers/nvme/target/nvmet.h
> @@ -248,6 +248,10 @@ struct nvmet_subsys {
>  	unsigned int		admin_timeout;
>  	unsigned int		io_timeout;
>  #endif /* CONFIG_NVME_TARGET_PASSTHRU */
> +
> +#ifdef CONFIG_BLK_DEV_ZONED
> +	u8			zasl;
> +#endif /* CONFIG_BLK_DEV_ZONED */
>  };
>  
>  static inline struct nvmet_subsys *to_subsys(struct config_item *item)
> @@ -333,6 +337,12 @@ struct nvmet_req {
>  			struct work_struct      work;
>  			bool			use_workqueue;
>  		} p;
> +#ifdef CONFIG_BLK_DEV_ZONED
> +		struct {
> +			struct bio		inline_bio;
> +			struct work_struct	zmgmt_work;
> +		} z;
> +#endif /* CONFIG_BLK_DEV_ZONED */
>  	};
>  	int			sg_cnt;
>  	int			metadata_sg_cnt;
> @@ -352,6 +362,7 @@ struct nvmet_req {
>  };
>  
>  extern struct workqueue_struct *buffered_io_wq;
> +extern struct workqueue_struct *zbd_wq;
>  
>  static inline void nvmet_set_result(struct nvmet_req *req, u32 result)
>  {
> @@ -401,6 +412,7 @@ u16 nvmet_parse_connect_cmd(struct nvmet_req *req);
>  void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id);
>  u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req);
>  u16 nvmet_file_parse_io_cmd(struct nvmet_req *req);
> +u16 nvmet_bdev_zns_parse_io_cmd(struct nvmet_req *req);
>  u16 nvmet_parse_admin_cmd(struct nvmet_req *req);
>  u16 nvmet_parse_discovery_cmd(struct nvmet_req *req);
>  u16 nvmet_parse_fabrics_cmd(struct nvmet_req *req);
> @@ -528,6 +540,14 @@ void nvmet_ns_changed(struct nvmet_subsys *subsys, u32 nsid);
>  void nvmet_bdev_ns_revalidate(struct nvmet_ns *ns);
>  int nvmet_file_ns_revalidate(struct nvmet_ns *ns);
>  void nvmet_ns_revalidate(struct nvmet_ns *ns);
> +u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts);
> +
> +bool nvmet_bdev_zns_enable(struct nvmet_ns *ns);
> +void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req);
> +void nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req);
> +void nvmet_bdev_execute_zone_mgmt_recv(struct nvmet_req *req);
> +void nvmet_bdev_execute_zone_mgmt_send(struct nvmet_req *req);
> +void nvmet_bdev_execute_zone_append(struct nvmet_req *req);
>  
>  static inline u32 nvmet_rw_data_len(struct nvmet_req *req)
>  {
> diff --git a/drivers/nvme/target/zns.c b/drivers/nvme/target/zns.c
> new file mode 100644
> index 000000000000..34135cbdd7e5
> --- /dev/null
> +++ b/drivers/nvme/target/zns.c
> @@ -0,0 +1,622 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * NVMe ZNS-ZBD command implementation.
> + * Copyright (C) 2021 Western Digital Corporation or its affiliates.
> + */
> +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
> +#include <linux/nvme.h>
> +#include <linux/blkdev.h>
> +#include "nvmet.h"
> +
> +/*
> + * We set the Memory Page Size Minimum (MPSMIN) for target controller to 0
> + * which gets added by 12 in the nvme_enable_ctrl() which results in 2^12 = 4k
> + * as page_shift value. When calculating the ZASL use shift by 12.
> + */
> +#define NVMET_MPSMIN_SHIFT	12
> +
> +static inline u8 nvmet_zasl(unsigned int zone_append_sects)
> +{
> +	/*
> +	 * Zone Append Size Limit (zasl) is expressed as a power of 2 value
> +	 * with the minimum memory page size (i.e. 12) as unit.
> +	 */
> +	return ilog2(zone_append_sects >> (NVMET_MPSMIN_SHIFT - 9));
> +}
> +
> +static int validate_conv_zones_cb(struct blk_zone *z,
> +				  unsigned int i, void *data)
> +{
> +	if (z->type == BLK_ZONE_TYPE_CONVENTIONAL)
> +		return -EOPNOTSUPP;
> +	return 0;
> +}
> +
> +bool nvmet_bdev_zns_enable(struct nvmet_ns *ns)
> +{
> +	struct request_queue *q = ns->bdev->bd_disk->queue;
> +	u8 zasl = nvmet_zasl(queue_max_zone_append_sectors(q));
> +	struct gendisk *bd_disk = ns->bdev->bd_disk;
> +	int ret;
> +
> +	if (ns->subsys->zasl) {
> +		if (ns->subsys->zasl > zasl)
> +			return false;
> +	}
> +	ns->subsys->zasl = zasl;
> +
> +	/*
> +	 * Generic zoned block devices may have a smaller last zone which is
> +	 * not supported by ZNS. Exclude zoned drives that have such smaller
> +	 * last zone.
> +	 */
> +	if (get_capacity(bd_disk) & (bdev_zone_sectors(ns->bdev) - 1))
> +		return false;
> +	/*
> +	 * ZNS does not define a conventional zone type. If the underlying
> +	 * device has a bitmap set indicating the existence of conventional
> +	 * zones, reject the device. Otherwise, use report zones to detect if
> +	 * the device has conventional zones.
> +	 */
> +	if (ns->bdev->bd_disk->queue->conv_zones_bitmap)
> +		return false;
> +
> +	ret = blkdev_report_zones(ns->bdev, 0, blkdev_nr_zones(bd_disk),
> +				  validate_conv_zones_cb, NULL);
> +	if (ret < 0)
> +		return false;
> +
> +	ns->blksize_shift = blksize_bits(bdev_logical_block_size(ns->bdev));
> +
> +	return true;
> +}
> +
> +void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req)
> +{
> +	u8 zasl = req->sq->ctrl->subsys->zasl;
> +	struct nvmet_ctrl *ctrl = req->sq->ctrl;
> +	struct nvme_id_ctrl_zns *id;
> +	u16 status;
> +
> +	id = kzalloc(sizeof(*id), GFP_KERNEL);
> +	if (!id) {
> +		status = NVME_SC_INTERNAL;
> +		goto out;
> +	}
> +
> +	if (ctrl->ops->get_mdts)
> +		id->zasl = min_t(u8, ctrl->ops->get_mdts(ctrl), zasl);
> +	else
> +		id->zasl = zasl;
> +
> +	status = nvmet_copy_to_sgl(req, 0, id, sizeof(*id));
> +
> +	kfree(id);
> +out:
> +	nvmet_req_complete(req, status);
> +}
> +
> +void nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req)
> +{
> +	struct nvme_id_ns_zns *id_zns;
> +	u64 zsze;
> +	u16 status;
> +
> +	if (le32_to_cpu(req->cmd->identify.nsid) == NVME_NSID_ALL) {
> +		req->error_loc = offsetof(struct nvme_identify, nsid);
> +		status = NVME_SC_INVALID_NS | NVME_SC_DNR;
> +		goto out;
> +	}
> +
> +	id_zns = kzalloc(sizeof(*id_zns), GFP_KERNEL);
> +	if (!id_zns) {
> +		status = NVME_SC_INTERNAL;
> +		goto out;
> +	}
> +
> +	status = nvmet_req_find_ns(req);
> +	if (status) {
> +		status = NVME_SC_INTERNAL;
> +		goto done;
> +	}
> +
> +	if (!bdev_is_zoned(req->ns->bdev)) {
> +		req->error_loc = offsetof(struct nvme_identify, nsid);
> +		status = NVME_SC_INVALID_NS | NVME_SC_DNR;
> +		goto done;
> +	}
> +
> +	nvmet_ns_revalidate(req->ns);
> +	zsze = (bdev_zone_sectors(req->ns->bdev) << 9) >>
> +					req->ns->blksize_shift;
> +	id_zns->lbafe[0].zsze = cpu_to_le64(zsze);
> +	id_zns->mor = cpu_to_le32(bdev_max_open_zones(req->ns->bdev));
> +	id_zns->mar = cpu_to_le32(bdev_max_active_zones(req->ns->bdev));
> +
> +done:
> +	status = nvmet_copy_to_sgl(req, 0, id_zns, sizeof(*id_zns));
> +	kfree(id_zns);
> +out:
> +	nvmet_req_complete(req, status);
> +}
> +
> +static u16 nvmet_bdev_validate_zone_mgmt_recv(struct nvmet_req *req)
> +{
> +	sector_t sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
> +	u32 out_bufsize = (le32_to_cpu(req->cmd->zmr.numd) + 1) << 2;
> +
> +	if (sect >= get_capacity(req->ns->bdev->bd_disk)) {
> +		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, slba);
> +		return NVME_SC_LBA_RANGE | NVME_SC_DNR;
> +	}
> +
> +	if (out_bufsize < sizeof(struct nvme_zone_report)) {
> +		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, numd);
> +		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
> +	}
> +
> +	if (req->cmd->zmr.zra != NVME_ZRA_ZONE_REPORT) {
> +		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, zra);
> +		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
> +	}
> +
> +	switch (req->cmd->zmr.pr) {
> +	case 0:
> +	case 1:
> +		break;
> +	default:
> +		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, pr);
> +		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
> +	}
> +
> +	switch (req->cmd->zmr.zrasf) {
> +	case NVME_ZRASF_ZONE_REPORT_ALL:
> +	case NVME_ZRASF_ZONE_STATE_EMPTY:
> +	case NVME_ZRASF_ZONE_STATE_IMP_OPEN:
> +	case NVME_ZRASF_ZONE_STATE_EXP_OPEN:
> +	case NVME_ZRASF_ZONE_STATE_CLOSED:
> +	case NVME_ZRASF_ZONE_STATE_FULL:
> +	case NVME_ZRASF_ZONE_STATE_READONLY:
> +	case NVME_ZRASF_ZONE_STATE_OFFLINE:
> +		break;
> +	default:
> +		req->error_loc =
> +			offsetof(struct nvme_zone_mgmt_recv_cmd, zrasf);
> +		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
> +	}
> +
> +	return NVME_SC_SUCCESS;
> +}
> +
> +struct nvmet_report_zone_data {
> +	struct nvmet_req *req;
> +	u64 out_buf_offset;
> +	u64 out_nr_zones;
> +	u64 nr_zones;
> +	u8 zrasf;
> +};
> +
> +static int nvmet_bdev_report_zone_cb(struct blk_zone *z, unsigned i, void *d)
> +{
> +	static const unsigned int nvme_zrasf_to_blk_zcond[] = {
> +		[NVME_ZRASF_ZONE_STATE_EMPTY]	 = BLK_ZONE_COND_EMPTY,
> +		[NVME_ZRASF_ZONE_STATE_IMP_OPEN] = BLK_ZONE_COND_IMP_OPEN,
> +		[NVME_ZRASF_ZONE_STATE_EXP_OPEN] = BLK_ZONE_COND_EXP_OPEN,
> +		[NVME_ZRASF_ZONE_STATE_CLOSED]	 = BLK_ZONE_COND_CLOSED,
> +		[NVME_ZRASF_ZONE_STATE_READONLY] = BLK_ZONE_COND_READONLY,
> +		[NVME_ZRASF_ZONE_STATE_FULL]	 = BLK_ZONE_COND_FULL,
> +		[NVME_ZRASF_ZONE_STATE_OFFLINE]	 = BLK_ZONE_COND_OFFLINE,
> +	};
> +	struct nvmet_report_zone_data *rz = d;
> +
> +	if (rz->zrasf != NVME_ZRASF_ZONE_REPORT_ALL &&
> +	    z->cond != nvme_zrasf_to_blk_zcond[rz->zrasf])
> +		return 0;
> +
> +	if (rz->nr_zones < rz->out_nr_zones) {
> +		struct nvme_zone_descriptor zdesc = { };
> +		u16 status;
> +
> +		zdesc.zcap = nvmet_sect_to_lba(rz->req->ns, z->capacity);
> +		zdesc.zslba = nvmet_sect_to_lba(rz->req->ns, z->start);
> +		zdesc.wp = nvmet_sect_to_lba(rz->req->ns, z->wp);
> +		zdesc.za = z->reset ? 1 << 2 : 0;
> +		zdesc.zs = z->cond << 4;
> +		zdesc.zt = z->type;
> +
> +		status = nvmet_copy_to_sgl(rz->req, rz->out_buf_offset, &zdesc,
> +					   sizeof(zdesc));
> +		if (status)
> +			return -EINVAL;
> +
> +		rz->out_buf_offset += sizeof(zdesc);
> +	}
> +
> +	rz->nr_zones++;
> +
> +	return 0;
> +}
> +
> +static unsigned long nvmet_req_nr_zones_from_slba(struct nvmet_req *req)
> +{
> +	unsigned int sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
> +
> +	return blkdev_nr_zones(req->ns->bdev->bd_disk) -
> +		(sect >> ilog2(bdev_zone_sectors(req->ns->bdev)));
> +}
> +
> +static unsigned long get_nr_zones_from_buf(struct nvmet_req *req, u32 bufsize)
> +{
> +	if (bufsize <= sizeof(struct nvme_zone_report))
> +		return 0;
> +
> +	return (bufsize - sizeof(struct nvme_zone_report)) /
> +		sizeof(struct nvme_zone_descriptor);
> +}
> +
> +void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
> +{
> +	struct nvmet_req *req = container_of(w, struct nvmet_req, z.zmgmt_work);
> +	sector_t start_sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
> +	unsigned long req_slba_nr_zones = nvmet_req_nr_zones_from_slba(req);
> +	u32 out_bufsize = (le32_to_cpu(req->cmd->zmr.numd) + 1) << 2;
> +	__le64 nr_zones;
> +	u16 status;
> +	int ret;
> +	struct nvmet_report_zone_data rz_data = {
> +		.out_nr_zones = get_nr_zones_from_buf(req, out_bufsize),
> +		/* leave the place for report zone header */
> +		.out_buf_offset = sizeof(struct nvme_zone_report),
> +		.zrasf = req->cmd->zmr.zrasf,
> +		.nr_zones = 0,
> +		.req = req,
> +	};
> +
> +	status = nvmet_bdev_validate_zone_mgmt_recv(req);
> +	if (status)
> +		goto out;
> +
> +	if (!req_slba_nr_zones) {
> +		status = NVME_SC_SUCCESS;
> +		goto out;
> +	}
> +
> +	ret = blkdev_report_zones(req->ns->bdev, start_sect, req_slba_nr_zones,
> +				 nvmet_bdev_report_zone_cb, &rz_data);
> +	if (ret < 0) {
> +		status = NVME_SC_INTERNAL;
> +		goto out;
> +	}
> +
> +	/*
> +	 * When partial bit is set nr_zones must indicate the number of zone
> +	 * descriptors actually transferred.
> +	 */
> +	if (req->cmd->zmr.pr)
> +		rz_data.nr_zones = min(rz_data.nr_zones, rz_data.out_nr_zones);
> +
> +	nr_zones = cpu_to_le64(rz_data.nr_zones);
> +	status = nvmet_copy_to_sgl(req, 0, &nr_zones, sizeof(nr_zones));
> +
> +out:
> +	nvmet_req_complete(req, status);
> +}
> +
> +void nvmet_bdev_execute_zone_mgmt_recv(struct nvmet_req *req)
> +{
> +	INIT_WORK(&req->z.zmgmt_work, nvmet_bdev_zone_zmgmt_recv_work);
> +	queue_work(zbd_wq, &req->z.zmgmt_work);
> +}
> +
> +static inline enum req_opf zsa_req_op(u8 zsa)
> +{
> +	switch (zsa) {
> +	case NVME_ZONE_OPEN:
> +		return REQ_OP_ZONE_OPEN;
> +	case NVME_ZONE_CLOSE:
> +		return REQ_OP_ZONE_CLOSE;
> +	case NVME_ZONE_FINISH:
> +		return REQ_OP_ZONE_FINISH;
> +	case NVME_ZONE_RESET:
> +		return REQ_OP_ZONE_RESET;
> +	default:
> +		return REQ_OP_LAST;
> +	}
> +}
> +
> +static u16 blkdev_zone_mgmt_errno_to_nvme_status(int ret)
> +{
> +	switch (ret) {
> +	case 0:
> +		return NVME_SC_SUCCESS;
> +	case -EINVAL:
> +	case -EIO:
> +		return NVME_SC_ZONE_INVALID_TRANSITION | NVME_SC_DNR;
> +	default:
> +		return NVME_SC_INTERNAL;
> +	}
> +}
> +
> +struct nvmet_zone_mgmt_send_all_data {
> +	unsigned long *zbitmap;
> +	struct nvmet_req *req;
> +};
> +
> +static int zmgmt_send_scan_cb(struct blk_zone *z, unsigned i, void *d)
> +{
> +	struct nvmet_zone_mgmt_send_all_data *data = d;
> +
> +	switch (zsa_req_op(data->req->cmd->zms.zsa)) {
> +	case REQ_OP_ZONE_OPEN:
> +		switch (z->cond) {
> +		case BLK_ZONE_COND_CLOSED:
> +			break;
> +		default:
> +			return 0;
> +		}
> +		break;
> +	case REQ_OP_ZONE_CLOSE:
> +		switch (z->cond) {
> +		case BLK_ZONE_COND_IMP_OPEN:
> +		case BLK_ZONE_COND_EXP_OPEN:
> +			break;
> +		default:
> +			return 0;
> +		}
> +		break;
> +	case REQ_OP_ZONE_FINISH:
> +		switch (z->cond) {
> +		case BLK_ZONE_COND_IMP_OPEN:
> +		case BLK_ZONE_COND_EXP_OPEN:
> +		case BLK_ZONE_COND_CLOSED:
> +			break;
> +		default:
> +			return 0;
> +		}
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	set_bit(i, (unsigned long *)(data->zbitmap));

You do not need the cast here. data->zbitmap is an unsigned long *.

> +
> +	return 0;
> +}
> +
> +static u16 nvmet_bdev_zone_mgmt_emulate_all(struct nvmet_req *req)
> +{
> +	struct block_device *bdev = req->ns->bdev;
> +	unsigned int nr_zones = blkdev_nr_zones(bdev->bd_disk);
> +	struct request_queue *q = bdev_get_queue(bdev);
> +	struct bio *bio = NULL;
> +	sector_t sector = 0;
> +	int ret;
> +	struct nvmet_zone_mgmt_send_all_data d = {
> +		.req = req,
> +	};
> +
> +	d.zbitmap = kcalloc_node(BITS_TO_LONGS(nr_zones), sizeof(*(d.zbitmap)),
> +				 GFP_NOIO, q->node);
> +	if (!d.zbitmap) {
> +		ret = -ENOMEM;
> +		goto out;
> +	}
> +
> +	/* Scan and build bitmap of the eligible zones */
> +	ret = blkdev_report_zones(bdev, 0, nr_zones, zmgmt_send_scan_cb, &d);
> +	if (ret != nr_zones) {
> +		if (ret > 0)
> +			ret = -EIO;
> +		goto out;
> +	} else {
> +		/* We scanned all the zones */
> +		ret = 0;
> +	}
> +
> +	while (sector < get_capacity(bdev->bd_disk)) {
> +		if (test_bit(blk_queue_zone_no(q, sector), d.zbitmap)) {
> +			bio = blk_next_bio(bio, 0, GFP_KERNEL);
> +			bio->bi_opf = zsa_req_op(req->cmd->zms.zsa) | REQ_SYNC;
> +			bio->bi_iter.bi_sector = sector;
> +			bio_set_dev(bio, bdev);
> +			/* This may take a while, so be nice to others */
> +			cond_resched();
> +		}
> +		sector += blk_queue_zone_sectors(q);
> +	}
> +
> +	if (bio) {
> +		ret = submit_bio_wait(bio);
> +		bio_put(bio);
> +	}
> +
> +out:
> +	kfree(d.zbitmap);
> +
> +	return blkdev_zone_mgmt_errno_to_nvme_status(ret);
> +}
> +
> +static u16 nvmet_bdev_execute_zmgmt_send_all(struct nvmet_req *req)
> +{
> +	int ret;
> +
> +	switch (zsa_req_op(req->cmd->zms.zsa)) {
> +	case REQ_OP_ZONE_RESET:
> +		ret = blkdev_zone_mgmt(req->ns->bdev, REQ_OP_ZONE_RESET, 0,
> +				       get_capacity(req->ns->bdev->bd_disk),
> +				       GFP_KERNEL);
> +		if (ret < 0)
> +			return blkdev_zone_mgmt_errno_to_nvme_status(ret);
> +		break;
> +	case REQ_OP_ZONE_OPEN:
> +	case REQ_OP_ZONE_CLOSE:
> +	case REQ_OP_ZONE_FINISH:
> +		return nvmet_bdev_zone_mgmt_emulate_all(req);
> +	default:
> +		/* this is needed to quiet compiler warning */
> +		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, zsa);
> +		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
> +	}
> +
> +	return NVME_SC_SUCCESS;
> +}
> +
> +static void nvmet_bdev_zmgmt_send_work(struct work_struct *w)
> +{
> +	struct nvmet_req *req = container_of(w, struct nvmet_req, z.zmgmt_work);
> +	sector_t sect = nvmet_lba_to_sect(req->ns, req->cmd->zms.slba);
> +	enum req_opf op = zsa_req_op(req->cmd->zms.zsa);
> +	struct block_device *bdev = req->ns->bdev;
> +	sector_t zone_sectors = bdev_zone_sectors(bdev);
> +	u16 status = NVME_SC_SUCCESS;
> +	int ret;
> +
> +	if (op == REQ_OP_LAST) {
> +		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, zsa);
> +		status = NVME_SC_ZONE_INVALID_TRANSITION | NVME_SC_DNR;
> +		goto out;
> +	}
> +
> +	/* when select all bit is set slba field is ignored */
> +	if (req->cmd->zms.select_all) {
> +		status = nvmet_bdev_execute_zmgmt_send_all(req);
> +		goto out;
> +	}
> +
> +	if (sect >= get_capacity(bdev->bd_disk)) {
> +		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, slba);
> +		status = NVME_SC_LBA_RANGE | NVME_SC_DNR;
> +		goto out;
> +	}
> +
> +	if (sect & (zone_sectors - 1)) {
> +		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, slba);
> +		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
> +		goto out;
> +	}
> +
> +	ret = blkdev_zone_mgmt(bdev, op, sect, zone_sectors, GFP_KERNEL);
> +	if (ret < 0)
> +		status = blkdev_zone_mgmt_errno_to_nvme_status(ret);
> +
> +out:
> +	nvmet_req_complete(req, status);
> +}
> +
> +void nvmet_bdev_execute_zone_mgmt_send(struct nvmet_req *req)
> +{
> +	INIT_WORK(&req->z.zmgmt_work, nvmet_bdev_zmgmt_send_work);
> +	queue_work(zbd_wq, &req->z.zmgmt_work);
> +}
> +
> +static void nvmet_bdev_zone_append_bio_done(struct bio *bio)
> +{
> +	struct nvmet_req *req = bio->bi_private;
> +
> +	if (bio->bi_status == BLK_STS_OK) {
> +		req->cqe->result.u64 =
> +			nvmet_sect_to_lba(req->ns, bio->bi_iter.bi_sector);
> +	}
> +
> +	nvmet_req_complete(req, blk_to_nvme_status(req, bio->bi_status));
> +	nvmet_req_bio_put(req, bio);
> +}
> +
> +void nvmet_bdev_execute_zone_append(struct nvmet_req *req)
> +{
> +	sector_t sect = nvmet_lba_to_sect(req->ns, req->cmd->rw.slba);
> +	u16 nr_sect = nvmet_lba_to_sect(req->ns, req->cmd->rw.length);
> +	u16 status = NVME_SC_SUCCESS;
> +	unsigned int total_len = 0;
> +	struct scatterlist *sg;
> +	struct bio *bio;
> +	int sg_cnt;
> +
> +	/* Request is completed on len mismatch in nvmet_check_transter_len() */
> +	if (!nvmet_check_transfer_len(req, nvmet_rw_data_len(req)))
> +		return;
> +
> +	if (!req->sg_cnt) {
> +		nvmet_req_complete(req, 0);
> +		return;
> +	}
> +
> +	if (sect >= get_capacity(req->ns->bdev->bd_disk)) {
> +		req->error_loc = offsetof(struct nvme_rw_command, slba);
> +		status = NVME_SC_LBA_RANGE | NVME_SC_DNR;
> +		goto out;
> +	}
> +
> +	if ((sect + nr_sect > get_capacity(req->ns->bdev->bd_disk))) {
> +		req->error_loc = offsetof(struct nvme_rw_command, length);
> +		status = NVME_SC_LBA_RANGE | NVME_SC_DNR;
> +		goto out;
> +	}
> +
> +	if (sect & (bdev_zone_sectors(req->ns->bdev) - 1)) {
> +		req->error_loc = offsetof(struct nvme_rw_command, slba);
> +		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
> +		goto out;
> +	}
> +
> +	if (nvmet_use_inline_bvec(req)) {
> +		bio = &req->z.inline_bio;
> +		bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec));
> +	} else {
> +		bio = bio_alloc(GFP_KERNEL, req->sg_cnt);
> +	}
> +
> +	bio->bi_opf = REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE;
> +	bio->bi_end_io = nvmet_bdev_zone_append_bio_done;
> +	bio_set_dev(bio, req->ns->bdev);
> +	bio->bi_iter.bi_sector = sect;
> +	bio->bi_private = req;
> +	if (req->cmd->rw.control & cpu_to_le16(NVME_RW_FUA))
> +		bio->bi_opf |= REQ_FUA;
> +
> +	for_each_sg(req->sg, sg, req->sg_cnt, sg_cnt) {
> +		struct page *p = sg_page(sg);
> +		unsigned int l = sg->length;
> +		unsigned int o = sg->offset;
> +		unsigned int ret;
> +
> +		ret = bio_add_zone_append_page(bio, p, l, o);
> +		if (ret != sg->length) {
> +			status = NVME_SC_INTERNAL;
> +			goto out_put_bio;
> +		}
> +		total_len += sg->length;
> +	}
> +
> +	if (total_len != nvmet_rw_data_len(req)) {
> +		status = NVME_SC_INTERNAL | NVME_SC_DNR;
> +		goto out_put_bio;
> +	}
> +
> +	submit_bio(bio);
> +	return;
> +
> +out_put_bio:
> +	nvmet_req_bio_put(req, bio);
> +out:
> +	nvmet_req_complete(req, status);
> +}
> +
> +u16 nvmet_bdev_zns_parse_io_cmd(struct nvmet_req *req)
> +{
> +	struct nvme_command *cmd = req->cmd;
> +
> +	switch (cmd->common.opcode) {
> +	case nvme_cmd_zone_append:
> +		req->execute = nvmet_bdev_execute_zone_append;
> +		return 0;
> +	case nvme_cmd_zone_mgmt_recv:
> +		req->execute = nvmet_bdev_execute_zone_mgmt_recv;
> +		return 0;
> +	case nvme_cmd_zone_mgmt_send:
> +		req->execute = nvmet_bdev_execute_zone_mgmt_send;
> +		return 0;
> +	default:
> +		return nvmet_bdev_parse_io_cmd(req);
> +	}
> +}
> diff --git a/include/linux/nvme.h b/include/linux/nvme.h
> index c7ba83144d52..cb1197f1cfed 100644
> --- a/include/linux/nvme.h
> +++ b/include/linux/nvme.h
> @@ -944,6 +944,13 @@ struct nvme_zone_mgmt_recv_cmd {
>  enum {
>  	NVME_ZRA_ZONE_REPORT		= 0,
>  	NVME_ZRASF_ZONE_REPORT_ALL	= 0,
> +	NVME_ZRASF_ZONE_STATE_EMPTY	= 0x01,
> +	NVME_ZRASF_ZONE_STATE_IMP_OPEN	= 0x02,
> +	NVME_ZRASF_ZONE_STATE_EXP_OPEN	= 0x03,
> +	NVME_ZRASF_ZONE_STATE_CLOSED	= 0x04,
> +	NVME_ZRASF_ZONE_STATE_READONLY	= 0x05,
> +	NVME_ZRASF_ZONE_STATE_FULL	= 0x06,
> +	NVME_ZRASF_ZONE_STATE_OFFLINE	= 0x07,
>  	NVME_REPORT_ZONE_PARTIAL	= 1,
>  };
>  
> 

With the nit above fixed, feel free to add

Reviewed-by: Damien Le Moal <damien.lemoal at wdc.com>


-- 
Damien Le Moal
Western Digital Research


From Amit.Engel at Dell.com  Wed Jun  9 04:14:14 2021
From: Amit.Engel at Dell.com (Engel, Amit)
Date: Wed, 9 Jun 2021 11:14:14 +0000
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at
 0000000000000230
In-Reply-To: <a76ecf8e-15d0-1ebc-aaf2-c93fee591602@grimberg.me>
References: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <CO1PR19MB4885314DCBFEB376C7305C98EE3D9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <e40414a9-41ef-f82c-7d1c-2583d8251c26@grimberg.me>
 <CO1PR19MB48850B44ED23C179A4E4F541EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
 <b9c7e6cc-fea5-b8ce-af07-7d004116e3e3@grimberg.me>
 <CO1PR19MB48859C3254E4F3EA3282EDC9EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
 <a76ecf8e-15d0-1ebc-aaf2-c93fee591602@grimberg.me>
Message-ID: <CO1PR19MB48859E1CCD596402D4007458EE369@CO1PR19MB4885.namprd19.prod.outlook.com>

Correct, free_queue is being called (sock->sk becomes NULL) before restore_sock_calls

When restore_sock_calls is called, we fail on 'write_lock_bh(&sock->sk->sk_callback_lock)' 

NULL pointer dereference at 0x230 ? 560 decimal
crash> struct sock -o
struct sock {
   [0] struct sock_common __sk_common;
   ?
   ...
   ?
   [560] rwlock_t sk_callback_lock;

stop queue in ctx2 does not really do anything since 'NVME_TCP_Q_LIVE' bit is already cleared (by ctx1).
can you please explain how stop the queue before free helps to serialize ctx1 ?

The race we are describing is based on the panic bt that I shared.

maybe our analysis is not accurate? 

Thanks,
Amit

-----Original Message-----
From: Sagi Grimberg <sagi at grimberg.me> 
Sent: Wednesday, June 9, 2021 12:11 PM
To: Engel, Amit; linux-nvme at lists.infradead.org
Cc: Anner, Ran; Grupi, Elad
Subject: Re: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at 0000000000000230


[EXTERNAL EMAIL] 


> Im not sure that using the queue_lock mutex ill help The race in this 
> case is between sock_release and nvme_tcp_restore_sock_calls 
> sock_release is being called as part of nvme_tcp_free_queue which is 
> destroying the mutex

Maybe I'm not understanding the issue here. What is the scenario again?
stop_queue is called (ctx1), that triggers error_recovery (ctx2) which then calls free_queue before ctx1 gets to restore sock callbacks?

err_work will first stop the queues before freeing them, so it will serialize behind ctx1. What am I missing?

From colin.king at canonical.com  Wed Jun  9 04:26:40 2021
From: colin.king at canonical.com (Colin King)
Date: Wed,  9 Jun 2021 12:26:40 +0100
Subject: [PATCH] nvmet-rdma: remove redundant initialization of variable ret
Message-ID: <20210609112640.184904-1-colin.king@canonical.com>

From: Colin Ian King <colin.king at canonical.com>

The variable ret is being initialized with a value that is never read,
it is being updated later on. The assignment is redundant and can be
removed.

Addresses-Coverity: ("Unused value")
Signed-off-by: Colin Ian King <colin.king at canonical.com>
---
 drivers/nvme/target/rdma.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 7d607f435e36..7ef2b55df80c 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -1545,7 +1545,7 @@ static int nvmet_rdma_cm_accept(struct rdma_cm_id *cm_id,
 {
 	struct rdma_conn_param  param = { };
 	struct nvme_rdma_cm_rep priv = { };
-	int ret = -ENOMEM;
+	int ret;
 
 	param.rnr_retry_count = 7;
 	param.flow_control = 1;
-- 
2.31.1



From joshi.k at samsung.com  Wed Jun  9 03:50:50 2021
From: joshi.k at samsung.com (Kanchan Joshi)
Date: Wed,  9 Jun 2021 16:20:50 +0530
Subject: [LSF/MM/BPF Topic] Towards more useful nvme-passthrough
References: <CGME20210609105347epcas5p42ab916655fca311157a38d54f79f95e7@epcas5p4.samsung.com>
Message-ID: <20210609105050.127009-1-joshi.k@samsung.com>

Background & objectives:
------------------------

The NVMe passthrough interface

Good part: allows new device-features to be usable (at least in raw
form) without having to build block-generic cmds, in-kernel users,
emulations and file-generic user-interfaces - all this take some time to
evolve.

Bad part: passthrough interface has remain tied to synchronous ioctl,
which is a blocker for performance-centric usage scenarios. User-space
can take the pain of implementing async-over-sync on its own but it does
not make much sense in a world that already has io_uring.

Passthrough is lean in the sense it cuts through layers of abstractions
and reaches to NVMe fast. One of the objective here is to build a
scalable pass-through that can be readily used to play with new/emerging
NVMe features.  Another is to surpass/match existing raw/direct block
I/O performance with this new in-kernel path.

Recent developments:
--------------------
- NVMe now has a per-namespace char interface that remains available/usable
  even for unsupported features and for new command-sets [1].

- Jens has proposed async-ioctl like facility 'uring_cmd' in io_uring. This
  introduces new possibilities (beyond storage); async-passthrough is one of
those. Last posted version is V4 [2].

- I have posted work on async nvme passthrough over block-dev [3]. Posted work
  is in V4 (in sync with the infra of [2]).

Early performance numbers:
--------------------------
fio, randread, 4k bs, 1 job
Kiops, with varying QD:

QD      Sync-PT         io_uring        Async-PT
1         10.8            10.6            10.6
2         10.9            24.5            24
4         10.6            45              46
8         10.9            90              89
16        11.0            169             170
32        10.6            308             307
64        10.8            503             506
128       10.9            592             596

Further steps/discussion points:
--------------------------------
1.Async-passthrough over nvme char-dev
It is in a shape to receive feedback, but I am not sure if community
would like to take a look at that before settling on uring-cmd infra.

2.Once above gets in shape, bring other perf-centric features of io_uring to
this path -
A. SQPoll and register-file: already functional.
B. Passthrough polling: This can be enabled for block and looks feasible for
char-interface as well.  Keith recently posted enabling polling for user
pass-through [4]
C. Pre-mapped buffers: Early thought is to let the buffers registered by
io_uring, and add a new passthrough ioctl/uring_cmd in driver which does
everything that passthrough does except pinning/unpinning the pages.

3. Are there more things in the "io_uring->nvme->[block-layer]->nvme" path
which can be optimized.

Ideally I'd like to cover good deal of ground before Dec. But there seems
plenty of possibilities on this path.  Discussion would help in how best to
move forward, and cement the ideas.

[1] https://lore.kernel.org/linux-nvme/20210421074504.57750-1-minwoo.im.dev at gmail.com/
[2] https://lore.kernel.org/linux-nvme/20210317221027.366780-1-axboe at kernel.dk/
[3] https://lore.kernel.org/linux-nvme/20210325170540.59619-1-joshi.k at samsung.com/
[4] https://lore.kernel.org/linux-block/20210517171443.GB2709391 at dhcp-10-100-145-180.wdc.com/#t

-- 
2.25.1



From smalin at marvell.com  Wed Jun  9 05:13:41 2021
From: smalin at marvell.com (Shai Malin)
Date: Wed, 9 Jun 2021 12:13:41 +0000
Subject: please drop the nvme code from net-next
In-Reply-To: <SJ0PR18MB3882C20793EA35A3E8DAE300CC379@SJ0PR18MB3882.namprd18.prod.outlook.com>
References: <SJ0PR18MB3882C20793EA35A3E8DAE300CC379@SJ0PR18MB3882.namprd18.prod.outlook.com>
Message-ID: <SJ0PR18MB3882585CE0FE3340D37A8399CC369@SJ0PR18MB3882.namprd18.prod.outlook.com>

On Tue, Jun 08, 2021 at 10:51:00PM +0200, Shai Malin wrote:
> On Tue, Jun 08, 2021 at 10:41:00PM -0700, Keith Busch wrote:
> > On Tue, Jun 08, 2021 at 12:08:39PM -0700, David Miller wrote:
> > > From: Christoph Hellwig <hch at lst.de>
> > > Date: Tue, 8 Jun 2021 15:43:03 +0200
> > >
> > > > please drop the nvme-offload code from net-next.  Code for drivers/nvme/
> > > > needs ACKs from us nvme maintainers and for something this significant
> > > > also needs to go through the NVMe tree.  And this code is not ready yet.
> > >
> > > Please send me a revert, and I will apply it, thank you.
> > >
> > > It's tricky because at least one driver uses the new interfaces.
> >
> > Shouldn't whoever merged un-ACK'ed patches from a different subsystem
> > get to own the tricky revert?
> 
> Dave, we will provide a revert patch.

The revert patch was sent to net-next.
https://lore.kernel.org/netdev/20210609104918.10329-1-smalin at marvell.com/


From hch at lst.de  Wed Jun  9 05:58:56 2021
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 9 Jun 2021 14:58:56 +0200
Subject: [linux-nvme:nvme-5.14 27/28] drivers/nvme/host/pci.c:539:7:
 error: implicit declaration of function 'nvme_ctrl_sgl_supported'
In-Reply-To: <E9067936-7A06-4EFF-A4A9-9405612333B6@wdc.com>
References: <202106090556.FQr8aEIa-lkp@intel.com>
 <E9067936-7A06-4EFF-A4A9-9405612333B6@wdc.com>
Message-ID: <20210609125856.GA1837@lst.de>

I'll drop the patch for now.


From hch at lst.de  Wed Jun  9 05:59:48 2021
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 9 Jun 2021 14:59:48 +0200
Subject: [PATCH RESEND 1/2] nvme-fc: add a helper to check ctrl sgl support
In-Reply-To: <20210608191037.44251-1-chaitanya.kulkarni@wdc.com>
References: <20210608191037.44251-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210609125948.GB1837@lst.de>

On Tue, Jun 08, 2021 at 12:10:36PM -0700, Chaitanya Kulkarni wrote:
> +static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
> +{
> +	if (!(ctrl->sgls & ((1 << 0) | (1 << 1))))
> +		return true;
> +
> +	return false;
> +}

Please move this into a separate patch as it isn't FC-related.


From hch at lst.de  Wed Jun  9 06:00:25 2021
From: hch at lst.de (Christoph Hellwig)
Date: Wed, 9 Jun 2021 15:00:25 +0200
Subject: [PATCH v6 1/2] ACPI: Move check for _DSD StorageD3Enable
 property to acpi
In-Reply-To: <20210608200101.GB339628@dhcp-10-100-145-180.wdc.com>
References: <20210607173156.5548-1-mario.limonciello@amd.com>
 <20210608053546.GA14116@lst.de>
 <20210608200101.GB339628@dhcp-10-100-145-180.wdc.com>
Message-ID: <20210609130025.GC1837@lst.de>

On Tue, Jun 08, 2021 at 01:01:01PM -0700, Keith Busch wrote:
> On Tue, Jun 08, 2021 at 07:35:46AM +0200, Christoph Hellwig wrote:
> > On Mon, Jun 07, 2021 at 12:31:55PM -0500, Mario Limonciello wrote:
> > > +/**
> > > + * acpi_storage_d3 - Check if a storage device should use D3.
> > > + * @dev: Device to check
> > > + *
> > > + * Returns %true if @dev should be put into D3 when the ->suspend method is
> > > + * called, else %false.  The name of this function is somewhat misleading
> > > + * as it has nothing to do with storage except for the name of the ACPI
> > > + * property.  On some platforms resume will not work if this hint is ignored.
> > > + *
> > > + */
> > 
> > I still do not like this comment.  There is nothing about only using this
> > for storage devices.  It is specific to a PCIe slot, and if I plug
> > something that is not a storage device into it the same restrictions
> > still apply.
> 
> Hm, I thought this comment was good. Despite the dupious connection to
> storage, the function name matches the ACPI property, so an explanation
> for this misnomer seems appropriate.

I suggested a comment earlier that explains the connection without being
completely misleading.


From rafael at kernel.org  Wed Jun  9 06:01:01 2021
From: rafael at kernel.org (Rafael J. Wysocki)
Date: Wed, 9 Jun 2021 15:01:01 +0200
Subject: [PATCH v7 1/2] ACPI: Move check for _DSD StorageD3Enable property
 to acpi
In-Reply-To: <20210608154255.8693-1-mario.limonciello@amd.com>
References: <20210608154255.8693-1-mario.limonciello@amd.com>
Message-ID: <CAJZ5v0ik+kEn9eZ_g4=i=qW_08p2-PekT2tW4AivwUNc-NmHiA@mail.gmail.com>

On Tue, Jun 8, 2021 at 5:43 PM Mario Limonciello
<mario.limonciello at amd.com> wrote:
>
> Although first implemented for NVME, this check may be usable by
> other drivers as well. Microsoft's specification explicitly mentions
> that is may be usable by SATA and AHCI devices.  Google also indicates
> that they have used this with SDHCI in a downstream kernel tree that
> a user can plug a storage device into.
>
> Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Suggested-by: Keith Busch <kbusch at kernel.org>
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> ---
>  drivers/acpi/device_pm.c | 25 +++++++++++++++++++++++++
>  drivers/nvme/host/pci.c  | 28 +---------------------------

This appears to be based on an nvme commit that is not there in the
mainline, so I cannot apply it as is.

If you change the subject to something like

"ACPI: Check StorageD3Enable _DSD property in ACPI code"

and modify the acpi_storage_d3() kerneldoc as per the remark below,
please feel free to add

Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki at intel.com>

to it and route it via nvme.

>  include/linux/acpi.h     |  5 +++++
>  3 files changed, 31 insertions(+), 27 deletions(-)
>
> Changes from v4->v5:
>  * Correct extra "Link:" word in commit message
> Changes from v5->v6:
>  * Add in commit message tags from Raul, Rafael and Keith
> Changes from v6->v7:
>  * Update kerneldoc comments per suggestsions.
>  * Drop all Acked/Reviewed tags
>
> diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
> index d260bc1f3e6e..c45f2d76b67e 100644
> --- a/drivers/acpi/device_pm.c
> +++ b/drivers/acpi/device_pm.c
> @@ -1340,4 +1340,29 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on)
>         return 1;
>  }
>  EXPORT_SYMBOL_GPL(acpi_dev_pm_attach);
> +
> +/**
> + * acpi_storage_d3 - Check if a storage device should use D3 in the suspend path

Please change this to something like "Check if D3 should be used in
the suspend path" as this generally isn't about storage only.

> + * @dev: Device to check
> + *
> + * Return %true if the platform firmware wants @dev to be programmed
> + * into D3hot or D3cold (if supported) in the suspend path, or %false
> + * when there is no specific preference. On some platforms, if this
> + * hint is ignored, @dev may remain unresponsive after suspending the
> + * platform as a whole.
> + */
> +bool acpi_storage_d3(struct device *dev)
> +{
> +       struct acpi_device *adev = ACPI_COMPANION(dev);
> +       u8 val;
> +
> +       if (!adev)
> +               return false;
> +       if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> +                       &val))
> +               return false;
> +       return val == 1;
> +}
> +EXPORT_SYMBOL_GPL(acpi_storage_d3);
> +
>  #endif /* CONFIG_PM */
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 3aa7245a505f..8fbc4c87a0d8 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -2828,32 +2828,6 @@ static unsigned long check_vendor_combination_bug(struct pci_dev *pdev)
>         return 0;
>  }
>
> -#ifdef CONFIG_ACPI
> -static bool nvme_acpi_storage_d3(struct pci_dev *dev)
> -{
> -       struct acpi_device *adev = ACPI_COMPANION(&dev->dev);
> -       u8 val;
> -
> -       /*
> -        * Look for _DSD property specifying that the storage device on the port
> -        * must use D3 to support deep platform power savings during
> -        * suspend-to-idle.
> -        */
> -
> -       if (!adev)
> -               return false;
> -       if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> -                       &val))
> -               return false;
> -       return val == 1;
> -}
> -#else
> -static inline bool nvme_acpi_storage_d3(struct pci_dev *dev)
> -{
> -       return false;
> -}
> -#endif /* CONFIG_ACPI */
> -
>  static void nvme_async_probe(void *data, async_cookie_t cookie)
>  {
>         struct nvme_dev *dev = data;
> @@ -2903,7 +2877,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>
>         quirks |= check_vendor_combination_bug(pdev);
>
> -       if (!noacpi && nvme_acpi_storage_d3(pdev)) {
> +       if (!noacpi && acpi_storage_d3(&pdev->dev)) {
>                 /*
>                  * Some systems use a bios work around to ask for D3 on
>                  * platforms that support kernel managed suspend.
> diff --git a/include/linux/acpi.h b/include/linux/acpi.h
> index c60745f657e9..dd0dafd21e33 100644
> --- a/include/linux/acpi.h
> +++ b/include/linux/acpi.h
> @@ -1004,6 +1004,7 @@ int acpi_dev_resume(struct device *dev);
>  int acpi_subsys_runtime_suspend(struct device *dev);
>  int acpi_subsys_runtime_resume(struct device *dev);
>  int acpi_dev_pm_attach(struct device *dev, bool power_on);
> +bool acpi_storage_d3(struct device *dev);
>  #else
>  static inline int acpi_subsys_runtime_suspend(struct device *dev) { return 0; }
>  static inline int acpi_subsys_runtime_resume(struct device *dev) { return 0; }
> @@ -1011,6 +1012,10 @@ static inline int acpi_dev_pm_attach(struct device *dev, bool power_on)
>  {
>         return 0;
>  }
> +static inline bool acpi_storage_d3(struct device *dev)
> +{
> +       return false;
> +}
>  #endif
>
>  #if defined(CONFIG_ACPI) && defined(CONFIG_PM_SLEEP)
> --
> 2.25.1
>


From rafael at kernel.org  Wed Jun  9 06:01:59 2021
From: rafael at kernel.org (Rafael J. Wysocki)
Date: Wed, 9 Jun 2021 15:01:59 +0200
Subject: [PATCH v7 2/2] ACPI: Add quirks for AMD Renoir/Lucienne CPUs to
 force the D3 hint
In-Reply-To: <20210608154255.8693-2-mario.limonciello@amd.com>
References: <20210608154255.8693-1-mario.limonciello@amd.com>
 <20210608154255.8693-2-mario.limonciello@amd.com>
Message-ID: <CAJZ5v0hOgQCyEaRWEM=wELSYKJyuvCi8r4EnaAFQv+5EQRcOJg@mail.gmail.com>

On Tue, Jun 8, 2021 at 5:43 PM Mario Limonciello
<mario.limonciello at amd.com> wrote:
>
> AMD systems from Renoir and Lucienne require that the NVME controller
> is put into D3 over a Modern Standby / suspend-to-idle
> cycle.  This is "typically" accomplished using the `StorageD3Enable`
> property in the _DSD, but this property was introduced after many
> of these systems launched and most OEM systems don't have it in
> their BIOS.
>
> On AMD Renoir without these drives going into D3 over suspend-to-idle
> the resume will fail with the NVME controller being reset and a trace
> like this in the kernel logs:
> ```
> [   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
> [   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
> [   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
> [   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
> [   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
> [   95.332843] nvme nvme0: Abort status: 0x371
> [   95.332852] nvme nvme0: Abort status: 0x371
> [   95.332856] nvme nvme0: Abort status: 0x371
> [   95.332859] nvme nvme0: Abort status: 0x371
> [   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
> [   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
> ```
>
> The Microsoft documentation for StorageD3Enable mentioned that Windows has
> a hardcoded allowlist for D3 support, which was used for these platforms.
> Introduce quirks to hardcode them for Linux as well.
>
> As this property is now "standardized", OEM systems using AMD Cezanne and
> newer APU's have adopted this property, and quirks like this should not be
> necessary.
>
> CC: Julian Sikorski <belegdol at gmail.com>
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Rafael J. Wysocki <rjw at rjwysocki.net>
> CC: Prike Liang <prike.liang at amd.com>
> Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>

Acked-by: Rafael J. Wysocki <rafael.j.wysocki at intel.com>

> ---
>  drivers/acpi/device_pm.c |  3 +++
>  drivers/acpi/internal.h  |  9 +++++++++
>  drivers/acpi/x86/utils.c | 25 +++++++++++++++++++++++++
>  3 files changed, 37 insertions(+)
>
> Changes from v4->v5:
>  * Add this patch back in as it's been made apparent that the
>    system needs to be hardcoded for these.
>    Changes:
>    - Drop Cezanne - it's now covered by StorageD3Enable
>    - Rebase ontop of acpi_storage_d3 outside of NVME
> Changes from v5->v6:
>  * Move the quirk check into drivers/acpi/x86/ as suggested by
>    Rafael.
> Changes from v6->v7:
>  * Move header location
>  * Optimization of force function
>
> diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
> index c45f2d76b67e..31e0025a913e 100644
> --- a/drivers/acpi/device_pm.c
> +++ b/drivers/acpi/device_pm.c
> @@ -1356,6 +1356,9 @@ bool acpi_storage_d3(struct device *dev)
>         struct acpi_device *adev = ACPI_COMPANION(dev);
>         u8 val;
>
> +       if (force_storage_d3())
> +               return true;
> +
>         if (!adev)
>                 return false;
>         if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> diff --git a/drivers/acpi/internal.h b/drivers/acpi/internal.h
> index f973bbe90e5e..e29ec463bb07 100644
> --- a/drivers/acpi/internal.h
> +++ b/drivers/acpi/internal.h
> @@ -236,6 +236,15 @@ static inline int suspend_nvs_save(void) { return 0; }
>  static inline void suspend_nvs_restore(void) {}
>  #endif
>
> +#ifdef CONFIG_X86
> +bool force_storage_d3(void);
> +#else
> +static inline bool force_storage_d3(void)
> +{
> +       return false;
> +}
> +#endif
> +
>  /*--------------------------------------------------------------------------
>                                 Device properties
>    -------------------------------------------------------------------------- */
> diff --git a/drivers/acpi/x86/utils.c b/drivers/acpi/x86/utils.c
> index bdc1ba00aee9..5298bb4d81fe 100644
> --- a/drivers/acpi/x86/utils.c
> +++ b/drivers/acpi/x86/utils.c
> @@ -135,3 +135,28 @@ bool acpi_device_always_present(struct acpi_device *adev)
>
>         return ret;
>  }
> +
> +/*
> + * AMD systems from Renoir and Lucienne *require* that the NVME controller
> + * is put into D3 over a Modern Standby / suspend-to-idle cycle.
> + *
> + * This is "typically" accomplished using the `StorageD3Enable`
> + * property in the _DSD that is checked via the `acpi_storage_d3` function
> + * but this property was introduced after many of these systems launched
> + * and most OEM systems don't have it in their BIOS.
> + *
> + * The Microsoft documentation for StorageD3Enable mentioned that Windows has
> + * a hardcoded allowlist for D3 support, which was used for these platforms.
> + *
> + * This allows quirking on Linux in a similar fashion.
> + */
> +const struct x86_cpu_id storage_d3_cpu_ids[] = {
> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),  /* Renoir */
> +       X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL), /* Lucienne */
> +       {}
> +};
> +
> +bool force_storage_d3(void)
> +{
> +       return x86_match_cpu(storage_d3_cpu_ids);
> +}
> --
> 2.25.1
>


From hare at suse.de  Wed Jun  9 08:01:18 2021
From: hare at suse.de (Hannes Reinecke)
Date: Wed,  9 Jun 2021 17:01:18 +0200
Subject: [PATCHv6] nvme: allow to re-attach namespaces after all paths are down
Message-ID: <20210609150118.130650-1-hare@suse.de>

We should only remove the ns head from the list of heads per
subsystem if the reference count drops to zero. That cleans up
reference counting, and allows us to call del_gendisk() once the last
path is removed (as then the ns_head should be removed anyway).
As this introduces a (theoretical) race condition where I/O might have
been requeued before the last path went down we also should be checking
if the gendisk is still present in nvme_ns_head_submit_bio(),
and failing I/O if so.

Changes to v5:
- Synchronize between nvme_init_ns_head() and nvme_mpath_check_last_path()
- Check for removed gendisk in nvme_ns_head_submit_bio()
Changes to v4:
- Call del_gendisk() in nvme_mpath_check_last_path() to avoid deadlock
Changes to v3:
- Simplify if() clause to detect duplicate namespaces
Changes to v2:
- Drop memcpy() statement
Changes to v1:
- Always check NSIDs after reattach

Signed-off-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/core.c      |  9 ++++-----
 drivers/nvme/host/multipath.c | 30 +++++++++++++++++++++++++-----
 drivers/nvme/host/nvme.h      | 11 ++---------
 3 files changed, 31 insertions(+), 19 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 177cae44b612..6d7c2958b3e2 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -566,6 +566,9 @@ static void nvme_free_ns_head(struct kref *ref)
 	struct nvme_ns_head *head =
 		container_of(ref, struct nvme_ns_head, ref);
 
+	mutex_lock(&head->subsys->lock);
+	list_del_init(&head->entry);
+	mutex_unlock(&head->subsys->lock);
 	nvme_mpath_remove_disk(head);
 	ida_simple_remove(&head->subsys->ns_ida, head->instance);
 	cleanup_srcu_struct(&head->srcu);
@@ -3806,8 +3809,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid,
  out_unlink_ns:
 	mutex_lock(&ctrl->subsys->lock);
 	list_del_rcu(&ns->siblings);
-	if (list_empty(&ns->head->list))
-		list_del_init(&ns->head->entry);
 	mutex_unlock(&ctrl->subsys->lock);
 	nvme_put_ns_head(ns->head);
  out_free_queue:
@@ -3828,8 +3829,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	mutex_lock(&ns->ctrl->subsys->lock);
 	list_del_rcu(&ns->siblings);
-	if (list_empty(&ns->head->list))
-		list_del_init(&ns->head->entry);
 	mutex_unlock(&ns->ctrl->subsys->lock);
 
 	synchronize_rcu(); /* guarantee not available in head->list */
@@ -3849,7 +3848,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	list_del_init(&ns->list);
 	up_write(&ns->ctrl->namespaces_rwsem);
 
-	nvme_mpath_check_last_path(ns);
+	nvme_mpath_check_last_path(ns->head);
 	nvme_put_ns(ns);
 }
 
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 23573fe3fc7d..31153f6ec582 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -266,6 +266,8 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
 	int node = numa_node_id();
 	struct nvme_ns *ns;
 
+	if (!(head->disk->flags & GENHD_FL_UP))
+		return NULL;
 	ns = srcu_dereference(head->current_path[node], &head->srcu);
 	if (unlikely(!ns))
 		return __nvme_find_path(head, node);
@@ -281,6 +283,8 @@ static bool nvme_available_path(struct nvme_ns_head *head)
 {
 	struct nvme_ns *ns;
 
+	if (!(head->disk->flags & GENHD_FL_UP))
+		return false;
 	list_for_each_entry_rcu(ns, &head->list, siblings) {
 		if (test_bit(NVME_CTRL_FAILFAST_EXPIRED, &ns->ctrl->flags))
 			continue;
@@ -771,20 +775,36 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id)
 #endif
 }
 
-void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+void nvme_mpath_check_last_path(struct nvme_ns_head *head)
 {
+	bool last_path = false;
 	if (!head->disk)
 		return;
-	if (head->disk->flags & GENHD_FL_UP) {
-		nvme_cdev_del(&head->cdev, &head->cdev_device);
-		del_gendisk(head->disk);
+
+	/* Synchronize with nvme_init_ns_head() */
+	mutex_lock(&head->subsys->lock);
+	if (list_empty(&head->list))
+		last_path = true;
+	mutex_unlock(&head->subsys->lock);
+	if (last_path) {
+		kblockd_schedule_work(&head->requeue_work);
+		if (head->disk->flags & GENHD_FL_UP) {
+			nvme_cdev_del(&head->cdev, &head->cdev_device);
+			del_gendisk(head->disk);
+		}
 	}
+}
+
+void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+{
+	if (!head->disk)
+		return;
 	blk_set_queue_dying(head->disk->queue);
 	/* make sure all pending bios are cleaned up */
 	kblockd_schedule_work(&head->requeue_work);
 	flush_work(&head->requeue_work);
 	blk_cleanup_queue(head->disk->queue);
-	if (!test_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
+	if (!test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
 		/*
 		 * if device_add_disk wasn't called, prevent
 		 * disk release to put a bogus reference on the
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1f397ecba16c..812fc1d273e3 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -716,14 +716,7 @@ void nvme_mpath_uninit(struct nvme_ctrl *ctrl);
 void nvme_mpath_stop(struct nvme_ctrl *ctrl);
 bool nvme_mpath_clear_current_path(struct nvme_ns *ns);
 void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl);
-
-static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)
-{
-	struct nvme_ns_head *head = ns->head;
-
-	if (head->disk && list_empty(&head->list))
-		kblockd_schedule_work(&head->requeue_work);
-}
+void nvme_mpath_check_last_path(struct nvme_ns_head *head);
 
 static inline void nvme_trace_bio_complete(struct request *req)
 {
@@ -772,7 +765,7 @@ static inline bool nvme_mpath_clear_current_path(struct nvme_ns *ns)
 static inline void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl)
 {
 }
-static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)
+static inline void nvme_mpath_check_last_path(struct nvme_ns_head *head)
 {
 }
 static inline void nvme_trace_bio_complete(struct request *req)
-- 
2.26.2



From James.Bottomley at HansenPartnership.com  Wed Jun  9 11:36:41 2021
From: James.Bottomley at HansenPartnership.com (James Bottomley)
Date: Wed, 09 Jun 2021 11:36:41 -0700
Subject: [LSF/MM/BPF TOPIC] block namespaces
In-Reply-To: <a189ec50-4c11-9ee9-0b9e-b492507adc1e@suse.de>
References: <a189ec50-4c11-9ee9-0b9e-b492507adc1e@suse.de>
Message-ID: <485837f392401bf35fb7fc8231d7a051f47b53d7.camel@HansenPartnership.com>

On Thu, 2021-05-27 at 10:01 +0200, Hannes Reinecke wrote:
> Hi all,
> 
> I guess it's time to tick off yet another item on my long-term to-do
> list:
> 
> Block namespaces
> ----------------
> 
> Idea is similar to what network already does: allowing each user
> namespace to have a different 'view' on the existing block devices.
> EG if the admin creates a ramdisk in one namespace this device should
> not be visible to other namespaces.
> But for me the most important use-case would be qemu; currently the
> devices need to be set up in the host, even though the host has no
> business touching it as they really belong to the qemu instance. This
> is causing quite some irritation eg when this device has LVM or MD
> metadata and udev is trying to activate it on the host.

I suppose the first question is "why block only?"  There are several
existing device namespace proposals which would be more generic.

> Overall plan is to restrict views of '/dev', '/sys/dev/block' and
> '/sys/block' to only present the devices 'visible' for this
> namespace.

We actually already have a devices cgroup that does some of this:

https://www.kernel.org/doc/Documentation/cgroup-v1/devices.txt

However, visibility isn't the only problem, for direct passthrough
there's also uevent handling and people have even asked about module
loading.

>  Initially the drivers would keep their global enumeration, but plan
> is to make the drivers namespace-aware, too, such that each namespace
> could have its own driver-specific device enumeration.

I really wouldn't do this.  Namespace/Cgroup separation should be kept
as high as possible.  If it leaks into the drivers it will become
unmaintainable.  Why do you think you need the drivers to be aware?  If
it's just enumeration, that should all be doable with the visibility
driver unless you want to do things like compact numbering?

> Goal of this topic is to get a consensus on whether block namespaces
> are a feature which would find interest, and also to discuss some
> design details here:
> - Only in certain cases can a namespace be assigned (eg by calling
> 'modprobe', starting iscsiadm, or calling nvme-cli); how do we handle
> devices for which no namespace can be identified?
> - Shall we allow for different device enumeration per namespace?
> - Into which level should we go with hiding sysfs structures?
>   Is blanking out the higher-level interfaces in /dev and /sys/block
>   enough?

First question is does the device cgroup do enough for you and if not
what's missing?

James




From mario.limonciello at amd.com  Wed Jun  9 11:40:17 2021
From: mario.limonciello at amd.com (Mario Limonciello)
Date: Wed,  9 Jun 2021 13:40:17 -0500
Subject: [PATCH v8 1/2] ACPI: Check StorageD3Enable _DSD property in ACPI code
Message-ID: <20210609184018.26017-1-mario.limonciello@amd.com>

Although first implemented for NVME, this check may be usable by
other drivers as well. Microsoft's specification explicitly mentions
that is may be usable by SATA and AHCI devices.  Google also indicates
that they have used this with SDHCI in a downstream kernel tree that
a user can plug a storage device into.

Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
Suggested-by: Keith Busch <kbusch at kernel.org>
CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
CC: Alexander Deucher <Alexander.Deucher at amd.com>
CC: Rafael J. Wysocki <rjw at rjwysocki.net>
CC: Prike Liang <prike.liang at amd.com>
Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki at intel.com>
---
 drivers/acpi/device_pm.c | 29 +++++++++++++++++++++++++++++
 drivers/nvme/host/pci.c  | 28 +---------------------------
 include/linux/acpi.h     |  5 +++++
 3 files changed, 35 insertions(+), 27 deletions(-)

Changes from v4->v5:
 * Correct extra "Link:" word in commit message
Changes from v5->v6:
 * Add in commit message tags from Raul, Rafael and Keith
Changes from v6->v7:
 * Update kerneldoc comments per suggestsions.
 * Drop all Acked/Reviewed tags
Changes from v7->v8
 * Add Rafael's tag and corrections for title and doc
 * Add in comment requested by Christoph for non-storage to kernel-doc
diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index d260bc1f3e6e..c8600978b283 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -1340,4 +1340,33 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on)
 	return 1;
 }
 EXPORT_SYMBOL_GPL(acpi_dev_pm_attach);
+
+/**
+ * acpi_storage_d3 - Check if D3 should be used in the suspend path
+ * @dev: Device to check
+ *
+ * Return %true if the platform firmware wants @dev to be programmed
+ * into D3hot or D3cold (if supported) in the suspend path, or %false
+ * when there is no specific preference. On some platforms, if this
+ * hint is ignored, @dev may remain unresponsive after suspending the
+ * platform as a whole.
+ *
+ * Although the property has storage in the name it actually is
+ * applied to the PCIe slot and plugging in a non-storage device the
+ * same platform restrictions will likely apply.
+ */
+bool acpi_storage_d3(struct device *dev)
+{
+	struct acpi_device *adev = ACPI_COMPANION(dev);
+	u8 val;
+
+	if (!adev)
+		return false;
+	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
+			&val))
+		return false;
+	return val == 1;
+}
+EXPORT_SYMBOL_GPL(acpi_storage_d3);
+
 #endif /* CONFIG_PM */
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..8fbc4c87a0d8 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2828,32 +2828,6 @@ static unsigned long check_vendor_combination_bug(struct pci_dev *pdev)
 	return 0;
 }
 
-#ifdef CONFIG_ACPI
-static bool nvme_acpi_storage_d3(struct pci_dev *dev)
-{
-	struct acpi_device *adev = ACPI_COMPANION(&dev->dev);
-	u8 val;
-
-	/*
-	 * Look for _DSD property specifying that the storage device on the port
-	 * must use D3 to support deep platform power savings during
-	 * suspend-to-idle.
-	 */
-
-	if (!adev)
-		return false;
-	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
-			&val))
-		return false;
-	return val == 1;
-}
-#else
-static inline bool nvme_acpi_storage_d3(struct pci_dev *dev)
-{
-	return false;
-}
-#endif /* CONFIG_ACPI */
-
 static void nvme_async_probe(void *data, async_cookie_t cookie)
 {
 	struct nvme_dev *dev = data;
@@ -2903,7 +2877,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	quirks |= check_vendor_combination_bug(pdev);
 
-	if (!noacpi && nvme_acpi_storage_d3(pdev)) {
+	if (!noacpi && acpi_storage_d3(&pdev->dev)) {
 		/*
 		 * Some systems use a bios work around to ask for D3 on
 		 * platforms that support kernel managed suspend.
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index c60745f657e9..dd0dafd21e33 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -1004,6 +1004,7 @@ int acpi_dev_resume(struct device *dev);
 int acpi_subsys_runtime_suspend(struct device *dev);
 int acpi_subsys_runtime_resume(struct device *dev);
 int acpi_dev_pm_attach(struct device *dev, bool power_on);
+bool acpi_storage_d3(struct device *dev);
 #else
 static inline int acpi_subsys_runtime_suspend(struct device *dev) { return 0; }
 static inline int acpi_subsys_runtime_resume(struct device *dev) { return 0; }
@@ -1011,6 +1012,10 @@ static inline int acpi_dev_pm_attach(struct device *dev, bool power_on)
 {
 	return 0;
 }
+static inline bool acpi_storage_d3(struct device *dev)
+{
+	return false;
+}
 #endif
 
 #if defined(CONFIG_ACPI) && defined(CONFIG_PM_SLEEP)
-- 
2.25.1



From mario.limonciello at amd.com  Wed Jun  9 11:40:18 2021
From: mario.limonciello at amd.com (Mario Limonciello)
Date: Wed,  9 Jun 2021 13:40:18 -0500
Subject: [PATCH v8 2/2] ACPI: Add quirks for AMD Renoir/Lucienne CPUs to force
 the D3 hint
In-Reply-To: <20210609184018.26017-1-mario.limonciello@amd.com>
References: <20210609184018.26017-1-mario.limonciello@amd.com>
Message-ID: <20210609184018.26017-2-mario.limonciello@amd.com>

AMD systems from Renoir and Lucienne require that the NVME controller
is put into D3 over a Modern Standby / suspend-to-idle
cycle.  This is "typically" accomplished using the `StorageD3Enable`
property in the _DSD, but this property was introduced after many
of these systems launched and most OEM systems don't have it in
their BIOS.

On AMD Renoir without these drives going into D3 over suspend-to-idle
the resume will fail with the NVME controller being reset and a trace
like this in the kernel logs:
```
[   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
[   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
[   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
[   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
[   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
[   95.332843] nvme nvme0: Abort status: 0x371
[   95.332852] nvme nvme0: Abort status: 0x371
[   95.332856] nvme nvme0: Abort status: 0x371
[   95.332859] nvme nvme0: Abort status: 0x371
[   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
[   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
```

The Microsoft documentation for StorageD3Enable mentioned that Windows has
a hardcoded allowlist for D3 support, which was used for these platforms.
Introduce quirks to hardcode them for Linux as well.

As this property is now "standardized", OEM systems using AMD Cezanne and
newer APU's have adopted this property, and quirks like this should not be
necessary.

CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
CC: Alexander Deucher <Alexander.Deucher at amd.com>
CC: Prike Liang <prike.liang at amd.com>
Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
Acked-by: Rafael J. Wysocki <rafael.j.wysocki at intel.com>
Tested-by: Julian Sikorski <belegdol at gmail.com>
---
 drivers/acpi/device_pm.c |  3 +++
 drivers/acpi/internal.h  |  9 +++++++++
 drivers/acpi/x86/utils.c | 25 +++++++++++++++++++++++++
 3 files changed, 37 insertions(+)
 
Changes from v4->v5:
 * Add this patch back in as it's been made apparent that the
   system needs to be hardcoded for these.
   Changes:
   - Drop Cezanne - it's now covered by StorageD3Enable
   - Rebase ontop of acpi_storage_d3 outside of NVME
Changes from v5->v6:
 * Move the quirk check into drivers/acpi/x86/ as suggested by
   Rafael.
Changes from v6->v7:
 * Move header location
 * Optimization of force function
Changes from v7->v8:
 * Add tags

diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index c8600978b283..c40fc135ff0c 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -1360,6 +1360,9 @@ bool acpi_storage_d3(struct device *dev)
 	struct acpi_device *adev = ACPI_COMPANION(dev);
 	u8 val;
 
+	if (force_storage_d3())
+		return true;
+
 	if (!adev)
 		return false;
 	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
diff --git a/drivers/acpi/internal.h b/drivers/acpi/internal.h
index f973bbe90e5e..e29ec463bb07 100644
--- a/drivers/acpi/internal.h
+++ b/drivers/acpi/internal.h
@@ -236,6 +236,15 @@ static inline int suspend_nvs_save(void) { return 0; }
 static inline void suspend_nvs_restore(void) {}
 #endif
 
+#ifdef CONFIG_X86
+bool force_storage_d3(void);
+#else
+static inline bool force_storage_d3(void)
+{
+	return false;
+}
+#endif
+
 /*--------------------------------------------------------------------------
 				Device properties
   -------------------------------------------------------------------------- */
diff --git a/drivers/acpi/x86/utils.c b/drivers/acpi/x86/utils.c
index bdc1ba00aee9..5298bb4d81fe 100644
--- a/drivers/acpi/x86/utils.c
+++ b/drivers/acpi/x86/utils.c
@@ -135,3 +135,28 @@ bool acpi_device_always_present(struct acpi_device *adev)
 
 	return ret;
 }
+
+/*
+ * AMD systems from Renoir and Lucienne *require* that the NVME controller
+ * is put into D3 over a Modern Standby / suspend-to-idle cycle.
+ *
+ * This is "typically" accomplished using the `StorageD3Enable`
+ * property in the _DSD that is checked via the `acpi_storage_d3` function
+ * but this property was introduced after many of these systems launched
+ * and most OEM systems don't have it in their BIOS.
+ *
+ * The Microsoft documentation for StorageD3Enable mentioned that Windows has
+ * a hardcoded allowlist for D3 support, which was used for these platforms.
+ *
+ * This allows quirking on Linux in a similar fashion.
+ */
+const struct x86_cpu_id storage_d3_cpu_ids[] = {
+	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),	/* Renoir */
+	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL),	/* Lucienne */
+	{}
+};
+
+bool force_storage_d3(void)
+{
+	return x86_match_cpu(storage_d3_cpu_ids);
+}
-- 
2.25.1



From hkallweit1 at gmail.com  Wed Jun  9 11:43:57 2021
From: hkallweit1 at gmail.com (Heiner Kallweit)
Date: Wed, 9 Jun 2021 20:43:57 +0200
Subject: linux-next: NVME using PCI legacy interrupts only
Message-ID: <52371274-20bc-a150-a3ed-ba3e1305ad3e@gmail.com>

I found that on linux-next from June 8th my nvme disk is using legacy
interrupts only. Some debugging lead me to irq_find_mapping() in
msi_domain_alloc() returning -EEXIST.

The nvme core first allocates a MSI-X interrupt for setup purposes
and later frees it and allocates the final number of MSI-X interrupts.

The following experimental change brought back the MSI-X interrupts.
This makes me think that somehow freeing a MSI-X interrupt doesn't
free it completely. I didn't see this behavior a few days ago,
therefore I think it's related to the recent changes to
irqdomain/genirq.

Didn't do a bisect yet, maybe you have an idea already.

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index a29b17070..8cc600819 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2381,7 +2381,7 @@ static int nvme_pci_enable(struct nvme_dev *dev)
         * interrupts. Pre-enable a single MSIX or MSI vec for setup. We'll
         * adjust this later.
         */
-       result = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_ALL_TYPES);
+       result = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_LEGACY);
        if (result < 0)
                return result;




From Chaitanya.Kulkarni at wdc.com  Wed Jun  9 12:01:32 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed, 9 Jun 2021 19:01:32 +0000
Subject: [PATCH 5/5] nvmet: add ZBD over ZNS backend support
References: <20210608054635.5661-1-chaitanya.kulkarni@wdc.com>
 <20210608054635.5661-6-chaitanya.kulkarni@wdc.com>
 <DM6PR04MB7081372FC5AB2D825F3372DAE7369@DM6PR04MB7081.namprd04.prod.outlook.com>
Message-ID: <BYAPR04MB4965272CA1EA6437BDD4274E86369@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/9/21 02:44, Damien Le Moal wrote:
> With the nit above fixed, feel free to add
>
> Reviewed-by: Damien Le Moal <damien.lemoal at wdc.com>
>

Thanks for the reviews, I'll send a next version with nit fix and
updated review tags, also CC Jen's and linux-block.




From Chaitanya.Kulkarni at wdc.com  Wed Jun  9 13:48:10 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed, 9 Jun 2021 20:48:10 +0000
Subject: [PATCH V6] nvme-core: use xarray for ctrl ns tracking
References: <20210608024247.35286-1-chaitanya.kulkarni@wdc.com>
 <dbab5916-a6de-5cd0-4bb5-b06d0349df0b@grimberg.me>
Message-ID: <BYAPR04MB49657620DF9C7EA9158DB60686369@BYAPR04MB4965.namprd04.prod.outlook.com>

Sagi,

On 6/8/21 16:25, Sagi Grimberg wrote:
>
> On 6/7/21 7:42 PM, Chaitanya Kulkarni wrote:
>> This patch replaces the ctrl->namespaces tracking from linked list to
>> xarray for better ns-mgmt on the host side. For host side
>> nvme_find_get_ns() falls into the fast path for NVMeOF passthru target.
>> This allows us to have better performance for NVMeOF passthru backend
>> since XArray has shows better performance numbers over having
>> a combination of read-write semapore read + linked list in the
>> nvme_find_get_ns() to find namespace in I/O patch from nsid specified
>> in the nvme_rw_cmd.
>>
[...]
>> @@ -3839,6 +3829,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>>   	list_del_rcu(&ns->siblings);
>>   	mutex_unlock(&ns->ctrl->subsys->lock);
>>   
>> +	xa_erase(&ns->ctrl->namespaces, ns->head->ns_id);
>>   	synchronize_rcu(); /* guarantee not available in head->list */
>>   	nvme_mpath_clear_current_path(ns);
>>   	synchronize_srcu(&ns->head->srcu); /* wait for concurrent submissions */
>> @@ -3852,10 +3843,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>>   			blk_integrity_unregister(ns->disk);
>>   	}
>>   
>> -	down_write(&ns->ctrl->namespaces_rwsem);
>> -	list_del_init(&ns->list);
>> -	up_write(&ns->ctrl->namespaces_rwsem);
>> -
> Why did this change position? this sort of magic makes me scared
> of this patch...

This is to void the another call to the synchronize_rcu(),

I'll add a second call to synchronize_rcu() and keep the original
position.

>
>>   	nvme_mpath_check_last_path(ns);
>>   	nvme_put_ns(ns);
>>   }
[...]
>>   
>>   static int nvme_scan_ns_list(struct nvme_ctrl *ctrl)
>> @@ -4068,10 +4051,6 @@ static void nvme_scan_work(struct work_struct *work)
>>   	if (nvme_scan_ns_list(ctrl) != 0)
>>   		nvme_scan_ns_sequential(ctrl);
>>   	mutex_unlock(&ctrl->scan_lock);
>> -
>> -	down_write(&ctrl->namespaces_rwsem);
>> -	list_sort(NULL, &ctrl->namespaces, ns_cmp);
>> -	up_write(&ctrl->namespaces_rwsem);
> Is xarray sorted by nsid? ana updates rely on ctrl->namespaces to be
> sorted. Where is nvme_update_ana_state change anyway? you tested this
> without CONFIG_NVME_MULTIPATH?
>

Looks like CONFIG_NVME_MULTIPATH disabled in my tree currently.

I've missed the multipath.c in this version that was there in
previous version [1].

Let send out a new version.

[1] https://lists.infradead.org/pipermail/linux-nvme/2020-July/018242.html


From lkp at intel.com  Wed Jun  9 18:12:48 2021
From: lkp at intel.com (kernel test robot)
Date: Thu, 10 Jun 2021 09:12:48 +0800
Subject: [linux-nvme:nvme-5.14] BUILD SUCCESS
 b2350195c4b764679e78f262e67378f148676002
Message-ID: <60c16710.0LWKqd3cqeX8v4nW%lkp@intel.com>

tree/branch: git://git.infradead.org/nvme.git nvme-5.14
branch HEAD: b2350195c4b764679e78f262e67378f148676002  nvme-pci: remove trailing lines for helpers

elapsed time: 725m

configs tested: 97
configs skipped: 2

The following configs have been built successfully.
More configs may be tested in the coming days.

gcc tested configs:
arm                                 defconfig
arm64                            allyesconfig
arm64                               defconfig
arm                              allyesconfig
arm                              allmodconfig
mips                         rt305x_defconfig
sh                         microdev_defconfig
sh                             espt_defconfig
powerpc                 mpc837x_rdb_defconfig
mips                        workpad_defconfig
powerpc                      mgcoge_defconfig
m68k                       m5249evb_defconfig
arm                           h5000_defconfig
microblaze                          defconfig
powerpc                     mpc5200_defconfig
parisc                           alldefconfig
arm                         assabet_defconfig
mips                           ip32_defconfig
sh                   rts7751r2dplus_defconfig
mips                           ci20_defconfig
sh                        edosk7760_defconfig
s390                             allyesconfig
sh                   sh7770_generic_defconfig
arm                     am200epdkit_defconfig
powerpc                      pmac32_defconfig
x86_64                            allnoconfig
ia64                             allmodconfig
ia64                                defconfig
ia64                             allyesconfig
m68k                             allmodconfig
m68k                                defconfig
m68k                             allyesconfig
nios2                               defconfig
arc                              allyesconfig
nds32                             allnoconfig
nds32                               defconfig
nios2                            allyesconfig
csky                                defconfig
alpha                               defconfig
alpha                            allyesconfig
xtensa                           allyesconfig
h8300                            allyesconfig
arc                                 defconfig
sh                               allmodconfig
parisc                              defconfig
s390                             allmodconfig
parisc                           allyesconfig
s390                                defconfig
i386                             allyesconfig
sparc                            allyesconfig
sparc                               defconfig
i386                                defconfig
mips                             allyesconfig
mips                             allmodconfig
powerpc                          allyesconfig
powerpc                          allmodconfig
powerpc                           allnoconfig
i386                 randconfig-a003-20210608
i386                 randconfig-a006-20210608
i386                 randconfig-a004-20210608
i386                 randconfig-a001-20210608
i386                 randconfig-a005-20210608
i386                 randconfig-a002-20210608
x86_64               randconfig-a004-20210608
x86_64               randconfig-a002-20210608
x86_64               randconfig-a003-20210608
x86_64               randconfig-a006-20210608
x86_64               randconfig-a005-20210608
x86_64               randconfig-a001-20210608
i386                 randconfig-a015-20210608
i386                 randconfig-a013-20210608
i386                 randconfig-a016-20210608
i386                 randconfig-a011-20210608
i386                 randconfig-a012-20210608
i386                 randconfig-a014-20210608
riscv                    nommu_k210_defconfig
riscv                            allyesconfig
riscv                    nommu_virt_defconfig
riscv                             allnoconfig
riscv                               defconfig
riscv                          rv32_defconfig
riscv                            allmodconfig
um                           x86_64_defconfig
um                             i386_defconfig
um                            kunit_defconfig
x86_64                           allyesconfig
x86_64                    rhel-8.3-kselftests
x86_64                              defconfig
x86_64                               rhel-8.3
x86_64                      rhel-8.3-kbuiltin
x86_64                                  kexec

clang tested configs:
x86_64               randconfig-a015-20210608
x86_64               randconfig-a012-20210608
x86_64               randconfig-a014-20210608
x86_64               randconfig-a011-20210608
x86_64               randconfig-a016-20210608
x86_64               randconfig-a013-20210608

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org


From chaitanya.kulkarni at wdc.com  Wed Jun  9 18:28:22 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 18:28:22 -0700
Subject: [PATCH 0/4] nvme: add ctrl sgl supported helper
Message-ID: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>

Hi,

For various transports such as fc/tcp/pci it is common to check if
NVMe SGLs are supported or not by the controller.

Instead of open coding that check add nvme_ctrl_sgl_supported()
helper and use it in the respective transports on the host side.

-ck

Chaitanya Kulkarni (4):
  nvme: add a helper to check ctrl sgl support
  nvme-fc: use ctrl sgl check helper
  nvm-pci: use ctrl sgl check helper
  nvme-tcp: use ctrl sgl check helper

 drivers/nvme/host/fc.c   | 2 +-
 drivers/nvme/host/nvme.h | 8 ++++++++
 drivers/nvme/host/pci.c  | 4 ++--
 drivers/nvme/host/tcp.c  | 2 +-
 4 files changed, 12 insertions(+), 4 deletions(-)

-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun  9 18:28:23 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 18:28:23 -0700
Subject: [PATCH 1/4] nvme: add a helper to check ctrl sgl support
In-Reply-To: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
References: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210610012826.53398-2-chaitanya.kulkarni@wdc.com>

For various transports such as fc/tcp/pci it is common to check if
NVMe SGLs are supported or not by the controller.

In this preparation patch we add a helper to avoid the open coding of
such checks in the various transport.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/nvme.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1f397ecba16c..1aab74128d40 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -869,6 +869,14 @@ static inline void nvme_hwmon_exit(struct nvme_ctrl *ctrl)
 }
 #endif
 
+static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
+{
+	if (!(ctrl->sgls & ((1 << 0) | (1 << 1))))
+		return true;
+
+	return false;
+}
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 void nvme_execute_passthru_rq(struct request *rq);
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun  9 18:28:24 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 18:28:24 -0700
Subject: [PATCH 2/4] nvme-fc: use ctrl sgl check helper
In-Reply-To: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
References: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210610012826.53398-3-chaitanya.kulkarni@wdc.com>

Use the helper to check NVMe controller's SGL support.

Reviewed-by: James Smart <jsmart2021 at gmail.com>
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/fc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index f183f9fa03d0..7600863f7752 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -3112,7 +3112,7 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	}
 
 	/* FC-NVME supports normal SGL Data Block Descriptors */
-	if (!(ctrl->ctrl.sgls & ((1 << 0) | (1 << 1)))) {
+	if (!nvme_ctrl_sgl_supported(&ctrl->ctrl)) {
 		dev_err(ctrl->ctrl.device,
 			"Mandatory sgls are not supported!\n");
 		ret = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun  9 18:28:25 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 18:28:25 -0700
Subject: [PATCH 3/4] nvm-pci: use ctrl sgl check helper
In-Reply-To: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
References: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210610012826.53398-4-chaitanya.kulkarni@wdc.com>

Use the helper to check NVMe controller's SGL support.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/pci.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index c5356028199d..5df1adc5a1db 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -536,7 +536,7 @@ static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req)
 
 	avg_seg_size = DIV_ROUND_UP(blk_rq_payload_bytes(req), nseg);
 
-	if (!(dev->ctrl.sgls & ((1 << 0) | (1 << 1))))
+	if (!nvme_ctrl_sgl_supported(&dev->ctrl))
 		return false;
 	if (!iod->nvmeq->qid)
 		return false;
@@ -853,7 +853,7 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,
 							     &cmnd->rw, &bv);
 
 			if (iod->nvmeq->qid && sgl_threshold &&
-			    dev->ctrl.sgls & ((1 << 0) | (1 << 1)))
+			    nvme_ctrl_sgl_supported(&dev->ctrl))
 				return nvme_setup_sgl_simple(dev, req,
 							     &cmnd->rw, &bv);
 		}
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun  9 18:28:26 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 18:28:26 -0700
Subject: [PATCH 4/4] nvme-tcp: use ctrl sgl check helper
In-Reply-To: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
References: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210610012826.53398-5-chaitanya.kulkarni@wdc.com>

Use the helper to check NVMe controller's SGL support.

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/tcp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 6a65b0516180..c7bd37103cf4 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1993,7 +1993,7 @@ static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new)
 		goto destroy_admin;
 	}
 
-	if (!(ctrl->sgls & ((1 << 0) | (1 << 1)))) {
+	if (!nvme_ctrl_sgl_supported(ctrl)) {
 		ret = -EOPNOTSUPP;
 		dev_err(ctrl->device, "Mandatory sgls are not supported!\n");
 		goto destroy_admin;
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun  9 18:32:47 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 18:32:47 -0700
Subject: [PATCH V15 0/5] nvmet: add ZBD backened support
Message-ID: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>

Hi,

NVMeOF Host is capable of handling the NVMe Protocol based Zoned Block
Devices (ZBD) in the Zoned Namespaces (ZNS) mode with the passthru
backend. There is no support for a generic block device backend to
handle the ZBDs which are not NVMe protocol compliant.

This adds support to export the ZBDs (which are not NVMe drives) to host
from the target via NVMeOF using the host side ZNS interface.

Note: This patch-series is based on nvme-5.14.

-ck

Chaitanya Kulkarni (5):
  block: export blk_next_bio()
  nvmet: add req cns error complete helper
  nvmet: add nvmet_req_bio put helper for backends
  nvmet: add Command Set Identifier support
  nvmet: add ZBD over ZNS backend support

 block/blk-lib.c                   |   1 +
 drivers/nvme/target/Makefile      |   1 +
 drivers/nvme/target/admin-cmd.c   | 121 +++++-
 drivers/nvme/target/core.c        |  43 ++-
 drivers/nvme/target/io-cmd-bdev.c |  29 +-
 drivers/nvme/target/nvmet.h       |  35 ++
 drivers/nvme/target/passthru.c    |   3 +-
 drivers/nvme/target/zns.c         | 622 ++++++++++++++++++++++++++++++
 include/linux/bio.h               |   2 +
 include/linux/nvme.h              |   8 +
 10 files changed, 828 insertions(+), 37 deletions(-)
 create mode 100644 drivers/nvme/target/zns.c

Changes since V14:-

1. Remove the unsingned long cast in the call to set_bit().
2. Update reviewed-by tag.

Changes since V13:-

1. Move the code cleanup patches upfront and export blk_next_bio().
2. Remove NVM in the subject line for the CSS patch.
3. Remove the switch from nvmet_execute_identify_desclist_csi()
   and use ns->csi directly.
4. Add csi switch in the nvmet_parse_io_cmd().
5. Rename nvmet_cc_css_check() -> nvmet_css_supported().
6. Remove switch from nvmet_set_csi_zns_effects().
7. Rename nvmet_set_csi_nvm_effects() ->
   nvmet_get_cmd_effects_nvm().
7. Add separate parse case for the different csi.
8. Remove the bdev_is_zoned().
9. Remove the csi check from the admin cmd handlers and move
   to the caller.
10. Add Zone State change emulation and add state machine check
    to handle zone mgmt send command.
12. Abort the zone append command with error when transfer len = 0.
13. Don't allocate the buffer for zone mgmt recv report zones instead
    use the command data buffer.
14. Overall small cleanups to reflect the new code changes.
15. Offload zone-mgmt commands to workqueue.

Changes since V12:-

1. Remove the comment :-
  /* bdev_is_zoned() is stubbed out of CONFIG_BLK_DEV_ZONED */
2. Update copyright header.
3. Use folliwng expression in nvmet_zasl(). 
   return ilog2(zone_append_sects >> (NVMET_MPSMIN_SHIFT - 9));
4. Add comment in nvmet_bdev_zns_enable().
5. Add zone filtering and partial bit handling.
6. Use if else pattern in nvmet_bdev_execute_zone_mgmt_send().
7. Make zone append async. 
8. Add a comment for nvmet_check_transfer_len().
9. Add a helper nvmet_req_cns_error_compplete() to remove the duplicate
   code.

Changes from V11:-

1. Use bdev_logical_block_size() in nvmet_bdev_zns_enable().
2. Return error if nr_zones calculation ends up having value 0.
4. Drop the comment patch from this series :-
   nvme: add comments to nvme_zns_alloc_report_buffer

Changes from V10:-

1.  Move CONFIG_BLK_DEV_ZONED check into the caller of
    nvmet_set_csi_zns_effects().
2.  Move ZNS related csi code from csi patch to its own ZNS backend
    patch.
3.  For ZNS command effects logs set default command effects with 
    nvmet_set_csi_nvm_effects() along with nvmet_set_csi_zns_effects().
4.  Use goto for failure case in the nvmet_set_csi_zns_effects().
5.  Return directly from swicth in nvmet_execute_identify_desclist_csi().
6.  Merge Patch 2nd/3rd into one patch and move ZNS related code
    into its own :-
     [PATCH V10 2/8] nvmet: add NVM Command Set Identifier support
     [PATCH V10 3/8] nvmet: add command set supported ctrl cap
    Merged into new patch minux ZNS calls :-
     [PATCH V11 1/4] nvmet: add NVM Command Set Identifier support
7.  Move req->cmd->identify.csi == NVME_CSI_ZNS checks in to respective
    caller in nvmet_execute_identify().
8.  Update error log page in nvmet_bdev_zns_checks().
9.  Remove the terney expression nvmet_zns_update_zasl().
10. Drop following patches:-
     [PATCH V10 1/8] nvmet: trim args for nvmet_copy_ns_identifier()
     [PATCH V10 6/8] nvme-core: check ctrl css before setting up zns
     [PATCH V10 7/8] nvme-core: add a helper to print css related error

Changes from V9:-

1.  Use bio_add_zone_append_page() for REQ_OP_ZONE_APPEND.
2.  Add a separate prep patch to reduce the arguments for
    nvmet_copy_ns_identifier().
3.  Add a separate patch for nvmet CSI support.
4.  Add a separate patch for nvmet CSS support.
5.  Move nvmet_cc_css_check() to multi-css supported patch.
6.  Return error in csi cmd effects helper when !CONFIG_BLK_DEV_ZONED.
7.  Return error in id desc list helper when !CONFIG_BLK_DEV_ZONED.
8.  Remove goto and return from nvmet_bdev_zns_checks().
9.  Move nr_zones calculations near call to blkdev_report_zones() in
    nvmet_bdev_execute_zone_mgmt_recv().
10. Split command effects logs into respective CSI helpers.
11. Don't use the local variables to pass NVME_CSI_XXX values, instead
    use req->ns->csi, also move this from ZBD support patch to nvmet
    CSI support.
12. Fix the bug that is chekcing cns value instead of csi in identify.
13. bdev_is_zoned() is stubbed out the CONFIG_BLK_DEV_ZONED, so remove
    the check for CONFIG_BLK_DEV_ZONED before calling bdev_is_zoned().
14  Drop following patches :-
    [PATCH V9 1/9] block: export bio_add_hw_pages().
    [PATCH V9 5/9] nvmet: add bio get helper.
    [PATCH V9 6/9] nvmet: add bio init helper.
    [PATCH V9 8/9] nvmet: add common I/O length check helper.
    [PATCH V9 9/9] nvmet: call nvmet_bio_done() for zone append.
15. Add a patch to check for ctrl bits to make sure ctrl supports the
    multi css on the host side when setting up Zoned Namespace.
17. Add a documentation patch for the host side when calculating the
    buffer size allocation size for the report zones.
16. Rebase and retest 5.12-rc1.

Changes from V8:-

1. Rebase and retest on latest nvme-5.11.
2. Export ctrl->cap csi support only if CONFIG_BLK_DEV_ZONE is set.
3. Add a fix to admin ns-desc list handler for handling default csi.

Changes from V7:-

1. Just like what block layer provides an API for bio_init(), provide
   nvmet_bio_init() such that we move bio initialization code for
   nvme-read-write commands from bdev and zns backend into the centralize
   helper. 
2. With bdev/zns/file now we have three backends that are checking for
   req->sg_cnt and calling nvmet_check_transfer_len() before we process
   nvme-read-write commands. Move this duplicate code from three
   backeneds into the helper.
3. Export and use nvmet_bio_done() callback in
   nvmet_execute_zone_append() instead of the open coding the function.
   This also avoids code duplication for bio & request completion with
   error log page update.
4. Add zonefs tests log for dm linear device created on the top of SMR HDD
   exported with NVMeOF ZNS backend with the help of nvme-loop.

Changes from V6:-

1. Instead of calling report zones to find conventional zones in the 
   loop use the loop inside LLD blkdev_report_zones()->LLD_report_zones,
   that also simplifies the report zone callback.
2. Fix the bug in nvmet_bdev_has_conv_zones().
3. Remove conditional operators in the nvmet_bdev_execute_zone_append().

Changes from V5:-

1.  Use bio->bi_iter.bi_sector for result of the REQ_OP_ZONE_APPEND
    command.
2.  Add endianness to the helper nvmet_sect_to_lba().
3.  Make bufsize u32 in zone mgmt recv command handler.
4.  Add __GFP_ZERO for report zone data buffer to return clean buffer.

Changes from V4:-

1.  Don't use bio_iov_iter_get_pages() instead add a patch to export
    bio_add_hw_page() and call it directly for zone append.
2.  Add inline vector optimization for append bio.
3.  Update the commit logs for the patches.
4.  Remove ZNS related identify data structures, use individual members.
5.  Add a comment for macro NVMET_MPSMIN_SHIFT.
6.  Remove nvmet_bdev() helper.
7.  Move the command set identifier code into common code.
8.  Use IS_ENABLED() and move helpers fomr zns.c into common code.
9.  Add a patch to support Command Set identifiers.
10. Open code nvmet_bdev_validate_zns_zones().
11. Remove the per namespace min zasl calculation and don't allow
    namespaces with zasl value > the first ns zasl value.
12. Move the stubs into the header file.
13. Add lba to/from sector conversion helpers and update the
    io-cmd-bdev.c to avoid the code duplication.
14. Add everything into one patch for zns command handlers and 
    respective calls from the target code.
15. Remove the trim ns-desclist admin callback patch from this series.
16. Add bio get and put helpers patches to reduce the duplicate code in
    generic bdev, passthru, and generic zns backend.

Changes from V3:-

1.  Get rid of the bio_max_zasl check.
2.  Remove extra lines.
3.  Remove the block layer api export patch.
4.  Remove the bvec check in the bio_iov_iter_get_pages() for
    REQ_OP_ZONE_APPEND so that we can reuse the code.

Changes from V2:-

1.  Move conventional zone bitmap check into 
    nvmet_bdev_validate_zns_zones(). 
2.  Don't use report zones call to check the runt zone.
3.  Trim nvmet_zasl() helper.
4.  Fix typo in the nvmet_zns_update_zasl().
5.  Remove the comment and fix the mdts calculation in
    nvmet_execute_identify_cns_cs_ctrl().
6.  Use u64 for bufsize in nvmet_bdev_execute_zone_mgmt_recv().
7.  Remove nvmet_zones_to_desc_size() and fix the nr_zones
    calculation.
8.  Remove the op variable in nvmet_bdev_execute_zone_append().
9.  Fix the nr_zones calculation nvmet_bdev_execute_zone_mgmt_recv().
10. Update cover letter subject.

Changes from V1:-

1.  Remove the nvmet-$(CONFIG_BLK_DEV_ZONED) += zns.o.
2.  Mark helpers inline.
3.  Fix typos in the comments and update the comments.
4.  Get rid of the curly brackets.
5.  Don't allow drives with last smaller zones.
6.  Calculate the zasl as a function of ax_zone_append_sectors,
    bio_max_pages so we don't have to split the bio.
7.  Add global subsys->zasl and update the zasl when new namespace
    is enabled.
8.  Rmove the loop in the nvmet_bdev_execute_zone_mgmt_recv() and
    move functionality in to the report zone callback.
9.  Add goto for default case in nvmet_bdev_execute_zone_mgmt_send().
10. Allocate the zones buffer with zones size instead of bdev nr_zones.

-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun  9 18:32:48 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 18:32:48 -0700
Subject: [PATCH V15 1/5] block: export blk_next_bio()
In-Reply-To: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
References: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210610013252.53874-2-chaitanya.kulkarni@wdc.com>

The block layer provides emulation of zone management operations
targeting all zones of a zoned block device only for the zone reset
operation (REQ_OP_ZONE_RESET). In order to correctly implement
exporting of zoned block devices with NVMeOF, emulating zone management
operations targeting all zones of a device is also necessary for the
open, close and finish zone operations (REQ_OP_ZONE_OPEN,
REQ_OP_ZONE_CLOSE and REQ_OP_ZONE_FINISH).

Instead of duplicating the code, export the existing helper from block
layer so we can use a bio chaining pattern that is present in the block
layer for REQ_OP_ZONE RESET all emulation in the NVMeOF zoned block
device backend.

Reviewed-by: Damien Le Moal <damien.lemoal at wdc.com>
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 block/blk-lib.c     | 1 +
 include/linux/bio.h | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/block/blk-lib.c b/block/blk-lib.c
index 7b256131b20b..9f09beadcbe3 100644
--- a/block/blk-lib.c
+++ b/block/blk-lib.c
@@ -21,6 +21,7 @@ struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp)
 
 	return new;
 }
+EXPORT_SYMBOL_GPL(blk_next_bio);
 
 int __blkdev_issue_discard(struct block_device *bdev, sector_t sector,
 		sector_t nr_sects, gfp_t gfp_mask, int flags,
diff --git a/include/linux/bio.h b/include/linux/bio.h
index a0b4cfdf62a4..b2491ead22a0 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -822,4 +822,6 @@ static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb)
 		bio->bi_opf |= REQ_NOWAIT;
 }
 
+struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp);
+
 #endif /* __LINUX_BIO_H */
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun  9 18:32:49 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 18:32:49 -0700
Subject: [PATCH V15 2/5] nvmet: add req cns error complete helper
In-Reply-To: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
References: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210610013252.53874-3-chaitanya.kulkarni@wdc.com>

We report error and complete the request when identify cns value is not
handled in nvmet_execute_identify(). This error reporting is also needed
for Zone Block Device backend for NVMeOF target.

Add a helper nvmet_req_cns_error_compplete() to report an error and
complete the request when idenitfy command cns not handled value.

Reviewed-by: Damien Le Moal <damien.lemoal at wdc.com>
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 5 +----
 drivers/nvme/target/nvmet.h     | 8 ++++++++
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index cd60a8184d04..3de6a6c99b01 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -637,10 +637,7 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 		return nvmet_execute_identify_desclist(req);
 	}
 
-	pr_debug("unhandled identify cns %d on qid %d\n",
-	       req->cmd->identify.cns, req->sq->qid);
-	req->error_loc = offsetof(struct nvme_identify, cns);
-	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
+	nvmet_req_cns_error_complete(req);
 }
 
 /*
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index bd0a0b91d843..f53ce96df498 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -624,4 +624,12 @@ static inline bool nvmet_use_inline_bvec(struct nvmet_req *req)
 	       req->sg_cnt <= NVMET_MAX_INLINE_BIOVEC;
 }
 
+static inline void nvmet_req_cns_error_complete(struct nvmet_req *req)
+{
+	pr_debug("unhandled identify cns %d on qid %d\n",
+	       req->cmd->identify.cns, req->sq->qid);
+	req->error_loc = offsetof(struct nvme_identify, cns);
+	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
+}
+
 #endif /* _NVMET_H */
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun  9 18:32:50 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 18:32:50 -0700
Subject: [PATCH V15 3/5] nvmet: add nvmet_req_bio put helper for backends
In-Reply-To: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
References: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210610013252.53874-4-chaitanya.kulkarni@wdc.com>

In current code there exists two backends which are using inline bio
optimization, that adds a duplicate code for freeing the bio.

For Zoned Block Device backend we also use the same optimzation and it
will lead to having duplicate code in the three backends: generic
bdev, passsthru, and generic zns.

Add a helper function to avoid duplicate code and update the respective
backends.

Reviewed-by: Damien Le Moal <damien.lemoal at wdc.com>
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/io-cmd-bdev.c | 3 +--
 drivers/nvme/target/nvmet.h       | 6 ++++++
 drivers/nvme/target/passthru.c    | 3 +--
 3 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index f673679d258a..40534d9a86d1 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -164,8 +164,7 @@ static void nvmet_bio_done(struct bio *bio)
 	struct nvmet_req *req = bio->bi_private;
 
 	nvmet_req_complete(req, blk_to_nvme_status(req, bio->bi_status));
-	if (bio != &req->b.inline_bio)
-		bio_put(bio);
+	nvmet_req_bio_put(req, bio);
 }
 
 #ifdef CONFIG_BLK_DEV_INTEGRITY
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index f53ce96df498..d4bcfeb570e5 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -632,4 +632,10 @@ static inline void nvmet_req_cns_error_complete(struct nvmet_req *req)
 	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
 }
 
+static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio)
+{
+	if (bio != &req->b.inline_bio)
+		bio_put(bio);
+}
+
 #endif /* _NVMET_H */
diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
index 39b1473f7204..fced52de33ce 100644
--- a/drivers/nvme/target/passthru.c
+++ b/drivers/nvme/target/passthru.c
@@ -206,8 +206,7 @@ static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq)
 	for_each_sg(req->sg, sg, req->sg_cnt, i) {
 		if (bio_add_pc_page(rq->q, bio, sg_page(sg), sg->length,
 				    sg->offset) < sg->length) {
-			if (bio != &req->p.inline_bio)
-				bio_put(bio);
+			nvmet_req_bio_put(req, bio);
 			return -EINVAL;
 		}
 	}
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun  9 18:32:51 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 18:32:51 -0700
Subject: [PATCH V15 4/5] nvmet: add Command Set Identifier support
In-Reply-To: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
References: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210610013252.53874-5-chaitanya.kulkarni@wdc.com>

NVMe TP 4056 allows controllers to support different command sets.
NVMeoF target currently only supports namespaces that contain
traditional logical blocks that may be randomly read and written. In
some applications there is a value in exposing namespaces that contain
logical blocks that have special access rules (e.g. sequentially write
required namespace such as Zoned Namespace (ZNS)).

In order to support the Zoned Block Devices (ZBD) backend, controllers
need to have support for ZNS Command Set Identifier (CSI).

In this preparation patch, we adjust the code such that it can now
support the default command set identifier. We update the namespace data
structure to store the CSI value which defaults to NVME_CSI_NVM
that represents traditional logical blocks namespace type.

The CSI support is required to implement the ZBD backend for NVMeOF
with host side NVMe ZNS interface, since ZNS commands belong to
the different command set than the default one.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 75 +++++++++++++++++++++++++++------
 drivers/nvme/target/core.c      | 28 +++++++++---
 drivers/nvme/target/nvmet.h     |  1 +
 include/linux/nvme.h            |  1 +
 4 files changed, 87 insertions(+), 18 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 3de6a6c99b01..93aaa7479e71 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -162,15 +162,8 @@ static void nvmet_execute_get_log_page_smart(struct nvmet_req *req)
 	nvmet_req_complete(req, status);
 }
 
-static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
+static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log)
 {
-	u16 status = NVME_SC_INTERNAL;
-	struct nvme_effects_log *log;
-
-	log = kzalloc(sizeof(*log), GFP_KERNEL);
-	if (!log)
-		goto out;
-
 	log->acs[nvme_admin_get_log_page]	= cpu_to_le32(1 << 0);
 	log->acs[nvme_admin_identify]		= cpu_to_le32(1 << 0);
 	log->acs[nvme_admin_abort_cmd]		= cpu_to_le32(1 << 0);
@@ -184,9 +177,30 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
 	log->iocs[nvme_cmd_flush]		= cpu_to_le32(1 << 0);
 	log->iocs[nvme_cmd_dsm]			= cpu_to_le32(1 << 0);
 	log->iocs[nvme_cmd_write_zeroes]	= cpu_to_le32(1 << 0);
+}
 
-	status = nvmet_copy_to_sgl(req, 0, log, sizeof(*log));
+static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
+{
+	struct nvme_effects_log *log;
+	u16 status = NVME_SC_SUCCESS;
+
+	log = kzalloc(sizeof(*log), GFP_KERNEL);
+	if (!log) {
+		status = NVME_SC_INTERNAL;
+		goto out;
+	}
+
+	switch (req->cmd->get_log_page.csi) {
+	case NVME_CSI_NVM:
+		nvmet_get_cmd_effects_nvm(log);
+		break;
+	default:
+		status = NVME_SC_INVALID_LOG_PAGE;
+		goto free;
+	}
 
+	status = nvmet_copy_to_sgl(req, 0, log, sizeof(*log));
+free:
 	kfree(log);
 out:
 	nvmet_req_complete(req, status);
@@ -613,6 +627,12 @@ static void nvmet_execute_identify_desclist(struct nvmet_req *req)
 			goto out;
 	}
 
+	status = nvmet_copy_ns_identifier(req, NVME_NIDT_CSI,
+					  NVME_NIDT_CSI_LEN,
+					  &req->ns->csi, &off);
+	if (status)
+		goto out;
+
 	if (sg_zero_buffer(req->sg, req->sg_cnt, NVME_IDENTIFY_DATA_SIZE - off,
 			off) != NVME_IDENTIFY_DATA_SIZE - off)
 		status = NVME_SC_INTERNAL | NVME_SC_DNR;
@@ -621,6 +641,17 @@ static void nvmet_execute_identify_desclist(struct nvmet_req *req)
 	nvmet_req_complete(req, status);
 }
 
+static bool nvmet_handle_identify_desclist(struct nvmet_req *req)
+{
+	switch (req->cmd->identify.csi) {
+	case NVME_CSI_NVM:
+		nvmet_execute_identify_desclist(req);
+		return true;
+	default:
+		return false;
+	}
+}
+
 static void nvmet_execute_identify(struct nvmet_req *req)
 {
 	if (!nvmet_check_transfer_len(req, NVME_IDENTIFY_DATA_SIZE))
@@ -628,13 +659,31 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 
 	switch (req->cmd->identify.cns) {
 	case NVME_ID_CNS_NS:
-		return nvmet_execute_identify_ns(req);
+		switch (req->cmd->identify.csi) {
+		case NVME_CSI_NVM:
+			return nvmet_execute_identify_ns(req);
+		default:
+			break;
+		}
+		break;
 	case NVME_ID_CNS_CTRL:
-		return nvmet_execute_identify_ctrl(req);
+		switch (req->cmd->identify.csi) {
+		case NVME_CSI_NVM:
+			return nvmet_execute_identify_ctrl(req);
+		}
+		break;
 	case NVME_ID_CNS_NS_ACTIVE_LIST:
-		return nvmet_execute_identify_nslist(req);
+		switch (req->cmd->identify.csi) {
+		case NVME_CSI_NVM:
+			return nvmet_execute_identify_nslist(req);
+		default:
+			break;
+		}
+		break;
 	case NVME_ID_CNS_NS_DESC_LIST:
-		return nvmet_execute_identify_desclist(req);
+		if (nvmet_handle_identify_desclist(req) == true)
+			return;
+		break;
 	}
 
 	nvmet_req_cns_error_complete(req);
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 146909486b8f..ac53a1307ce4 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -692,6 +692,7 @@ struct nvmet_ns *nvmet_ns_alloc(struct nvmet_subsys *subsys, u32 nsid)
 
 	uuid_gen(&ns->uuid);
 	ns->buffered_io = false;
+	ns->csi = NVME_CSI_NVM;
 
 	return ns;
 }
@@ -887,10 +888,14 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 		return ret;
 	}
 
-	if (req->ns->file)
-		return nvmet_file_parse_io_cmd(req);
-
-	return nvmet_bdev_parse_io_cmd(req);
+	switch (req->ns->csi) {
+	case NVME_CSI_NVM:
+		if (req->ns->file)
+			return nvmet_file_parse_io_cmd(req);
+		return nvmet_bdev_parse_io_cmd(req);
+	default:
+		return NVME_SC_INVALID_IO_CMD_SET;
+	}
 }
 
 bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
@@ -1112,6 +1117,17 @@ static inline u8 nvmet_cc_iocqes(u32 cc)
 	return (cc >> NVME_CC_IOCQES_SHIFT) & 0xf;
 }
 
+static inline bool nvmet_css_supported(u8 cc_css)
+{
+	switch (cc_css <<= NVME_CC_CSS_SHIFT) {
+	case NVME_CC_CSS_NVM:
+	case NVME_CC_CSS_CSI:
+		return true;
+	default:
+		return false;
+	}
+}
+
 static void nvmet_start_ctrl(struct nvmet_ctrl *ctrl)
 {
 	lockdep_assert_held(&ctrl->lock);
@@ -1131,7 +1147,7 @@ static void nvmet_start_ctrl(struct nvmet_ctrl *ctrl)
 
 	if (nvmet_cc_mps(ctrl->cc) != 0 ||
 	    nvmet_cc_ams(ctrl->cc) != 0 ||
-	    nvmet_cc_css(ctrl->cc) != 0) {
+	    !nvmet_css_supported(nvmet_cc_css(ctrl->cc))) {
 		ctrl->csts = NVME_CSTS_CFS;
 		return;
 	}
@@ -1182,6 +1198,8 @@ static void nvmet_init_cap(struct nvmet_ctrl *ctrl)
 {
 	/* command sets supported: NVMe command set: */
 	ctrl->cap = (1ULL << 37);
+	/* Controller supports one or more I/O Command Sets */
+	ctrl->cap |= (1ULL << 43);
 	/* CC.EN timeout in 500msec units: */
 	ctrl->cap |= (15ULL << 24);
 	/* maximum queue entries supported: */
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index d4bcfeb570e5..4ca558b94955 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -83,6 +83,7 @@ struct nvmet_ns {
 	struct pci_dev		*p2p_dev;
 	int			pi_type;
 	int			metadata_size;
+	u8			csi;
 };
 
 static inline struct nvmet_ns *to_nvmet_ns(struct config_item *item)
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index edcbd60b88b9..c7ba83144d52 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -1504,6 +1504,7 @@ enum {
 	NVME_SC_NS_WRITE_PROTECTED	= 0x20,
 	NVME_SC_CMD_INTERRUPTED		= 0x21,
 	NVME_SC_TRANSIENT_TR_ERR	= 0x22,
+	NVME_SC_INVALID_IO_CMD_SET	= 0x2C,
 
 	NVME_SC_LBA_RANGE		= 0x80,
 	NVME_SC_CAP_EXCEEDED		= 0x81,
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun  9 18:32:52 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 18:32:52 -0700
Subject: [PATCH V15 5/5] nvmet: add ZBD over ZNS backend support
In-Reply-To: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
References: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210610013252.53874-6-chaitanya.kulkarni@wdc.com>

NVMe TP 4053 ? Zoned Namespaces (ZNS) allows host software to
communicate with a non-volatile memory subsystem using zones for NVMe
protocol-based controllers. NVMeOF already support the ZNS NVMe
Protocol compliant devices on the target in the passthru mode. There
are generic zoned block devices like  Shingled Magnetic Recording (SMR)
HDDs that are not based on the NVMe protocol.

This patch adds ZNS backend support for non-ZNS zoned block devices as
NVMeOF targets.

This support includes implementing the new command set NVME_CSI_ZNS,
adding different command handlers for ZNS command set such as NVMe
Identify Controller, NVMe Identify Namespace, NVMe Zone Append,
NVMe Zone Management Send and NVMe Zone Management Receive.

With the new command set identifier, we also update the target command
effects logs to reflect the ZNS compliant commands.

Reviewed-by: Damien Le Moal <damien.lemoal at wdc.com>
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/Makefile      |   1 +
 drivers/nvme/target/admin-cmd.c   |  41 ++
 drivers/nvme/target/core.c        |  15 +-
 drivers/nvme/target/io-cmd-bdev.c |  26 +-
 drivers/nvme/target/nvmet.h       |  20 +
 drivers/nvme/target/zns.c         | 622 ++++++++++++++++++++++++++++++
 include/linux/nvme.h              |   7 +
 7 files changed, 721 insertions(+), 11 deletions(-)
 create mode 100644 drivers/nvme/target/zns.c

diff --git a/drivers/nvme/target/Makefile b/drivers/nvme/target/Makefile
index ebf91fc4c72e..9837e580fa7e 100644
--- a/drivers/nvme/target/Makefile
+++ b/drivers/nvme/target/Makefile
@@ -12,6 +12,7 @@ obj-$(CONFIG_NVME_TARGET_TCP)		+= nvmet-tcp.o
 nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
 			discovery.o io-cmd-file.o io-cmd-bdev.o
 nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
+nvmet-$(CONFIG_BLK_DEV_ZONED)		+= zns.o
 nvme-loop-y	+= loop.o
 nvmet-rdma-y	+= rdma.o
 nvmet-fc-y	+= fc.o
diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 93aaa7479e71..363e357d2f20 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -179,6 +179,13 @@ static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log)
 	log->iocs[nvme_cmd_write_zeroes]	= cpu_to_le32(1 << 0);
 }
 
+static void nvmet_get_cmd_effects_zns(struct nvme_effects_log *log)
+{
+	log->iocs[nvme_cmd_zone_append]		= cpu_to_le32(1 << 0);
+	log->iocs[nvme_cmd_zone_mgmt_send]	= cpu_to_le32(1 << 0);
+	log->iocs[nvme_cmd_zone_mgmt_recv]	= cpu_to_le32(1 << 0);
+}
+
 static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
 {
 	struct nvme_effects_log *log;
@@ -194,6 +201,14 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
 	case NVME_CSI_NVM:
 		nvmet_get_cmd_effects_nvm(log);
 		break;
+	case NVME_CSI_ZNS:
+		if (!IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
+			status = NVME_SC_INVALID_IO_CMD_SET;
+			goto free;
+		}
+		nvmet_get_cmd_effects_nvm(log);
+		nvmet_get_cmd_effects_zns(log);
+		break;
 	default:
 		status = NVME_SC_INVALID_LOG_PAGE;
 		goto free;
@@ -647,6 +662,12 @@ static bool nvmet_handle_identify_desclist(struct nvmet_req *req)
 	case NVME_CSI_NVM:
 		nvmet_execute_identify_desclist(req);
 		return true;
+	case NVME_CSI_ZNS:
+		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
+			nvmet_execute_identify_desclist(req);
+			return true;
+		}
+		return false;
 	default:
 		return false;
 	}
@@ -666,12 +687,32 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 			break;
 		}
 		break;
+	case NVME_ID_CNS_CS_NS:
+		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
+			switch (req->cmd->identify.csi) {
+			case NVME_CSI_ZNS:
+				return nvmet_execute_identify_cns_cs_ns(req);
+			default:
+				break;
+			}
+		}
+		break;
 	case NVME_ID_CNS_CTRL:
 		switch (req->cmd->identify.csi) {
 		case NVME_CSI_NVM:
 			return nvmet_execute_identify_ctrl(req);
 		}
 		break;
+	case NVME_ID_CNS_CS_CTRL:
+		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
+			switch (req->cmd->identify.csi) {
+			case NVME_CSI_ZNS:
+				return nvmet_execute_identify_cns_cs_ctrl(req);
+			default:
+				break;
+			}
+		}
+		break;
 	case NVME_ID_CNS_NS_ACTIVE_LIST:
 		switch (req->cmd->identify.csi) {
 		case NVME_CSI_NVM:
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index ac53a1307ce4..d061e9946568 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -16,6 +16,7 @@
 #include "nvmet.h"
 
 struct workqueue_struct *buffered_io_wq;
+struct workqueue_struct *zbd_wq;
 static const struct nvmet_fabrics_ops *nvmet_transports[NVMF_TRTYPE_MAX];
 static DEFINE_IDA(cntlid_ida);
 
@@ -893,6 +894,10 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 		if (req->ns->file)
 			return nvmet_file_parse_io_cmd(req);
 		return nvmet_bdev_parse_io_cmd(req);
+	case NVME_CSI_ZNS:
+		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED))
+			return nvmet_bdev_zns_parse_io_cmd(req);
+		return NVME_SC_INVALID_IO_CMD_SET;
 	default:
 		return NVME_SC_INVALID_IO_CMD_SET;
 	}
@@ -1602,11 +1607,15 @@ static int __init nvmet_init(void)
 
 	nvmet_ana_group_enabled[NVMET_DEFAULT_ANA_GRPID] = 1;
 
+	zbd_wq = alloc_workqueue("nvmet-zbd-wq", WQ_MEM_RECLAIM, 0);
+	if (!zbd_wq)
+		return -ENOMEM;
+
 	buffered_io_wq = alloc_workqueue("nvmet-buffered-io-wq",
 			WQ_MEM_RECLAIM, 0);
 	if (!buffered_io_wq) {
 		error = -ENOMEM;
-		goto out;
+		goto out_free_zbd_work_queue;
 	}
 
 	error = nvmet_init_discovery();
@@ -1622,7 +1631,8 @@ static int __init nvmet_init(void)
 	nvmet_exit_discovery();
 out_free_work_queue:
 	destroy_workqueue(buffered_io_wq);
-out:
+out_free_zbd_work_queue:
+	destroy_workqueue(zbd_wq);
 	return error;
 }
 
@@ -1632,6 +1642,7 @@ static void __exit nvmet_exit(void)
 	nvmet_exit_discovery();
 	ida_destroy(&cntlid_ida);
 	destroy_workqueue(buffered_io_wq);
+	destroy_workqueue(zbd_wq);
 
 	BUILD_BUG_ON(sizeof(struct nvmf_disc_rsp_page_entry) != 1024);
 	BUILD_BUG_ON(sizeof(struct nvmf_disc_rsp_page_hdr) != 1024);
diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index 40534d9a86d1..57f619aa7cba 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -47,6 +47,14 @@ void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id)
 	id->nows = to0based(ql->io_opt / ql->logical_block_size);
 }
 
+void nvmet_bdev_ns_disable(struct nvmet_ns *ns)
+{
+	if (ns->bdev) {
+		blkdev_put(ns->bdev, FMODE_WRITE | FMODE_READ);
+		ns->bdev = NULL;
+	}
+}
+
 static void nvmet_bdev_ns_enable_integrity(struct nvmet_ns *ns)
 {
 	struct blk_integrity *bi = bdev_get_integrity(ns->bdev);
@@ -86,15 +94,15 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
 	if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY_T10))
 		nvmet_bdev_ns_enable_integrity(ns);
 
-	return 0;
-}
-
-void nvmet_bdev_ns_disable(struct nvmet_ns *ns)
-{
-	if (ns->bdev) {
-		blkdev_put(ns->bdev, FMODE_WRITE | FMODE_READ);
-		ns->bdev = NULL;
+	if (bdev_is_zoned(ns->bdev)) {
+		if (!nvmet_bdev_zns_enable(ns)) {
+			nvmet_bdev_ns_disable(ns);
+			return -EINVAL;
+		}
+		ns->csi = NVME_CSI_ZNS;
 	}
+
+	return 0;
 }
 
 void nvmet_bdev_ns_revalidate(struct nvmet_ns *ns)
@@ -102,7 +110,7 @@ void nvmet_bdev_ns_revalidate(struct nvmet_ns *ns)
 	ns->size = i_size_read(ns->bdev->bd_inode);
 }
 
-static u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts)
+u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts)
 {
 	u16 status = NVME_SC_SUCCESS;
 
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 4ca558b94955..f7fbf9690ce8 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -250,6 +250,10 @@ struct nvmet_subsys {
 	unsigned int		admin_timeout;
 	unsigned int		io_timeout;
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
+#ifdef CONFIG_BLK_DEV_ZONED
+	u8			zasl;
+#endif /* CONFIG_BLK_DEV_ZONED */
 };
 
 static inline struct nvmet_subsys *to_subsys(struct config_item *item)
@@ -335,6 +339,12 @@ struct nvmet_req {
 			struct work_struct      work;
 			bool			use_workqueue;
 		} p;
+#ifdef CONFIG_BLK_DEV_ZONED
+		struct {
+			struct bio		inline_bio;
+			struct work_struct	zmgmt_work;
+		} z;
+#endif /* CONFIG_BLK_DEV_ZONED */
 	};
 	int			sg_cnt;
 	int			metadata_sg_cnt;
@@ -354,6 +364,7 @@ struct nvmet_req {
 };
 
 extern struct workqueue_struct *buffered_io_wq;
+extern struct workqueue_struct *zbd_wq;
 
 static inline void nvmet_set_result(struct nvmet_req *req, u32 result)
 {
@@ -403,6 +414,7 @@ u16 nvmet_parse_connect_cmd(struct nvmet_req *req);
 void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id);
 u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req);
 u16 nvmet_file_parse_io_cmd(struct nvmet_req *req);
+u16 nvmet_bdev_zns_parse_io_cmd(struct nvmet_req *req);
 u16 nvmet_parse_admin_cmd(struct nvmet_req *req);
 u16 nvmet_parse_discovery_cmd(struct nvmet_req *req);
 u16 nvmet_parse_fabrics_cmd(struct nvmet_req *req);
@@ -530,6 +542,14 @@ void nvmet_ns_changed(struct nvmet_subsys *subsys, u32 nsid);
 void nvmet_bdev_ns_revalidate(struct nvmet_ns *ns);
 int nvmet_file_ns_revalidate(struct nvmet_ns *ns);
 void nvmet_ns_revalidate(struct nvmet_ns *ns);
+u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts);
+
+bool nvmet_bdev_zns_enable(struct nvmet_ns *ns);
+void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req);
+void nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req);
+void nvmet_bdev_execute_zone_mgmt_recv(struct nvmet_req *req);
+void nvmet_bdev_execute_zone_mgmt_send(struct nvmet_req *req);
+void nvmet_bdev_execute_zone_append(struct nvmet_req *req);
 
 static inline u32 nvmet_rw_data_len(struct nvmet_req *req)
 {
diff --git a/drivers/nvme/target/zns.c b/drivers/nvme/target/zns.c
new file mode 100644
index 000000000000..b8ff75293e53
--- /dev/null
+++ b/drivers/nvme/target/zns.c
@@ -0,0 +1,622 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe ZNS-ZBD command implementation.
+ * Copyright (C) 2021 Western Digital Corporation or its affiliates.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/nvme.h>
+#include <linux/blkdev.h>
+#include "nvmet.h"
+
+/*
+ * We set the Memory Page Size Minimum (MPSMIN) for target controller to 0
+ * which gets added by 12 in the nvme_enable_ctrl() which results in 2^12 = 4k
+ * as page_shift value. When calculating the ZASL use shift by 12.
+ */
+#define NVMET_MPSMIN_SHIFT	12
+
+static inline u8 nvmet_zasl(unsigned int zone_append_sects)
+{
+	/*
+	 * Zone Append Size Limit (zasl) is expressed as a power of 2 value
+	 * with the minimum memory page size (i.e. 12) as unit.
+	 */
+	return ilog2(zone_append_sects >> (NVMET_MPSMIN_SHIFT - 9));
+}
+
+static int validate_conv_zones_cb(struct blk_zone *z,
+				  unsigned int i, void *data)
+{
+	if (z->type == BLK_ZONE_TYPE_CONVENTIONAL)
+		return -EOPNOTSUPP;
+	return 0;
+}
+
+bool nvmet_bdev_zns_enable(struct nvmet_ns *ns)
+{
+	struct request_queue *q = ns->bdev->bd_disk->queue;
+	u8 zasl = nvmet_zasl(queue_max_zone_append_sectors(q));
+	struct gendisk *bd_disk = ns->bdev->bd_disk;
+	int ret;
+
+	if (ns->subsys->zasl) {
+		if (ns->subsys->zasl > zasl)
+			return false;
+	}
+	ns->subsys->zasl = zasl;
+
+	/*
+	 * Generic zoned block devices may have a smaller last zone which is
+	 * not supported by ZNS. Exclude zoned drives that have such smaller
+	 * last zone.
+	 */
+	if (get_capacity(bd_disk) & (bdev_zone_sectors(ns->bdev) - 1))
+		return false;
+	/*
+	 * ZNS does not define a conventional zone type. If the underlying
+	 * device has a bitmap set indicating the existence of conventional
+	 * zones, reject the device. Otherwise, use report zones to detect if
+	 * the device has conventional zones.
+	 */
+	if (ns->bdev->bd_disk->queue->conv_zones_bitmap)
+		return false;
+
+	ret = blkdev_report_zones(ns->bdev, 0, blkdev_nr_zones(bd_disk),
+				  validate_conv_zones_cb, NULL);
+	if (ret < 0)
+		return false;
+
+	ns->blksize_shift = blksize_bits(bdev_logical_block_size(ns->bdev));
+
+	return true;
+}
+
+void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req)
+{
+	u8 zasl = req->sq->ctrl->subsys->zasl;
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	struct nvme_id_ctrl_zns *id;
+	u16 status;
+
+	id = kzalloc(sizeof(*id), GFP_KERNEL);
+	if (!id) {
+		status = NVME_SC_INTERNAL;
+		goto out;
+	}
+
+	if (ctrl->ops->get_mdts)
+		id->zasl = min_t(u8, ctrl->ops->get_mdts(ctrl), zasl);
+	else
+		id->zasl = zasl;
+
+	status = nvmet_copy_to_sgl(req, 0, id, sizeof(*id));
+
+	kfree(id);
+out:
+	nvmet_req_complete(req, status);
+}
+
+void nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req)
+{
+	struct nvme_id_ns_zns *id_zns;
+	u64 zsze;
+	u16 status;
+
+	if (le32_to_cpu(req->cmd->identify.nsid) == NVME_NSID_ALL) {
+		req->error_loc = offsetof(struct nvme_identify, nsid);
+		status = NVME_SC_INVALID_NS | NVME_SC_DNR;
+		goto out;
+	}
+
+	id_zns = kzalloc(sizeof(*id_zns), GFP_KERNEL);
+	if (!id_zns) {
+		status = NVME_SC_INTERNAL;
+		goto out;
+	}
+
+	status = nvmet_req_find_ns(req);
+	if (status) {
+		status = NVME_SC_INTERNAL;
+		goto done;
+	}
+
+	if (!bdev_is_zoned(req->ns->bdev)) {
+		req->error_loc = offsetof(struct nvme_identify, nsid);
+		status = NVME_SC_INVALID_NS | NVME_SC_DNR;
+		goto done;
+	}
+
+	nvmet_ns_revalidate(req->ns);
+	zsze = (bdev_zone_sectors(req->ns->bdev) << 9) >>
+					req->ns->blksize_shift;
+	id_zns->lbafe[0].zsze = cpu_to_le64(zsze);
+	id_zns->mor = cpu_to_le32(bdev_max_open_zones(req->ns->bdev));
+	id_zns->mar = cpu_to_le32(bdev_max_active_zones(req->ns->bdev));
+
+done:
+	status = nvmet_copy_to_sgl(req, 0, id_zns, sizeof(*id_zns));
+	kfree(id_zns);
+out:
+	nvmet_req_complete(req, status);
+}
+
+static u16 nvmet_bdev_validate_zone_mgmt_recv(struct nvmet_req *req)
+{
+	sector_t sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
+	u32 out_bufsize = (le32_to_cpu(req->cmd->zmr.numd) + 1) << 2;
+
+	if (sect >= get_capacity(req->ns->bdev->bd_disk)) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, slba);
+		return NVME_SC_LBA_RANGE | NVME_SC_DNR;
+	}
+
+	if (out_bufsize < sizeof(struct nvme_zone_report)) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, numd);
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+	}
+
+	if (req->cmd->zmr.zra != NVME_ZRA_ZONE_REPORT) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, zra);
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+	}
+
+	switch (req->cmd->zmr.pr) {
+	case 0:
+	case 1:
+		break;
+	default:
+		req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, pr);
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+	}
+
+	switch (req->cmd->zmr.zrasf) {
+	case NVME_ZRASF_ZONE_REPORT_ALL:
+	case NVME_ZRASF_ZONE_STATE_EMPTY:
+	case NVME_ZRASF_ZONE_STATE_IMP_OPEN:
+	case NVME_ZRASF_ZONE_STATE_EXP_OPEN:
+	case NVME_ZRASF_ZONE_STATE_CLOSED:
+	case NVME_ZRASF_ZONE_STATE_FULL:
+	case NVME_ZRASF_ZONE_STATE_READONLY:
+	case NVME_ZRASF_ZONE_STATE_OFFLINE:
+		break;
+	default:
+		req->error_loc =
+			offsetof(struct nvme_zone_mgmt_recv_cmd, zrasf);
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+	}
+
+	return NVME_SC_SUCCESS;
+}
+
+struct nvmet_report_zone_data {
+	struct nvmet_req *req;
+	u64 out_buf_offset;
+	u64 out_nr_zones;
+	u64 nr_zones;
+	u8 zrasf;
+};
+
+static int nvmet_bdev_report_zone_cb(struct blk_zone *z, unsigned i, void *d)
+{
+	static const unsigned int nvme_zrasf_to_blk_zcond[] = {
+		[NVME_ZRASF_ZONE_STATE_EMPTY]	 = BLK_ZONE_COND_EMPTY,
+		[NVME_ZRASF_ZONE_STATE_IMP_OPEN] = BLK_ZONE_COND_IMP_OPEN,
+		[NVME_ZRASF_ZONE_STATE_EXP_OPEN] = BLK_ZONE_COND_EXP_OPEN,
+		[NVME_ZRASF_ZONE_STATE_CLOSED]	 = BLK_ZONE_COND_CLOSED,
+		[NVME_ZRASF_ZONE_STATE_READONLY] = BLK_ZONE_COND_READONLY,
+		[NVME_ZRASF_ZONE_STATE_FULL]	 = BLK_ZONE_COND_FULL,
+		[NVME_ZRASF_ZONE_STATE_OFFLINE]	 = BLK_ZONE_COND_OFFLINE,
+	};
+	struct nvmet_report_zone_data *rz = d;
+
+	if (rz->zrasf != NVME_ZRASF_ZONE_REPORT_ALL &&
+	    z->cond != nvme_zrasf_to_blk_zcond[rz->zrasf])
+		return 0;
+
+	if (rz->nr_zones < rz->out_nr_zones) {
+		struct nvme_zone_descriptor zdesc = { };
+		u16 status;
+
+		zdesc.zcap = nvmet_sect_to_lba(rz->req->ns, z->capacity);
+		zdesc.zslba = nvmet_sect_to_lba(rz->req->ns, z->start);
+		zdesc.wp = nvmet_sect_to_lba(rz->req->ns, z->wp);
+		zdesc.za = z->reset ? 1 << 2 : 0;
+		zdesc.zs = z->cond << 4;
+		zdesc.zt = z->type;
+
+		status = nvmet_copy_to_sgl(rz->req, rz->out_buf_offset, &zdesc,
+					   sizeof(zdesc));
+		if (status)
+			return -EINVAL;
+
+		rz->out_buf_offset += sizeof(zdesc);
+	}
+
+	rz->nr_zones++;
+
+	return 0;
+}
+
+static unsigned long nvmet_req_nr_zones_from_slba(struct nvmet_req *req)
+{
+	unsigned int sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
+
+	return blkdev_nr_zones(req->ns->bdev->bd_disk) -
+		(sect >> ilog2(bdev_zone_sectors(req->ns->bdev)));
+}
+
+static unsigned long get_nr_zones_from_buf(struct nvmet_req *req, u32 bufsize)
+{
+	if (bufsize <= sizeof(struct nvme_zone_report))
+		return 0;
+
+	return (bufsize - sizeof(struct nvme_zone_report)) /
+		sizeof(struct nvme_zone_descriptor);
+}
+
+void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
+{
+	struct nvmet_req *req = container_of(w, struct nvmet_req, z.zmgmt_work);
+	sector_t start_sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
+	unsigned long req_slba_nr_zones = nvmet_req_nr_zones_from_slba(req);
+	u32 out_bufsize = (le32_to_cpu(req->cmd->zmr.numd) + 1) << 2;
+	__le64 nr_zones;
+	u16 status;
+	int ret;
+	struct nvmet_report_zone_data rz_data = {
+		.out_nr_zones = get_nr_zones_from_buf(req, out_bufsize),
+		/* leave the place for report zone header */
+		.out_buf_offset = sizeof(struct nvme_zone_report),
+		.zrasf = req->cmd->zmr.zrasf,
+		.nr_zones = 0,
+		.req = req,
+	};
+
+	status = nvmet_bdev_validate_zone_mgmt_recv(req);
+	if (status)
+		goto out;
+
+	if (!req_slba_nr_zones) {
+		status = NVME_SC_SUCCESS;
+		goto out;
+	}
+
+	ret = blkdev_report_zones(req->ns->bdev, start_sect, req_slba_nr_zones,
+				 nvmet_bdev_report_zone_cb, &rz_data);
+	if (ret < 0) {
+		status = NVME_SC_INTERNAL;
+		goto out;
+	}
+
+	/*
+	 * When partial bit is set nr_zones must indicate the number of zone
+	 * descriptors actually transferred.
+	 */
+	if (req->cmd->zmr.pr)
+		rz_data.nr_zones = min(rz_data.nr_zones, rz_data.out_nr_zones);
+
+	nr_zones = cpu_to_le64(rz_data.nr_zones);
+	status = nvmet_copy_to_sgl(req, 0, &nr_zones, sizeof(nr_zones));
+
+out:
+	nvmet_req_complete(req, status);
+}
+
+void nvmet_bdev_execute_zone_mgmt_recv(struct nvmet_req *req)
+{
+	INIT_WORK(&req->z.zmgmt_work, nvmet_bdev_zone_zmgmt_recv_work);
+	queue_work(zbd_wq, &req->z.zmgmt_work);
+}
+
+static inline enum req_opf zsa_req_op(u8 zsa)
+{
+	switch (zsa) {
+	case NVME_ZONE_OPEN:
+		return REQ_OP_ZONE_OPEN;
+	case NVME_ZONE_CLOSE:
+		return REQ_OP_ZONE_CLOSE;
+	case NVME_ZONE_FINISH:
+		return REQ_OP_ZONE_FINISH;
+	case NVME_ZONE_RESET:
+		return REQ_OP_ZONE_RESET;
+	default:
+		return REQ_OP_LAST;
+	}
+}
+
+static u16 blkdev_zone_mgmt_errno_to_nvme_status(int ret)
+{
+	switch (ret) {
+	case 0:
+		return NVME_SC_SUCCESS;
+	case -EINVAL:
+	case -EIO:
+		return NVME_SC_ZONE_INVALID_TRANSITION | NVME_SC_DNR;
+	default:
+		return NVME_SC_INTERNAL;
+	}
+}
+
+struct nvmet_zone_mgmt_send_all_data {
+	unsigned long *zbitmap;
+	struct nvmet_req *req;
+};
+
+static int zmgmt_send_scan_cb(struct blk_zone *z, unsigned i, void *d)
+{
+	struct nvmet_zone_mgmt_send_all_data *data = d;
+
+	switch (zsa_req_op(data->req->cmd->zms.zsa)) {
+	case REQ_OP_ZONE_OPEN:
+		switch (z->cond) {
+		case BLK_ZONE_COND_CLOSED:
+			break;
+		default:
+			return 0;
+		}
+		break;
+	case REQ_OP_ZONE_CLOSE:
+		switch (z->cond) {
+		case BLK_ZONE_COND_IMP_OPEN:
+		case BLK_ZONE_COND_EXP_OPEN:
+			break;
+		default:
+			return 0;
+		}
+		break;
+	case REQ_OP_ZONE_FINISH:
+		switch (z->cond) {
+		case BLK_ZONE_COND_IMP_OPEN:
+		case BLK_ZONE_COND_EXP_OPEN:
+		case BLK_ZONE_COND_CLOSED:
+			break;
+		default:
+			return 0;
+		}
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	set_bit(i, data->zbitmap);
+
+	return 0;
+}
+
+static u16 nvmet_bdev_zone_mgmt_emulate_all(struct nvmet_req *req)
+{
+	struct block_device *bdev = req->ns->bdev;
+	unsigned int nr_zones = blkdev_nr_zones(bdev->bd_disk);
+	struct request_queue *q = bdev_get_queue(bdev);
+	struct bio *bio = NULL;
+	sector_t sector = 0;
+	int ret;
+	struct nvmet_zone_mgmt_send_all_data d = {
+		.req = req,
+	};
+
+	d.zbitmap = kcalloc_node(BITS_TO_LONGS(nr_zones), sizeof(*(d.zbitmap)),
+				 GFP_NOIO, q->node);
+	if (!d.zbitmap) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	/* Scan and build bitmap of the eligible zones */
+	ret = blkdev_report_zones(bdev, 0, nr_zones, zmgmt_send_scan_cb, &d);
+	if (ret != nr_zones) {
+		if (ret > 0)
+			ret = -EIO;
+		goto out;
+	} else {
+		/* We scanned all the zones */
+		ret = 0;
+	}
+
+	while (sector < get_capacity(bdev->bd_disk)) {
+		if (test_bit(blk_queue_zone_no(q, sector), d.zbitmap)) {
+			bio = blk_next_bio(bio, 0, GFP_KERNEL);
+			bio->bi_opf = zsa_req_op(req->cmd->zms.zsa) | REQ_SYNC;
+			bio->bi_iter.bi_sector = sector;
+			bio_set_dev(bio, bdev);
+			/* This may take a while, so be nice to others */
+			cond_resched();
+		}
+		sector += blk_queue_zone_sectors(q);
+	}
+
+	if (bio) {
+		ret = submit_bio_wait(bio);
+		bio_put(bio);
+	}
+
+out:
+	kfree(d.zbitmap);
+
+	return blkdev_zone_mgmt_errno_to_nvme_status(ret);
+}
+
+static u16 nvmet_bdev_execute_zmgmt_send_all(struct nvmet_req *req)
+{
+	int ret;
+
+	switch (zsa_req_op(req->cmd->zms.zsa)) {
+	case REQ_OP_ZONE_RESET:
+		ret = blkdev_zone_mgmt(req->ns->bdev, REQ_OP_ZONE_RESET, 0,
+				       get_capacity(req->ns->bdev->bd_disk),
+				       GFP_KERNEL);
+		if (ret < 0)
+			return blkdev_zone_mgmt_errno_to_nvme_status(ret);
+		break;
+	case REQ_OP_ZONE_OPEN:
+	case REQ_OP_ZONE_CLOSE:
+	case REQ_OP_ZONE_FINISH:
+		return nvmet_bdev_zone_mgmt_emulate_all(req);
+	default:
+		/* this is needed to quiet compiler warning */
+		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, zsa);
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+	}
+
+	return NVME_SC_SUCCESS;
+}
+
+static void nvmet_bdev_zmgmt_send_work(struct work_struct *w)
+{
+	struct nvmet_req *req = container_of(w, struct nvmet_req, z.zmgmt_work);
+	sector_t sect = nvmet_lba_to_sect(req->ns, req->cmd->zms.slba);
+	enum req_opf op = zsa_req_op(req->cmd->zms.zsa);
+	struct block_device *bdev = req->ns->bdev;
+	sector_t zone_sectors = bdev_zone_sectors(bdev);
+	u16 status = NVME_SC_SUCCESS;
+	int ret;
+
+	if (op == REQ_OP_LAST) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, zsa);
+		status = NVME_SC_ZONE_INVALID_TRANSITION | NVME_SC_DNR;
+		goto out;
+	}
+
+	/* when select all bit is set slba field is ignored */
+	if (req->cmd->zms.select_all) {
+		status = nvmet_bdev_execute_zmgmt_send_all(req);
+		goto out;
+	}
+
+	if (sect >= get_capacity(bdev->bd_disk)) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, slba);
+		status = NVME_SC_LBA_RANGE | NVME_SC_DNR;
+		goto out;
+	}
+
+	if (sect & (zone_sectors - 1)) {
+		req->error_loc = offsetof(struct nvme_zone_mgmt_send_cmd, slba);
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		goto out;
+	}
+
+	ret = blkdev_zone_mgmt(bdev, op, sect, zone_sectors, GFP_KERNEL);
+	if (ret < 0)
+		status = blkdev_zone_mgmt_errno_to_nvme_status(ret);
+
+out:
+	nvmet_req_complete(req, status);
+}
+
+void nvmet_bdev_execute_zone_mgmt_send(struct nvmet_req *req)
+{
+	INIT_WORK(&req->z.zmgmt_work, nvmet_bdev_zmgmt_send_work);
+	queue_work(zbd_wq, &req->z.zmgmt_work);
+}
+
+static void nvmet_bdev_zone_append_bio_done(struct bio *bio)
+{
+	struct nvmet_req *req = bio->bi_private;
+
+	if (bio->bi_status == BLK_STS_OK) {
+		req->cqe->result.u64 =
+			nvmet_sect_to_lba(req->ns, bio->bi_iter.bi_sector);
+	}
+
+	nvmet_req_complete(req, blk_to_nvme_status(req, bio->bi_status));
+	nvmet_req_bio_put(req, bio);
+}
+
+void nvmet_bdev_execute_zone_append(struct nvmet_req *req)
+{
+	sector_t sect = nvmet_lba_to_sect(req->ns, req->cmd->rw.slba);
+	u16 nr_sect = nvmet_lba_to_sect(req->ns, req->cmd->rw.length);
+	u16 status = NVME_SC_SUCCESS;
+	unsigned int total_len = 0;
+	struct scatterlist *sg;
+	struct bio *bio;
+	int sg_cnt;
+
+	/* Request is completed on len mismatch in nvmet_check_transter_len() */
+	if (!nvmet_check_transfer_len(req, nvmet_rw_data_len(req)))
+		return;
+
+	if (!req->sg_cnt) {
+		nvmet_req_complete(req, 0);
+		return;
+	}
+
+	if (sect >= get_capacity(req->ns->bdev->bd_disk)) {
+		req->error_loc = offsetof(struct nvme_rw_command, slba);
+		status = NVME_SC_LBA_RANGE | NVME_SC_DNR;
+		goto out;
+	}
+
+	if ((sect + nr_sect > get_capacity(req->ns->bdev->bd_disk))) {
+		req->error_loc = offsetof(struct nvme_rw_command, length);
+		status = NVME_SC_LBA_RANGE | NVME_SC_DNR;
+		goto out;
+	}
+
+	if (sect & (bdev_zone_sectors(req->ns->bdev) - 1)) {
+		req->error_loc = offsetof(struct nvme_rw_command, slba);
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		goto out;
+	}
+
+	if (nvmet_use_inline_bvec(req)) {
+		bio = &req->z.inline_bio;
+		bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec));
+	} else {
+		bio = bio_alloc(GFP_KERNEL, req->sg_cnt);
+	}
+
+	bio->bi_opf = REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE;
+	bio->bi_end_io = nvmet_bdev_zone_append_bio_done;
+	bio_set_dev(bio, req->ns->bdev);
+	bio->bi_iter.bi_sector = sect;
+	bio->bi_private = req;
+	if (req->cmd->rw.control & cpu_to_le16(NVME_RW_FUA))
+		bio->bi_opf |= REQ_FUA;
+
+	for_each_sg(req->sg, sg, req->sg_cnt, sg_cnt) {
+		struct page *p = sg_page(sg);
+		unsigned int l = sg->length;
+		unsigned int o = sg->offset;
+		unsigned int ret;
+
+		ret = bio_add_zone_append_page(bio, p, l, o);
+		if (ret != sg->length) {
+			status = NVME_SC_INTERNAL;
+			goto out_put_bio;
+		}
+		total_len += sg->length;
+	}
+
+	if (total_len != nvmet_rw_data_len(req)) {
+		status = NVME_SC_INTERNAL | NVME_SC_DNR;
+		goto out_put_bio;
+	}
+
+	submit_bio(bio);
+	return;
+
+out_put_bio:
+	nvmet_req_bio_put(req, bio);
+out:
+	nvmet_req_complete(req, status);
+}
+
+u16 nvmet_bdev_zns_parse_io_cmd(struct nvmet_req *req)
+{
+	struct nvme_command *cmd = req->cmd;
+
+	switch (cmd->common.opcode) {
+	case nvme_cmd_zone_append:
+		req->execute = nvmet_bdev_execute_zone_append;
+		return 0;
+	case nvme_cmd_zone_mgmt_recv:
+		req->execute = nvmet_bdev_execute_zone_mgmt_recv;
+		return 0;
+	case nvme_cmd_zone_mgmt_send:
+		req->execute = nvmet_bdev_execute_zone_mgmt_send;
+		return 0;
+	default:
+		return nvmet_bdev_parse_io_cmd(req);
+	}
+}
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index c7ba83144d52..cb1197f1cfed 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -944,6 +944,13 @@ struct nvme_zone_mgmt_recv_cmd {
 enum {
 	NVME_ZRA_ZONE_REPORT		= 0,
 	NVME_ZRASF_ZONE_REPORT_ALL	= 0,
+	NVME_ZRASF_ZONE_STATE_EMPTY	= 0x01,
+	NVME_ZRASF_ZONE_STATE_IMP_OPEN	= 0x02,
+	NVME_ZRASF_ZONE_STATE_EXP_OPEN	= 0x03,
+	NVME_ZRASF_ZONE_STATE_CLOSED	= 0x04,
+	NVME_ZRASF_ZONE_STATE_READONLY	= 0x05,
+	NVME_ZRASF_ZONE_STATE_FULL	= 0x06,
+	NVME_ZRASF_ZONE_STATE_OFFLINE	= 0x07,
 	NVME_REPORT_ZONE_PARTIAL	= 1,
 };
 
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun  9 19:45:39 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed,  9 Jun 2021 19:45:39 -0700
Subject: [PATCH] Revert "nvme: verify MNAN value if ANA is enabled"
Message-ID: <20210610024539.28875-1-chaitanya.kulkarni@wdc.com>

This reverts commit 8872c159c7a83daf633768cee7a7ef7154010341. This is
needed to move forward with the blktests for now, without this patch
all the testcases result in the error :-

[ 3502.072798] nvme nvme1: Invalid MNAN value 1024

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/multipath.c | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 23573fe3fc7d..a501917d4c6e 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -813,13 +813,6 @@ int nvme_mpath_init_identify(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 	    !(ctrl->subsys->cmic & NVME_CTRL_CMIC_ANA))
 		return 0;
 
-	if (!ctrl->max_namespaces ||
-	    ctrl->max_namespaces > le32_to_cpu(id->nn)) {
-		dev_err(ctrl->device,
-			"Invalid MNAN value %u\n", ctrl->max_namespaces);
-		return -EINVAL;
-	}
-
 	ctrl->anacap = id->anacap;
 	ctrl->anatt = id->anatt;
 	ctrl->nanagrpid = le32_to_cpu(id->nanagrpid);
-- 
2.22.1



From Chaitanya.Kulkarni at wdc.com  Wed Jun  9 19:55:08 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Thu, 10 Jun 2021 02:55:08 +0000
Subject: [PATCH] Revert "nvme: verify MNAN value if ANA is enabled"
References: <20210610024539.28875-1-chaitanya.kulkarni@wdc.com>
Message-ID: <BYAPR04MB49655A9B821FEE29D7A89E3986359@BYAPR04MB4965.namprd04.prod.outlook.com>

Daniel,

On 6/9/21 19:45, Chaitanya Kulkarni wrote:
> This reverts commit 8872c159c7a83daf633768cee7a7ef7154010341. This is
> needed to move forward with the blktests for now, without this patch
> all the testcases result in the error :-
>
> [ 3502.072798] nvme nvme1: Invalid MNAN value 1024
>
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

Thinking about the code again I think following should work :-

diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 23573fe3fc7d..4277f1554bd5 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -813,7 +813,7 @@ int nvme_mpath_init_identify(struct nvme_ctrl *ctrl,
struct nvme_id_ctrl *id)
            !(ctrl->subsys->cmic & NVME_CTRL_CMIC_ANA))
                return 0;
 
-       if (!ctrl->max_namespaces ||
+       if (ctrl->max_namespaces &&
            ctrl->max_namespaces > le32_to_cpu(id->nn)) {
                dev_err(ctrl->device,
                        "Invalid MNAN value %u\n", ctrl->max_namespaces);
diff --git a/drivers/nvme/target/admin-cmd.c
b/drivers/nvme/target/admin-cmd.c
index cd60a8184d04..a8ec377bb68d 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -394,7 +394,7 @@ static void nvmet_execute_identify_ctrl(struct
nvmet_req *req)
        id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
 
        id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
-       id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
+       id->mnan = cpu_to_le32(ctrl->subsys->max_nsid);
        id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
                        NVME_CTRL_ONCS_WRITE_ZEROES);
 

If it does, then we can drop the this revert and I'll send above 2 fixes.

Daniel, can you please confirm ?




From Chaitanya.Kulkarni at wdc.com  Wed Jun  9 20:53:21 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Thu, 10 Jun 2021 03:53:21 +0000
Subject: [PATCH V6] nvme-core: use xarray for ctrl ns tracking
References: <20210608024247.35286-1-chaitanya.kulkarni@wdc.com>
 <dbab5916-a6de-5cd0-4bb5-b06d0349df0b@grimberg.me>
 <BYAPR04MB49657620DF9C7EA9158DB60686369@BYAPR04MB4965.namprd04.prod.outlook.com>
Message-ID: <BYAPR04MB49651675991A7EC80AB1836186359@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/9/21 13:48, Chaitanya Kulkarni wrote:
>>>   
>>>   static int nvme_scan_ns_list(struct nvme_ctrl *ctrl)
>>> @@ -4068,10 +4051,6 @@ static void nvme_scan_work(struct work_struct *work)
>>>   	if (nvme_scan_ns_list(ctrl) != 0)
>>>   		nvme_scan_ns_sequential(ctrl);
>>>   	mutex_unlock(&ctrl->scan_lock);
>>> -
>>> -	down_write(&ctrl->namespaces_rwsem);
>>> -	list_sort(NULL, &ctrl->namespaces, ns_cmp);
>>> -	up_write(&ctrl->namespaces_rwsem);
>> Is xarray sorted by nsid? ana updates rely on ctrl->namespaces to be
>> sorted. Where is nvme_update_ana_state change anyway? you tested this
>> without CONFIG_NVME_MULTIPATH?
>>

We insert the code ns into xarray with nsid as index so there is 1:1 mapping
between the ns, nsid, and the array index. Given that it is an array in
iteration it always starts with the first element. Hence no sorting is
needed.

To check the correctness of the code (with multipath enabled) I used the
quick
debug.patch [1] and here is the the output [2].

nvme: print all the ns in id-ns from xarray

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 6fdbb0a22f3b..1f33fd46bf34 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1425,6 +1425,8 @@ static int nvme_identify_ns(struct nvme_ctrl
*ctrl, unsigned nsid,
 {
        struct nvme_command c = { };
        int error;
+       struct nvme_ns *ns;
+       unsigned long idx;
 
        /* gcc-4.4.4 (at least) has issues with initializers and anon
unions */
        c.identify.opcode = nvme_admin_identify;
@@ -1452,6 +1454,10 @@ static int nvme_identify_ns(struct nvme_ctrl
*ctrl, unsigned nsid,
            !memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
                memcpy(ids->nguid, (*id)->nguid, sizeof(ids->nguid));
 
+       xa_for_each(&ctrl->namespaces, idx, ns)
+               pr_info("%s %d nsid %d\n", __func__, __LINE__,
ns->head->ns_id);
+
+       pr_info("%s %d------------------------------", __func__, __LINE__);
        return 0;
 
 out_free_id:


Creation of the bdev controller with xarray (/dev/nvme1) and using that
as an
input to the passthru controller (/dev/nvme2), iterating over the xarray and
printing all the elements. All the ns are printed in sorted manner for the
passthru controller :-

[  229.877166] nvmet: creating controller 2 for subsystem pt-nqn for NQN
nqn.2014-08.org.nvmexpress:uuid:e4cfc949-8f19-4db2-a232-ab360b79204a.
[  229.880143] nvme nvme2: creating 64 I/O queues.
[  229.901741] nvme nvme2: new ctrl: "pt-nqn"
[  229.902230] nvme_identify_ns 1460------------------------------
[  229.932624] nvme_identify_ns 1458 nsid 1
[  229.932634] nvme_identify_ns 1460------------------------------
[  229.961368] nvme_identify_ns 1458 nsid 1
[  229.961375] nvme_identify_ns 1458 nsid 2
[  229.961381] nvme_identify_ns 1460------------------------------
[  229.986053] nvme_identify_ns 1458 nsid 1
[  229.986060] nvme_identify_ns 1458 nsid 2
[  229.986063] nvme_identify_ns 1458 nsid 3
[  229.986068] nvme_identify_ns 1460------------------------------
[  230.010679] nvme_identify_ns 1458 nsid 1
[  230.010686] nvme_identify_ns 1458 nsid 2
[  230.010689] nvme_identify_ns 1458 nsid 3
[  230.010691] nvme_identify_ns 1458 nsid 4
[  230.010696] nvme_identify_ns 1460------------------------------
[  230.034748] nvme_identify_ns 1458 nsid 1
[  230.034755] nvme_identify_ns 1458 nsid 2
[  230.034758] nvme_identify_ns 1458 nsid 3
[  230.034761] nvme_identify_ns 1458 nsid 4
[  230.034763] nvme_identify_ns 1458 nsid 5
[  230.034769] nvme_identify_ns 1460------------------------------
[  230.058996] nvme_identify_ns 1458 nsid 1
[  230.059003] nvme_identify_ns 1458 nsid 2
[  230.059006] nvme_identify_ns 1458 nsid 3
[  230.059009] nvme_identify_ns 1458 nsid 4
[  230.059012] nvme_identify_ns 1458 nsid 5
[  230.059014] nvme_identify_ns 1458 nsid 6
[  230.059019] nvme_identify_ns 1460------------------------------
[  230.083433] nvme_identify_ns 1458 nsid 1
[  230.083440] nvme_identify_ns 1458 nsid 2
[  230.083443] nvme_identify_ns 1458 nsid 3
[  230.083446] nvme_identify_ns 1458 nsid 4
[  230.083450] nvme_identify_ns 1458 nsid 5
[  230.083452] nvme_identify_ns 1458 nsid 6
[  230.083455] nvme_identify_ns 1458 nsid 7
[  230.083459] nvme_identify_ns 1460------------------------------
[  230.107702] nvme_identify_ns 1458 nsid 1
[  230.107709] nvme_identify_ns 1458 nsid 2
[  230.107744] nvme_identify_ns 1458 nsid 3
[  230.107748] nvme_identify_ns 1458 nsid 4
[  230.107751] nvme_identify_ns 1458 nsid 5
[  230.107754] nvme_identify_ns 1458 nsid 6
[  230.107756] nvme_identify_ns 1458 nsid 7
[  230.107758] nvme_identify_ns 1458 nsid 8
[  230.107763] nvme_identify_ns 1460------------------------------
[  230.132132] nvme_identify_ns 1458 nsid 1
[  230.132139] nvme_identify_ns 1458 nsid 2
[  230.132141] nvme_identify_ns 1458 nsid 3
[  230.132144] nvme_identify_ns 1458 nsid 4
[  230.132147] nvme_identify_ns 1458 nsid 5
[  230.132149] nvme_identify_ns 1458 nsid 6
[  230.132152] nvme_identify_ns 1458 nsid 7
[  230.132154] nvme_identify_ns 1458 nsid 8
[  230.132156] nvme_identify_ns 1458 nsid 9



Now disable the 5th and 8th namespace from target that should leave nid
5 & 8
but rest of the namespaces should be printed in the ascending order
in the ns rescan sequence :-


# echo 0 > /sys/kernel/config/nvmet/subsystems/x/namespaces/5/enable
# echo 0 > /sys/kernel/config/nvmet/subsystems/x/namespaces/8/enable

[  620.836412] nvme nvme1: rescanning namespaces.
[  620.836985] nvme_identify_ns 1458 nsid 1
[  620.836993] nvme_identify_ns 1458 nsid 2
[  620.836998] nvme_identify_ns 1458 nsid 3
[  620.837003] nvme_identify_ns 1458 nsid 4
[  620.837007] nvme_identify_ns 1458 nsid 6
[  620.837012] nvme_identify_ns 1458 nsid 7
[  620.837017] nvme_identify_ns 1458 nsid 8
[  620.837021] nvme_identify_ns 1458 nsid 9
[  620.837026] nvme_identify_ns 1458 nsid 10
[  620.837034] nvme_identify_ns 1460------------------------------
[  620.857901] nvme_identify_ns 1458 nsid 1
[  620.857909] nvme_identify_ns 1458 nsid 2
[  620.857913] nvme_identify_ns 1458 nsid 3
[  620.857929] nvme_identify_ns 1458 nsid 4
[  620.857933] nvme_identify_ns 1458 nsid 6
[  620.857937] nvme_identify_ns 1458 nsid 7
[  620.857940] nvme_identify_ns 1458 nsid 8
[  620.857944] nvme_identify_ns 1458 nsid 9
[  620.857948] nvme_identify_ns 1458 nsid 10
[  620.857956] nvme_identify_ns 1460------------------------------
[  620.873913] nvme_identify_ns 1458 nsid 1
[  620.873921] nvme_identify_ns 1458 nsid 2
[  620.873925] nvme_identify_ns 1458 nsid 3
[  620.873939] nvme_identify_ns 1458 nsid 4
[  620.873943] nvme_identify_ns 1458 nsid 6
[  620.873947] nvme_identify_ns 1458 nsid 7
[  620.873951] nvme_identify_ns 1458 nsid 8
[  620.873955] nvme_identify_ns 1458 nsid 9
[  620.873960] nvme_identify_ns 1458 nsid 10
[  620.873969] nvme_identify_ns 1460------------------------------
[  620.887976] nvme_identify_ns 1458 nsid 1
[  620.887983] nvme_identify_ns 1458 nsid 2
[  620.887988] nvme_identify_ns 1458 nsid 3
[  620.887991] nvme_identify_ns 1458 nsid 4
[  620.887995] nvme_identify_ns 1458 nsid 6
[  620.887999] nvme_identify_ns 1458 nsid 7
[  620.888002] nvme_identify_ns 1458 nsid 8
[  620.888006] nvme_identify_ns 1458 nsid 9
[  620.888010] nvme_identify_ns 1458 nsid 10
[  620.888018] nvme_identify_ns 1460------------------------------
[  620.901911] nvme_identify_ns 1458 nsid 1
[  620.901919] nvme_identify_ns 1458 nsid 2
[  620.901924] nvme_identify_ns 1458 nsid 3
[  620.901928] nvme_identify_ns 1458 nsid 4
[  620.901932] nvme_identify_ns 1458 nsid 6
[  620.901936] nvme_identify_ns 1458 nsid 7
[  620.901941] nvme_identify_ns 1458 nsid 8
[  620.901945] nvme_identify_ns 1458 nsid 9
[  620.901948] nvme_identify_ns 1458 nsid 10
[  620.901957] nvme_identify_ns 1460------------------------------
[  620.919913] nvme_identify_ns 1458 nsid 1
[  620.919921] nvme_identify_ns 1458 nsid 2
[  620.919926] nvme_identify_ns 1458 nsid 3
[  620.919930] nvme_identify_ns 1458 nsid 4
[  620.919934] nvme_identify_ns 1458 nsid 6
[  620.919938] nvme_identify_ns 1458 nsid 7
[  620.919942] nvme_identify_ns 1458 nsid 8
[  620.919946] nvme_identify_ns 1458 nsid 9
[  620.919950] nvme_identify_ns 1458 nsid 10
[  620.919959] nvme_identify_ns 1460------------------------------
[  620.933910] nvme_identify_ns 1458 nsid 1
[  620.933918] nvme_identify_ns 1458 nsid 2
[  620.933923] nvme_identify_ns 1458 nsid 3
[  620.933927] nvme_identify_ns 1458 nsid 4
[  620.933931] nvme_identify_ns 1458 nsid 6
[  620.933935] nvme_identify_ns 1458 nsid 7
[  620.933939] nvme_identify_ns 1458 nsid 8
[  620.933943] nvme_identify_ns 1458 nsid 9
[  620.933947] nvme_identify_ns 1458 nsid 10
[  620.933956] nvme_identify_ns 1460------------------------------
[  621.022320] nvme_identify_ns 1458 nsid 1
[  621.022330] nvme_identify_ns 1458 nsid 2
[  621.022335] nvme_identify_ns 1458 nsid 3
[  621.022340] nvme_identify_ns 1458 nsid 4
[  621.022344] nvme_identify_ns 1458 nsid 6
[  621.022348] nvme_identify_ns 1458 nsid 7
[  621.022352] nvme_identify_ns 1458 nsid 9
[  621.022355] nvme_identify_ns 1458 nsid 10

Hope this answers your questions.




From hare at suse.de  Wed Jun  9 22:49:14 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 10 Jun 2021 07:49:14 +0200
Subject: [LSF/MM/BPF TOPIC] block namespaces
In-Reply-To: <485837f392401bf35fb7fc8231d7a051f47b53d7.camel@HansenPartnership.com>
References: <a189ec50-4c11-9ee9-0b9e-b492507adc1e@suse.de>
 <485837f392401bf35fb7fc8231d7a051f47b53d7.camel@HansenPartnership.com>
Message-ID: <539b35c6-34f7-62e0-4d93-6d27145bb78f@suse.de>

On 6/9/21 8:36 PM, James Bottomley wrote:
> On Thu, 2021-05-27 at 10:01 +0200, Hannes Reinecke wrote:
>> Hi all,
>>
>> I guess it's time to tick off yet another item on my long-term to-do
>> list:
>>
>> Block namespaces
>> ----------------
>>
>> Idea is similar to what network already does: allowing each user
>> namespace to have a different 'view' on the existing block devices.
>> EG if the admin creates a ramdisk in one namespace this device should
>> not be visible to other namespaces.
>> But for me the most important use-case would be qemu; currently the
>> devices need to be set up in the host, even though the host has no
>> business touching it as they really belong to the qemu instance. This
>> is causing quite some irritation eg when this device has LVM or MD
>> metadata and udev is trying to activate it on the host.
> 
> I suppose the first question is "why block only?"  There are several
> existing device namespace proposals which would be more generic.
> 

Well; I'm more of a storage person, and do know the needs and 
shortcomings in that area. Less well so in other areas...

>> Overall plan is to restrict views of '/dev', '/sys/dev/block' and
>> '/sys/block' to only present the devices 'visible' for this
>> namespace.
> 
> We actually already have a devices cgroup that does some of this:
> 
> https://www.kernel.org/doc/Documentation/cgroup-v1/devices.txt
> 
I know. But this essentially is a filter on '/dev' only, and needs to be 
configured. Which makes it very unwieldy to use.
And the contents of sysfs are not modified, so there's a mismatch 
between contents in /dev and /sys.
Which might cause issues with monitoring tools.

> However, visibility isn't the only problem, for direct passthrough
> there's also uevent handling and people have even asked about module
> loading.
> 
I am aware, and that's another reason why device cgroup doesn't cut it.

>>   Initially the drivers would keep their global enumeration, but plan
>> is to make the drivers namespace-aware, too, such that each namespace
>> could have its own driver-specific device enumeration.
> 
> I really wouldn't do this.  Namespace/Cgroup separation should be kept
> as high as possible.  If it leaks into the drivers it will become
> unmaintainable.  Why do you think you need the drivers to be aware?  If
> it's just enumeration, that should all be doable with the visibility
> driver unless you want to do things like compact numbering?
> 
Which is precisely why I mentioned device modifications.
On a generic level we can influence the visibility of devices in 
relation to namespaces, we cannot influence the devices themselves.
This will lead to namespaces seeing disjunct device numbers (ie 8:0 and 
8:8 on ns 1, 8:4 on ns 2). Not that I think that will be an issue, but 
certainly a change in behaviour.

>> Goal of this topic is to get a consensus on whether block namespaces
>> are a feature which would find interest, and also to discuss some
>> design details here:
>> - Only in certain cases can a namespace be assigned (eg by calling
>> 'modprobe', starting iscsiadm, or calling nvme-cli); how do we handle
>> devices for which no namespace can be identified?
>> - Shall we allow for different device enumeration per namespace?
>> - Into which level should we go with hiding sysfs structures?
>>    Is blanking out the higher-level interfaces in /dev and /sys/block
>>    enough?
> 
> First question is does the device cgroup do enough for you and if not
> what's missing?
> 
See above. sysfs modifications and uevent filtering are missing.
This infrastructure for that is already in place thanks to network 
namespaces, we 'just' need to make use of it.
Additional drawback is the manual configuration of device-cgroup.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From dwagner at suse.de  Thu Jun 10 00:45:46 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Thu, 10 Jun 2021 09:45:46 +0200
Subject: [PATCH] Revert "nvme: verify MNAN value if ANA is enabled"
In-Reply-To: <BYAPR04MB49655A9B821FEE29D7A89E3986359@BYAPR04MB4965.namprd04.prod.outlook.com>
References: <20210610024539.28875-1-chaitanya.kulkarni@wdc.com>
 <BYAPR04MB49655A9B821FEE29D7A89E3986359@BYAPR04MB4965.namprd04.prod.outlook.com>
Message-ID: <20210610074546.jyqjalpldii6reg6@beryllium.lan>

Hi Chaitanya,

On Thu, Jun 10, 2021 at 02:55:08AM +0000, Chaitanya Kulkarni wrote:
> On 6/9/21 19:45, Chaitanya Kulkarni wrote:
> > This reverts commit 8872c159c7a83daf633768cee7a7ef7154010341. This is
> > needed to move forward with the blktests for now, without this patch
> > all the testcases result in the error :-
> >
> > [ 3502.072798] nvme nvme1: Invalid MNAN value 1024
> 
> Thinking about the code again I think following should work :-
> 
> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
> index 23573fe3fc7d..4277f1554bd5 100644
> --- a/drivers/nvme/host/multipath.c
> +++ b/drivers/nvme/host/multipath.c
> @@ -813,7 +813,7 @@ int nvme_mpath_init_identify(struct nvme_ctrl *ctrl,
> struct nvme_id_ctrl *id)
>             !(ctrl->subsys->cmic & NVME_CTRL_CMIC_ANA))
>                 return 0;
>  
> -       if (!ctrl->max_namespaces ||
> +       if (ctrl->max_namespaces &&
>             ctrl->max_namespaces > le32_to_cpu(id->nn)) {
>                 dev_err(ctrl->device,
>                         "Invalid MNAN value %u\n", ctrl->max_namespaces);

'!ctrl->max_namespace' could also be written as
'ctrl->max_namespace != 0' which makes it more obvious what the intend
is here:

  If the controller supports Asymmetric Namespace Access Reporting, then
  this field shall be set to a non-zero value that is less than or equal
  to the NN value.


> diff --git a/drivers/nvme/target/admin-cmd.c
> b/drivers/nvme/target/admin-cmd.c
> index cd60a8184d04..a8ec377bb68d 100644
> --- a/drivers/nvme/target/admin-cmd.c
> +++ b/drivers/nvme/target/admin-cmd.c
> @@ -394,7 +394,7 @@ static void nvmet_execute_identify_ctrl(struct
> nvmet_req *req)
>         id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
>  
>         id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
> -       id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
> +       id->mnan = cpu_to_le32(ctrl->subsys->max_nsid);
>         id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
>                         NVME_CTRL_ONCS_WRITE_ZEROES);

This looks like the right fix for the upper limit problem.
I can see the tests are still failing with

  nvme nvme0: Invalid MNAN value 0

which indicates there is another problem in the nvmet core.

I try to find it.x

Thanks,
Daniel


From maz at kernel.org  Thu Jun 10 01:06:19 2021
From: maz at kernel.org (Marc Zyngier)
Date: Thu, 10 Jun 2021 09:06:19 +0100
Subject: linux-next: NVME using PCI legacy interrupts only
In-Reply-To: <52371274-20bc-a150-a3ed-ba3e1305ad3e@gmail.com>
References: <52371274-20bc-a150-a3ed-ba3e1305ad3e@gmail.com>
Message-ID: <87k0n2qh38.wl-maz@kernel.org>

On Wed, 09 Jun 2021 19:43:57 +0100,
Heiner Kallweit <hkallweit1 at gmail.com> wrote:
> 
> I found that on linux-next from June 8th my nvme disk is using legacy
> interrupts only. Some debugging lead me to irq_find_mapping() in
> msi_domain_alloc() returning -EEXIST.
> 
> The nvme core first allocates a MSI-X interrupt for setup purposes
> and later frees it and allocates the final number of MSI-X interrupts.
> 
> The following experimental change brought back the MSI-X interrupts.
> This makes me think that somehow freeing a MSI-X interrupt doesn't
> free it completely. I didn't see this behavior a few days ago,
> therefore I think it's related to the recent changes to
> irqdomain/genirq.
> 
> Didn't do a bisect yet, maybe you have an idea already.

Yeah, recent changes in the irqdomain subsystem seem to have uncovered
a long standing issue where we are leaving dangling references in some
domains....

I've now dropped the branch from -next while I figure it out.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.


From amit.engel at dell.com  Thu Jun 10 01:41:12 2021
From: amit.engel at dell.com (amit.engel at dell.com)
Date: Thu, 10 Jun 2021 11:41:12 +0300
Subject: [PATCH] nvmet-tcp: Add a multex lock to start_queue failure case
Message-ID: <20210610084112.30400-1-amit.engel@dell.com>

From: Amit Engel <amit.engel at dell.com>

this is in order to serialize nvme_tcp_stop_queue

Signed-off-by: Amit Engel <amit.engel at dell.com>
---
 drivers/nvme/host/tcp.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 6a65b0516180..091f861c2067 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1571,6 +1571,7 @@ static void nvme_tcp_stop_queue(struct nvme_ctrl *nctrl, int qid)
 static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx)
 {
 	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
+	struct nvme_tcp_queue *queue = &ctrl->queues[idx];
 	int ret;
 
 	if (idx)
@@ -1579,10 +1580,12 @@ static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx)
 		ret = nvmf_connect_admin_queue(nctrl);
 
 	if (!ret) {
-		set_bit(NVME_TCP_Q_LIVE, &ctrl->queues[idx].flags);
+		set_bit(NVME_TCP_Q_LIVE, &queue->flags);
 	} else {
-		if (test_bit(NVME_TCP_Q_ALLOCATED, &ctrl->queues[idx].flags))
-			__nvme_tcp_stop_queue(&ctrl->queues[idx]);
+		mutex_lock(&queue->queue_lock);
+		if (test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags))
+			__nvme_tcp_stop_queue(queue);
+		mutex_unlock(&queue->queue_lock);
 		dev_err(nctrl->device,
 			"failed to connect queue: %d ret=%d\n", idx, ret);
 	}
-- 
2.18.2



From Amit.Engel at Dell.com  Thu Jun 10 01:44:12 2021
From: Amit.Engel at Dell.com (Engel, Amit)
Date: Thu, 10 Jun 2021 08:44:12 +0000
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at
 0000000000000230
In-Reply-To: <CO1PR19MB48859E1CCD596402D4007458EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
References: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <CO1PR19MB4885314DCBFEB376C7305C98EE3D9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <e40414a9-41ef-f82c-7d1c-2583d8251c26@grimberg.me>
 <CO1PR19MB48850B44ED23C179A4E4F541EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
 <b9c7e6cc-fea5-b8ce-af07-7d004116e3e3@grimberg.me>
 <CO1PR19MB48859C3254E4F3EA3282EDC9EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
 <a76ecf8e-15d0-1ebc-aaf2-c93fee591602@grimberg.me>
 <CO1PR19MB48859E1CCD596402D4007458EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
Message-ID: <CO1PR19MB48851BDA450C1CAF23965958EE359@CO1PR19MB4885.namprd19.prod.outlook.com>

We are trying to reproduce with the upstream code now, will update

In addition, I just posted a patch to add mutex lock also to start queue failure case, as suggested by Sagi.

Thanks
Amit 

-----Original Message-----
From: Engel, Amit 
Sent: Wednesday, June 9, 2021 2:14 PM
To: 'Sagi Grimberg'; linux-nvme at lists.infradead.org
Cc: Anner, Ran; Grupi, Elad
Subject: RE: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at 0000000000000230

Correct, free_queue is being called (sock->sk becomes NULL) before restore_sock_calls

When restore_sock_calls is called, we fail on 'write_lock_bh(&sock->sk->sk_callback_lock)' 

NULL pointer dereference at 0x230 ? 560 decimal
crash> struct sock -o
struct sock {
   [0] struct sock_common __sk_common;
   ?
   ...
   ?
   [560] rwlock_t sk_callback_lock;

stop queue in ctx2 does not really do anything since 'NVME_TCP_Q_LIVE' bit is already cleared (by ctx1).
can you please explain how stop the queue before free helps to serialize ctx1 ?

The race we are describing is based on the panic bt that I shared.

maybe our analysis is not accurate? 

Thanks,
Amit

-----Original Message-----
From: Sagi Grimberg <sagi at grimberg.me> 
Sent: Wednesday, June 9, 2021 12:11 PM
To: Engel, Amit; linux-nvme at lists.infradead.org
Cc: Anner, Ran; Grupi, Elad
Subject: Re: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at 0000000000000230


[EXTERNAL EMAIL] 


> Im not sure that using the queue_lock mutex ill help The race in this 
> case is between sock_release and nvme_tcp_restore_sock_calls 
> sock_release is being called as part of nvme_tcp_free_queue which is 
> destroying the mutex

Maybe I'm not understanding the issue here. What is the scenario again?
stop_queue is called (ctx1), that triggers error_recovery (ctx2) which then calls free_queue before ctx1 gets to restore sock callbacks?

err_work will first stop the queues before freeing them, so it will serialize behind ctx1. What am I missing?

From dwagner at suse.de  Thu Jun 10 04:51:26 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Thu, 10 Jun 2021 13:51:26 +0200
Subject: [PATCH] Revert "nvme: verify MNAN value if ANA is enabled"
In-Reply-To: <20210610074546.jyqjalpldii6reg6@beryllium.lan>
References: <20210610024539.28875-1-chaitanya.kulkarni@wdc.com>
 <BYAPR04MB49655A9B821FEE29D7A89E3986359@BYAPR04MB4965.namprd04.prod.outlook.com>
 <20210610074546.jyqjalpldii6reg6@beryllium.lan>
Message-ID: <20210610115126.r3fxxtgnuyq447su@beryllium.lan>

On Thu, Jun 10, 2021 at 09:45:46AM +0200, Daniel Wagner wrote:
> >         id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
> > -       id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
> > +       id->mnan = cpu_to_le32(ctrl->subsys->max_nsid);
> >         id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
> >                         NVME_CTRL_ONCS_WRITE_ZEROES);
> 

I am a bit confused how the target code indexes the namespaces. In 6.1
Namespaces the valid range for NSID is defined from 0 to namespace
-1. In 6.1.2 it says 'Any NSID is valid, except if that NSID is 0h or
greater than the Number of Namespaces'. And the figure 348 shows clearly
NSID 0 is not valid.

Looking through the code I think the nsid range used starts at 0 and not
at 1. Which would explain why the above change is not enough. The valid
range for max_nsid should start at 1 and not 0.

Thoughts?


From James.Bottomley at HansenPartnership.com  Thu Jun 10 07:29:18 2021
From: James.Bottomley at HansenPartnership.com (James Bottomley)
Date: Thu, 10 Jun 2021 07:29:18 -0700
Subject: [LSF/MM/BPF TOPIC] block namespaces
In-Reply-To: <539b35c6-34f7-62e0-4d93-6d27145bb78f@suse.de>
References: <a189ec50-4c11-9ee9-0b9e-b492507adc1e@suse.de>
 <485837f392401bf35fb7fc8231d7a051f47b53d7.camel@HansenPartnership.com>
 <539b35c6-34f7-62e0-4d93-6d27145bb78f@suse.de>
Message-ID: <f31fbb6d2f374a39d22e3fca122d757e905d2711.camel@HansenPartnership.com>

On Thu, 2021-06-10 at 07:49 +0200, Hannes Reinecke wrote:
> On 6/9/21 8:36 PM, James Bottomley wrote:
> > On Thu, 2021-05-27 at 10:01 +0200, Hannes Reinecke wrote:
> > > Hi all,
> > > 
> > > I guess it's time to tick off yet another item on my long-term
> > > to-do list:
> > > 
> > > Block namespaces
> > > ----------------
> > > 
> > > Idea is similar to what network already does: allowing each user
> > > namespace to have a different 'view' on the existing block
> > > devices.  EG if the admin creates a ramdisk in one namespace this
> > > device should not be visible to other namespaces.  But for me the
> > > most important use-case would be qemu; currently the devices need
> > > to be set up in the host, even though the host has no business
> > > touching it as they really belong to the qemu instance.  This
> > > is causing quite some irritation eg when this device has LVM or
> > > MD metadata and udev is trying to activate it on the host.
> > 
> > I suppose the first question is "why block only?"  There are
> > several existing device namespace proposals which would be more
> > generic.
> > 
> 
> Well; I'm more of a storage person, and do know the needs and 
> shortcomings in that area. Less well so in other areas...

OK, but this should work for all devices, just like the device cgroup
if it's going to be an adjunct to it.

> > > Overall plan is to restrict views of '/dev', '/sys/dev/block' and
> > > '/sys/block' to only present the devices 'visible' for this
> > > namespace.
> > 
> > We actually already have a devices cgroup that does some of this:
> > 
> > https://www.kernel.org/doc/Documentation/cgroup-v1/devices.txt
> > 
> I know. But this essentially is a filter on '/dev' only, and needs to
> be configured. Which makes it very unwieldy to use.
> And the contents of sysfs are not modified, so there's a mismatch 
> between contents in /dev and /sys.
> Which might cause issues with monitoring tools.

Firstly, since it does part of what you want, we at least need to
understand why you think it can't be enhanced to do everything.

The /sys problem has been discussed many times.  GregKH really doesn't
like the idea of filtering /sys (and most container people agree), so
the options available seem to be don't mount /sys in a container or
emulate it via fuse.  If you pick either does the device cgroup now
work for you?

> > However, visibility isn't the only problem, for direct passthrough
> > there's also uevent handling and people have even asked about
> > module loading.
> > 
> I am aware, and that's another reason why device cgroup doesn't cut
> it.

Christian Brauner is looking at this ... apparently Ubuntu has some
thumb drive inside lxc container use case that needs it.

> > >   Initially the drivers would keep their global enumeration, but
> > > plan is to make the drivers namespace-aware, too, such that each
> > > namespace could have its own driver-specific device enumeration.
> > 
> > I really wouldn't do this.  Namespace/Cgroup separation should be
> > kept as high as possible.  If it leaks into the drivers it will
> > become unmaintainable.  Why do you think you need the drivers to be
> > aware?  If it's just enumeration, that should all be doable with
> > the visibility driver unless you want to do things like compact
> > numbering?
> > 
> Which is precisely why I mentioned device modifications.
> On a generic level we can influence the visibility of devices in 
> relation to namespaces, we cannot influence the devices themselves.
> This will lead to namespaces seeing disjunct device numbers (ie 8:0
> and 8:8 on ns 1, 8:4 on ns 2). Not that I think that will be an
> issue, but  certainly a change in behaviour.

Well, not necessarily, the pid namespace is an example of a remapping
namespace.  The same thing could be done for device numbering, but
there really needs to be a compelling case.  Given our use of hotplug,
why would any tool assume compact numbering?  And if you don't need
compact numbering there's no need to bother with remapping.

> > > Goal of this topic is to get a consensus on whether block
> > > namespaces are a feature which would find interest, and also to
> > > discuss some design details here:
> > > - Only in certain cases can a namespace be assigned (eg by
> > > calling
> > > 'modprobe', starting iscsiadm, or calling nvme-cli); how do we
> > > handle
> > > devices for which no namespace can be identified?
> > > - Shall we allow for different device enumeration per namespace?
> > > - Into which level should we go with hiding sysfs structures?
> > >    Is blanking out the higher-level interfaces in /dev and
> > > /sys/block    enough?
> > 
> > First question is does the device cgroup do enough for you and if
> > not what's missing?
> > 
> See above. sysfs modifications and uevent filtering are missing.
> This infrastructure for that is already in place thanks to network 
> namespaces, we 'just' need to make use of it.
> Additional drawback is the manual configuration of device-cgroup.

OK, so still, why not fix or enhance the device cgroup?  The current
proposal seems to want to duplicate it as a namespace.

James




From hare at suse.de  Thu Jun 10 08:05:07 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 10 Jun 2021 17:05:07 +0200
Subject: [LSF/MM/BPF TOPIC] block namespaces
In-Reply-To: <f31fbb6d2f374a39d22e3fca122d757e905d2711.camel@HansenPartnership.com>
References: <a189ec50-4c11-9ee9-0b9e-b492507adc1e@suse.de>
 <485837f392401bf35fb7fc8231d7a051f47b53d7.camel@HansenPartnership.com>
 <539b35c6-34f7-62e0-4d93-6d27145bb78f@suse.de>
 <f31fbb6d2f374a39d22e3fca122d757e905d2711.camel@HansenPartnership.com>
Message-ID: <33f720ce-484b-0c42-b14a-0a7df3e09e81@suse.de>

On 6/10/21 4:29 PM, James Bottomley wrote:
> On Thu, 2021-06-10 at 07:49 +0200, Hannes Reinecke wrote:
>> On 6/9/21 8:36 PM, James Bottomley wrote:
>>> On Thu, 2021-05-27 at 10:01 +0200, Hannes Reinecke wrote:
>>>> Hi all,
>>>>
>>>> I guess it's time to tick off yet another item on my long-term
>>>> to-do list:
>>>>
>>>> Block namespaces
>>>> ----------------
>>>>
>>>> Idea is similar to what network already does: allowing each user
>>>> namespace to have a different 'view' on the existing block
>>>> devices.  EG if the admin creates a ramdisk in one namespace this
>>>> device should not be visible to other namespaces.  But for me the
>>>> most important use-case would be qemu; currently the devices need
>>>> to be set up in the host, even though the host has no business
>>>> touching it as they really belong to the qemu instance.  This
>>>> is causing quite some irritation eg when this device has LVM or
>>>> MD metadata and udev is trying to activate it on the host.
>>>
>>> I suppose the first question is "why block only?"  There are
>>> several existing device namespace proposals which would be more
>>> generic.
>>>
>>
>> Well; I'm more of a storage person, and do know the needs and 
>> shortcomings in that area. Less well so in other areas...
> 
> OK, but this should work for all devices, just like the device cgroup
> if it's going to be an adjunct to it.
> 
>>>> Overall plan is to restrict views of '/dev', '/sys/dev/block' and
>>>> '/sys/block' to only present the devices 'visible' for this
>>>> namespace.
>>>
>>> We actually already have a devices cgroup that does some of this:
>>>
>>> https://www.kernel.org/doc/Documentation/cgroup-v1/devices.txt
>>>
>> I know. But this essentially is a filter on '/dev' only, and needs to
>> be configured. Which makes it very unwieldy to use.
>> And the contents of sysfs are not modified, so there's a mismatch 
>> between contents in /dev and /sys.
>> Which might cause issues with monitoring tools.
> 
> Firstly, since it does part of what you want, we at least need to
> understand why you think it can't be enhanced to do everything.
> 
> The /sys problem has been discussed many times.  GregKH really doesn't
> like the idea of filtering /sys (and most container people agree), so
> the options available seem to be don't mount /sys in a container or
> emulate it via fuse.  If you pick either does the device cgroup now
> work for you?
> 

... begging the question why he relented for network namespaces.
They _do_ modify sysfs, and that via generic hooks (ie making struct
class namespace-aware).
And as this is a public interface I don't see why I can't be using it...

>>> However, visibility isn't the only problem, for direct passthrough
>>> there's also uevent handling and people have even asked about
>>> module loading.
>>>
>> I am aware, and that's another reason why device cgroup doesn't cut
>> it.
> 
> Christian Brauner is looking at this ... apparently Ubuntu has some
> thumb drive inside lxc container use case that needs it.
> 

Guess with whom I'm discussing on how to implement this.

>>>>   Initially the drivers would keep their global enumeration, but
>>>> plan is to make the drivers namespace-aware, too, such that each
>>>> namespace could have its own driver-specific device enumeration.
>>>
>>> I really wouldn't do this.  Namespace/Cgroup separation should be
>>> kept as high as possible.  If it leaks into the drivers it will
>>> become unmaintainable.  Why do you think you need the drivers to be
>>> aware?  If it's just enumeration, that should all be doable with
>>> the visibility driver unless you want to do things like compact
>>> numbering?
>>>
>> Which is precisely why I mentioned device modifications.
>> On a generic level we can influence the visibility of devices in 
>> relation to namespaces, we cannot influence the devices themselves.
>> This will lead to namespaces seeing disjunct device numbers (ie 8:0
>> and 8:8 on ns 1, 8:4 on ns 2). Not that I think that will be an
>> issue, but  certainly a change in behaviour.
> 
> Well, not necessarily, the pid namespace is an example of a remapping
> namespace.  The same thing could be done for device numbering, but
> there really needs to be a compelling case.  Given our use of hotplug,
> why would any tool assume compact numbering?  And if you don't need
> compact numbering there's no need to bother with remapping.
> 

Which is my assumption, too. I have just mentioned it here for
completeness sake. It's not that I'll be implementing it for now.

>>>> Goal of this topic is to get a consensus on whether block
>>>> namespaces are a feature which would find interest, and also to
>>>> discuss some design details here:
>>>> - Only in certain cases can a namespace be assigned (eg by
>>>> calling
>>>> 'modprobe', starting iscsiadm, or calling nvme-cli); how do we
>>>> handle
>>>> devices for which no namespace can be identified?
>>>> - Shall we allow for different device enumeration per namespace?
>>>> - Into which level should we go with hiding sysfs structures?
>>>>    Is blanking out the higher-level interfaces in /dev and
>>>> /sys/block    enough?
>>>
>>> First question is does the device cgroup do enough for you and if
>>> not what's missing?
>>>
>> See above. sysfs modifications and uevent filtering are missing.
>> This infrastructure for that is already in place thanks to network 
>> namespaces, we 'just' need to make use of it.
>> Additional drawback is the manual configuration of device-cgroup.
> 
> OK, so still, why not fix or enhance the device cgroup?  The current
> proposal seems to want to duplicate it as a namespace.
> 
Primarily because device-group is really limited in its functionality,
and the interface into the device core is, quite frankly, horrible.
(Implementing it under security/device_cgroup.c, and having direct
function calls from fs/block_dev.c and fs/namei.c?
hch will yet at me if I attempted that.)

Re-implementing that as a namespace is the cleaner solution.
Actually, it should be possible to re-implementing device-cgroup on to
of the block namespaces; for that we'll have to expand it to cover all
devices, but you're arguing for that anyway.
So, let's see.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		        Kernel Storage Architect
hare at suse.de			               +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 N?rnberg
GF: F. Imend?rffer, HRB 36809 (AG N?rnberg)


From sagi at grimberg.me  Thu Jun 10 13:03:03 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 10 Jun 2021 13:03:03 -0700
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at
 0000000000000230
In-Reply-To: <CO1PR19MB48859E1CCD596402D4007458EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
References: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <CO1PR19MB4885314DCBFEB376C7305C98EE3D9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <e40414a9-41ef-f82c-7d1c-2583d8251c26@grimberg.me>
 <CO1PR19MB48850B44ED23C179A4E4F541EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
 <b9c7e6cc-fea5-b8ce-af07-7d004116e3e3@grimberg.me>
 <CO1PR19MB48859C3254E4F3EA3282EDC9EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
 <a76ecf8e-15d0-1ebc-aaf2-c93fee591602@grimberg.me>
 <CO1PR19MB48859E1CCD596402D4007458EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
Message-ID: <172e3128-d1dc-ca9a-c679-752b621e956f@grimberg.me>


> Correct, free_queue is being called (sock->sk becomes NULL) before restore_sock_calls
> 
> When restore_sock_calls is called, we fail on 'write_lock_bh(&sock->sk->sk_callback_lock)'
> 
> NULL pointer dereference at 0x230 ? 560 decimal
> crash> struct sock -o
> struct sock {
>     [0] struct sock_common __sk_common;
>     ?
>     ...
>     ?
>     [560] rwlock_t sk_callback_lock;
> 
> stop queue in ctx2 does not really do anything since 'NVME_TCP_Q_LIVE' bit is already cleared (by ctx1).
> can you please explain how stop the queue before free helps to serialize ctx1 ?

What I understood from your description is:
1. ctx1 calls stop_queue - calls kernel_sock_shutdown
2. ctx1 gets to restore_sock_calls (just before)
3. ctx2 is triggered from state_change - scheduling err_work
4. ctx2 does stop_queues
5. ctx2 calls destroy_queues -> there does sock_release
6. ctx1 does frwd progress and access an already freed sk

Hence with the mutex protection, ctx2 will be serialized on step (4)
until ctx2 releases the mutex and hence cannot get to step (5) but
only after ctx1 releases the mutex, in step (6).

But maybe I'm not interpreting this correctly?


From Chaitanya.Kulkarni at wdc.com  Thu Jun 10 13:32:56 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Thu, 10 Jun 2021 20:32:56 +0000
Subject: [PATCH] Revert "nvme: verify MNAN value if ANA is enabled"
References: <20210610024539.28875-1-chaitanya.kulkarni@wdc.com>
 <BYAPR04MB49655A9B821FEE29D7A89E3986359@BYAPR04MB4965.namprd04.prod.outlook.com>
 <20210610074546.jyqjalpldii6reg6@beryllium.lan>
Message-ID: <BYAPR04MB4965781915C1D4AFF2830E9586359@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/10/21 00:45, Daniel Wagner wrote:
> Hi Chaitanya,
>
> On Thu, Jun 10, 2021 at 02:55:08AM +0000, Chaitanya Kulkarni wrote:
>> On 6/9/21 19:45, Chaitanya Kulkarni wrote:
>>> This reverts commit 8872c159c7a83daf633768cee7a7ef7154010341. This is
>>> needed to move forward with the blktests for now, without this patch
>>> all the testcases result in the error :-
>>>
>>> [ 3502.072798] nvme nvme1: Invalid MNAN value 1024
>> Thinking about the code again I think following should work :-
>>
>> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
>> index 23573fe3fc7d..4277f1554bd5 100644
>> --- a/drivers/nvme/host/multipath.c
>> +++ b/drivers/nvme/host/multipath.c
>> @@ -813,7 +813,7 @@ int nvme_mpath_init_identify(struct nvme_ctrl *ctrl,
>> struct nvme_id_ctrl *id)
>>             !(ctrl->subsys->cmic & NVME_CTRL_CMIC_ANA))
>>                 return 0;
>>  
>> -       if (!ctrl->max_namespaces ||
>> +       if (ctrl->max_namespaces &&
>>             ctrl->max_namespaces > le32_to_cpu(id->nn)) {
>>                 dev_err(ctrl->device,
>>                         "Invalid MNAN value %u\n", ctrl->max_namespaces);
> '!ctrl->max_namespace' could also be written as
> 'ctrl->max_namespace != 0' which makes it more obvious what the intend
> is here:
>
>   If the controller supports Asymmetric Namespace Access Reporting, then
>   this field shall be set to a non-zero value that is less than or equal
>   to the NN value.
>
>

Let us look into the host side issue first then we can move to target side.

Consider a scenario where ANA enabled subsys with 0 namespaces on the target
side. When host issues connect command to such a controller ctrl->mnan
should be 0 and ctrl->nn should be 0 which should be valid.

With original check in the code :-

if (!ctrl->max_namespaces ||
    ctrl->max_namespaces > le32_to_cpu(id->nn))

!ctrl->max_namespaces will return in 1 and due to || condition will be
true and code will report the error. The change proposed in this patch
with above mentioned scenario :- if (ctrl->max_namespaces &&
ctrl->max_namespaces > le32_to_cpu(id->nn)) { ctrl->max_namespaces will
return 0 and due to && condition will be false and new code will not
report zero. So I think above suggested patch is needed irrespective of
the target fix.



From Chaitanya.Kulkarni at wdc.com  Thu Jun 10 14:01:14 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Thu, 10 Jun 2021 21:01:14 +0000
Subject: [PATCH] Revert "nvme: verify MNAN value if ANA is enabled"
References: <20210610024539.28875-1-chaitanya.kulkarni@wdc.com>
 <BYAPR04MB49655A9B821FEE29D7A89E3986359@BYAPR04MB4965.namprd04.prod.outlook.com>
 <20210610074546.jyqjalpldii6reg6@beryllium.lan>
Message-ID: <BYAPR04MB4965AE59EF50A714B12DF42886359@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/10/21 00:45, Daniel Wagner wrote:
>> diff --git a/drivers/nvme/target/admin-cmd.c
>> b/drivers/nvme/target/admin-cmd.c
>> index cd60a8184d04..a8ec377bb68d 100644
>> --- a/drivers/nvme/target/admin-cmd.c
>> +++ b/drivers/nvme/target/admin-cmd.c
>> @@ -394,7 +394,7 @@ static void nvmet_execute_identify_ctrl(struct
>> nvmet_req *req)
>>         id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
>>  
>>         id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
>> -       id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
>> +       id->mnan = cpu_to_le32(ctrl->subsys->max_nsid);
>>         id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
>>                         NVME_CTRL_ONCS_WRITE_ZEROES);
> This looks like the right fix for the upper limit problem.
> I can see the tests are still failing with
>
>   nvme nvme0: Invalid MNAN value 0
>
> which indicates there is another problem in the nvmet core.
>
> I try to find it.x
>
> Thanks,
> Daniel
>


With following two patches blktests test-cases are passing for me.

root at vm nvme (nvme-5.14) # git log -2
commit cd5d6d55a405f97ad5c31a738976eea0e16ebe20 (HEAD -> nvme-5.14)
Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Date:   Wed Jun 9 20:07:56 2021 -0700

    nvmet: set the mnan value to nn
   
    From spec :-
    "MNAN : If the controller supports Asymmetric Namespace Access
Reporting,
    then this field shall be set to a non-zero value that is less than or
    equal tothe NN value".
   
    In nvmet_execute_identify_ctrl() when building the identify controller
    data structure set the mnan value to nn to follow the spec.
   
    Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

commit 1368a1a5e7566d726bf74234d05895c3f0d54690
Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Date:   Wed Jun 9 20:07:00 2021 -0700

    nvme: fix the comparison in the mnan check
   
    The existing check for the valid mnan value will result in the error
    when ctrl->max_namespaces are set to the 1024 from NVMeOF target since
    !1024 == 0 so it will lead to next comparison 1024 > is->nn which will
    be always true untill target has 1024 namespaces.
   
    This patch fixes the comparison.
   
    Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
root at vm nvme (nvme-5.14) # cat
0001-nvme-fix-the-comparison-in-the-mnan-check.patch
>From f1f0947ca495b7d9b8721411e407ebf9efad0df9 Mon Sep 17 00:00:00 2001
From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Date: Wed, 9 Jun 2021 20:07:00 -0700
Subject: [PATCH 1/2] nvme: fix the comparison in the mnan check

The existing check for the valid mnan value will result in the error
when ctrl->max_namespaces are set to the 1024 from NVMeOF target since
!1024 == 0 so it will lead to next comparison 1024 > is->nn which will
be always true untill target has 1024 namespaces.

This patch fixes the comparison.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/multipath.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 23573fe3fc7d..4277f1554bd5 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -813,7 +813,7 @@ int nvme_mpath_init_identify(struct nvme_ctrl *ctrl,
struct nvme_id_ctrl *id)
         !(ctrl->subsys->cmic & NVME_CTRL_CMIC_ANA))
         return 0;
 
-    if (!ctrl->max_namespaces ||
+    if (ctrl->max_namespaces &&
         ctrl->max_namespaces > le32_to_cpu(id->nn)) {
         dev_err(ctrl->device,
             "Invalid MNAN value %u\n", ctrl->max_namespaces);
-- 
2.22.1

root at vm nvme (nvme-5.14) # cat 0002-nvmet-set-the-mnan-value-to-nn.patch
>From d18c2c851686c4e5fa8ddb457e6726cafb86b85f Mon Sep 17 00:00:00 2001
From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Date: Wed, 9 Jun 2021 20:07:56 -0700
Subject: [PATCH 2/2] nvmet: set the mnan value to nn

>From spec :-
"MNAN : If the controller supports Asymmetric Namespace Access Reporting,
then this field shall be set to a non-zero value that is less than or
equal tothe NN value".

In nvmet_execute_identify_ctrl() when building the identify controller
data structure set the mnan value to nn to follow the spec.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/admin-cmd.c
b/drivers/nvme/target/admin-cmd.c
index cd60a8184d04..88e85d9a5be8 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -394,7 +394,11 @@ static void nvmet_execute_identify_ctrl(struct
nvmet_req *req)
     id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
 
     id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
-    id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
+    if (IS_ENABLED(CONFIG_NVME_MULTIPATH))
+        id->mnan = cpu_to_le32(ctrl->subsys->max_nsid);
+    else
+        id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
+
     id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
             NVME_CTRL_ONCS_WRITE_ZEROES);
 
-- 
2.22.1

root at vm nvme (nvme-5.14) # ./compile_nvme.sh
+ umount /mnt/nvme0n1
+ clear_dmesg
umount: /mnt/nvme0n1: not mounted
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ git apply ./all-fixes.diff
+ sleep 1
++ nproc
+ make -j 64 M=drivers/nvme/ modules
  CC [M]  drivers/nvme//target/loop.o
  LD [M]  drivers/nvme//target/nvme-loop.o
  MODPOST drivers/nvme//Module.symvers
  LD [M]  drivers/nvme//target/nvme-loop.ko
+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/host/
++ uname -r
+ TARGET_DEST=/lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko
drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko
drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko
/lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko
drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko
drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko
/lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/host/
/lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/target//
/lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/host/:
total 9.1M
-rw-r--r--. 1 root root 4.0M Jun 10 13:50 nvme-core.ko
-rw-r--r--. 1 root root 639K Jun 10 13:50 nvme-fabrics.ko
-rw-r--r--. 1 root root 1.2M Jun 10 13:50 nvme-fc.ko
-rw-r--r--. 1 root root 1.1M Jun 10 13:50 nvme.ko
-rw-r--r--. 1 root root 1.2M Jun 10 13:50 nvme-rdma.ko
-rw-r--r--. 1 root root 1.1M Jun 10 13:50 nvme-tcp.ko

/lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/target//:
total 8.3M
-rw-r--r--. 1 root root 718K Jun 10 13:50 nvme-fcloop.ko
-rw-r--r--. 1 root root 623K Jun 10 13:50 nvme-loop.ko
-rw-r--r--. 1 root root 1.1M Jun 10 13:50 nvmet-fc.ko
-rw-r--r--. 1 root root 4.0M Jun 10 13:50 nvmet.ko
-rw-r--r--. 1 root root 1.1M Jun 10 13:50 nvmet-rdma.ko
-rw-r--r--. 1 root root 868K Jun 10 13:50 nvmet-tcp.ko
+ modprobe nvme
+ git co drivers/nvme/target/loop.c
Updated 1 path from the index
root at vm nvme (nvme-5.14) # cdblktests
root at vm blktests (master) # ./check nvme
nvme/002 (create many subsystems and test discovery)         [passed]
    runtime  40.408s  ...  40.579s
nvme/003 (test if we're sending keep-alives to a discovery controller)
[passed]
    runtime  10.179s  ...  10.180s
nvme/004 (test nvme and nvmet UUID NS descriptors)           [passed]
    runtime  1.786s  ...  1.794s
nvme/005 (reset local loopback target)                       [passed]
    runtime  2.241s  ...  2.233s
nvme/006 (create an NVMeOF target with a block device-backed ns) [passed]
    runtime  0.147s  ...  0.152s
nvme/007 (create an NVMeOF target with a file-backed ns)     [passed]
    runtime  0.097s  ...  0.102s
nvme/008 (create an NVMeOF host with a block device-backed ns) [passed]
    runtime  1.793s  ...  1.792s
nvme/009 (create an NVMeOF host with a file-backed ns)       [passed]
    runtime  1.749s  ...  1.751s
nvme/010 (run data verification fio job on NVMeOF block device-backed
ns) [passed]
    runtime  38.113s  ...  31.199s
nvme/011 (run data verification fio job on NVMeOF file-backed ns) [passed]
    runtime  296.910s  ...  302.248s
nvme/012 (run mkfs and data verification fio job on NVMeOF block
device-backed ns) [passed]
    runtime  8.823s  ...  9.221s
nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed
ns) [passed]
    runtime  23.517s  ...  24.288s
nvme/014 (flush a NVMeOF block device-backed ns)             [passed]
    runtime  22.787s  ...  20.386s
nvme/015 (unit test for NVMe flush for file backed ns)       [passed]
    runtime  20.599s  ...  23.212s
nvme/016 (create/delete many NVMeOF block device-backed ns and test
discovery) [passed]
    runtime  21.616s  ...  21.016s
nvme/017 (create/delete many file-ns and test discovery)     [passed]
    runtime    ...  20.721s
nvme/018 (unit test NVMe-oF out of range access on a file backend) [passed]
    runtime  1.777s  ...  1.752s
nvme/019 (test NVMe DSM Discard command on NVMeOF block-device ns) [passed]
    runtime  1.795s  ...  1.805s
nvme/020 (test NVMe DSM Discard command on NVMeOF file-backed ns) [passed]
    runtime  1.756s  ...  1.746s
nvme/021 (test NVMe list command on NVMeOF file-backed ns)   [passed]
    runtime  1.734s  ...  1.745s
nvme/022 (test NVMe reset command on NVMeOF file-backed ns)  [passed]
    runtime  2.168s  ...  2.181s
nvme/023 (test NVMe smart-log command on NVMeOF block-device ns) [passed]
    runtime  1.782s  ...  1.776s
nvme/024 (test NVMe smart-log command on NVMeOF file-backed ns) [passed]
    runtime  1.730s  ...  1.743s
nvme/025 (test NVMe effects-log command on NVMeOF file-backed ns) [passed]
    runtime  1.719s  ...  1.767s
nvme/026 (test NVMe ns-descs command on NVMeOF file-backed ns) [passed]
    runtime  1.718s  ...  1.732s
nvme/027 (test NVMe ns-rescan command on NVMeOF file-backed ns) [passed]
    runtime  1.754s  ...  1.748s
nvme/028 (test NVMe list-subsys command on NVMeOF file-backed ns) [passed]
    runtime  1.753s  ...  1.776s
nvme/029 (test userspace IO via nvme-cli read/write interface) [passed]
    runtime  2.280s  ...  2.244s
nvme/030 (ensure the discovery generation counter is updated
appropriately) [passed]
    runtime  0.425s  ...  0.401s
nvme/031 (test deletion of NVMeOF controllers immediately after setup)
[passed]
    runtime  6.134s  ...  5.996s
nvme/038 (test deletion of NVMeOF subsystem without enabling) [passed]
    runtime  0.075s  ...  0.059s
root at vm blktests (master) #




From kbusch at kernel.org  Thu Jun 10 14:44:33 2021
From: kbusch at kernel.org (Keith Busch)
Date: Thu, 10 Jun 2021 14:44:33 -0700
Subject: [PATCHv4 0/4] block and nvme passthrough error handling
Message-ID: <20210610214437.641245-1-kbusch@kernel.org>

This series has blk_execute_rq() return queueing errors so the caller
may know if their request wasn't dispatched, and adds polled hctx
support.

Chances since v3:

  Added recieved "Reviewed-by:" tags.

  Retain the REQ_HIPRI flag for nvme polled passthrough requests
  (patch 2)

  Combined nvme request dispatch with the status decoding into single
  function (patch 4)

Keith Busch (4):
  block: support polling through blk_execute_rq
  nvme: use blk_execute_rq() for passthrough commands
  block: return errors from blk_execute_rq()
  nvme: use return value from blk_execute_rq()

 block/blk-exec.c               | 25 +++++++++++--
 drivers/nvme/host/core.c       | 65 +++++++++++++++-------------------
 drivers/nvme/host/fabrics.c    | 13 ++++---
 drivers/nvme/host/fabrics.h    |  2 +-
 drivers/nvme/host/fc.c         |  2 +-
 drivers/nvme/host/ioctl.c      |  6 +---
 drivers/nvme/host/nvme.h       |  4 +--
 drivers/nvme/host/rdma.c       |  3 +-
 drivers/nvme/host/tcp.c        |  2 +-
 drivers/nvme/target/loop.c     |  2 +-
 drivers/nvme/target/passthru.c |  8 ++---
 include/linux/blkdev.h         |  4 ++-
 12 files changed, 72 insertions(+), 64 deletions(-)

-- 
2.25.4



From kbusch at kernel.org  Thu Jun 10 14:44:34 2021
From: kbusch at kernel.org (Keith Busch)
Date: Thu, 10 Jun 2021 14:44:34 -0700
Subject: [PATCHv4 1/4] block: support polling through blk_execute_rq
In-Reply-To: <20210610214437.641245-1-kbusch@kernel.org>
References: <20210610214437.641245-1-kbusch@kernel.org>
Message-ID: <20210610214437.641245-2-kbusch@kernel.org>

Poll for completions if the request's hctx is a polling type.

Reviewed-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Ming Lei <ming.lei at redhat.com>
Signed-off-by: Keith Busch <kbusch at kernel.org>
---
 block/blk-exec.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/block/blk-exec.c b/block/blk-exec.c
index beae70a0e5e5..38f88552aa31 100644
--- a/block/blk-exec.c
+++ b/block/blk-exec.c
@@ -63,6 +63,19 @@ void blk_execute_rq_nowait(struct gendisk *bd_disk, struct request *rq,
 }
 EXPORT_SYMBOL_GPL(blk_execute_rq_nowait);
 
+static bool blk_rq_is_poll(struct request *rq)
+{
+	return rq->mq_hctx && rq->mq_hctx->type == HCTX_TYPE_POLL;
+}
+
+static void blk_rq_poll_completion(struct request *rq, struct completion *wait)
+{
+	do {
+		blk_poll(rq->q, request_to_qc_t(rq->mq_hctx, rq), true);
+		cond_resched();
+	} while (!completion_done(wait));
+}
+
 /**
  * blk_execute_rq - insert a request into queue for execution
  * @bd_disk:	matching gendisk
@@ -83,7 +96,10 @@ void blk_execute_rq(struct gendisk *bd_disk, struct request *rq, int at_head)
 
 	/* Prevent hang_check timer from firing at us during very long I/O */
 	hang_check = sysctl_hung_task_timeout_secs;
-	if (hang_check)
+
+	if (blk_rq_is_poll(rq))
+		blk_rq_poll_completion(rq, &wait);
+	else if (hang_check)
 		while (!wait_for_completion_io_timeout(&wait, hang_check * (HZ/2)));
 	else
 		wait_for_completion_io(&wait);
-- 
2.25.4



From kbusch at kernel.org  Thu Jun 10 14:44:35 2021
From: kbusch at kernel.org (Keith Busch)
Date: Thu, 10 Jun 2021 14:44:35 -0700
Subject: [PATCHv4 2/4] nvme: use blk_execute_rq() for passthrough commands
In-Reply-To: <20210610214437.641245-1-kbusch@kernel.org>
References: <20210610214437.641245-1-kbusch@kernel.org>
Message-ID: <20210610214437.641245-3-kbusch@kernel.org>

The generic blk_execute_rq() knows how to handle polled completions. Use
that instead of implementing an nvme specific handler.

Signed-off-by: Keith Busch <kbusch at kernel.org>
---
v2->v3:

  Retain the REQ_HIPRI flag for nvme polled passthrough requests

 drivers/nvme/host/core.c    | 40 +++++++------------------------------
 drivers/nvme/host/fabrics.c | 13 ++++++------
 drivers/nvme/host/fabrics.h |  2 +-
 drivers/nvme/host/fc.c      |  2 +-
 drivers/nvme/host/nvme.h    |  2 +-
 drivers/nvme/host/rdma.c    |  3 +--
 drivers/nvme/host/tcp.c     |  2 +-
 drivers/nvme/target/loop.c  |  2 +-
 8 files changed, 19 insertions(+), 47 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 784679307c03..2b9deaffa4e1 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -631,6 +631,8 @@ static inline void nvme_init_request(struct request *req,
 	cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 
 	req->cmd_flags |= REQ_FAILFAST_DRIVER;
+	if (req->mq_hctx->type == HCTX_TYPE_POLL)
+		req->cmd_flags |= REQ_HIPRI;
 	nvme_clear_nvme_request(req);
 	memcpy(nvme_req(req)->cmd, cmd, sizeof(*cmd));
 }
@@ -1032,31 +1034,6 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req)
 }
 EXPORT_SYMBOL_GPL(nvme_setup_cmd);
 
-static void nvme_end_sync_rq(struct request *rq, blk_status_t error)
-{
-	struct completion *waiting = rq->end_io_data;
-
-	rq->end_io_data = NULL;
-	complete(waiting);
-}
-
-static void nvme_execute_rq_polled(struct request_queue *q,
-		struct gendisk *bd_disk, struct request *rq, int at_head)
-{
-	DECLARE_COMPLETION_ONSTACK(wait);
-
-	WARN_ON_ONCE(!test_bit(QUEUE_FLAG_POLL, &q->queue_flags));
-
-	rq->cmd_flags |= REQ_HIPRI;
-	rq->end_io_data = &wait;
-	blk_execute_rq_nowait(bd_disk, rq, at_head, nvme_end_sync_rq);
-
-	while (!completion_done(&wait)) {
-		blk_poll(q, request_to_qc_t(rq->mq_hctx, rq), true);
-		cond_resched();
-	}
-}
-
 /*
  * Returns 0 on success.  If the result is negative, it's a Linux error code;
  * if the result is positive, it's an NVM Express status code
@@ -1064,7 +1041,7 @@ static void nvme_execute_rq_polled(struct request_queue *q,
 int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 		union nvme_result *result, void *buffer, unsigned bufflen,
 		unsigned timeout, int qid, int at_head,
-		blk_mq_req_flags_t flags, bool poll)
+		blk_mq_req_flags_t flags)
 {
 	struct request *req;
 	int ret;
@@ -1085,10 +1062,7 @@ int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 			goto out;
 	}
 
-	if (poll)
-		nvme_execute_rq_polled(req->q, NULL, req, at_head);
-	else
-		blk_execute_rq(NULL, req, at_head);
+	blk_execute_rq(NULL, req, at_head);
 	if (result)
 		*result = nvme_req(req)->result;
 	if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
@@ -1105,7 +1079,7 @@ int nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 		void *buffer, unsigned bufflen)
 {
 	return __nvme_submit_sync_cmd(q, cmd, NULL, buffer, bufflen, 0,
-			NVME_QID_ANY, 0, 0, false);
+			NVME_QID_ANY, 0, 0);
 }
 EXPORT_SYMBOL_GPL(nvme_submit_sync_cmd);
 
@@ -1469,7 +1443,7 @@ static int nvme_features(struct nvme_ctrl *dev, u8 op, unsigned int fid,
 	c.features.dword11 = cpu_to_le32(dword11);
 
 	ret = __nvme_submit_sync_cmd(dev->admin_q, &c, &res,
-			buffer, buflen, 0, NVME_QID_ANY, 0, 0, false);
+			buffer, buflen, 0, NVME_QID_ANY, 0, 0);
 	if (ret >= 0 && result)
 		*result = le32_to_cpu(res.u32);
 	return ret;
@@ -2054,7 +2028,7 @@ int nvme_sec_submit(void *data, u16 spsp, u8 secp, void *buffer, size_t len,
 	cmd.common.cdw11 = cpu_to_le32(len);
 
 	return __nvme_submit_sync_cmd(ctrl->admin_q, &cmd, NULL, buffer, len, 0,
-			NVME_QID_ANY, 1, 0, false);
+			NVME_QID_ANY, 1, 0);
 }
 EXPORT_SYMBOL_GPL(nvme_sec_submit);
 #endif /* CONFIG_BLK_SED_OPAL */
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 1239a63e3ac2..7f189f7da617 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -154,7 +154,7 @@ int nvmf_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val)
 	cmd.prop_get.offset = cpu_to_le32(off);
 
 	ret = __nvme_submit_sync_cmd(ctrl->fabrics_q, &cmd, &res, NULL, 0, 0,
-			NVME_QID_ANY, 0, 0, false);
+			NVME_QID_ANY, 0, 0);
 
 	if (ret >= 0)
 		*val = le64_to_cpu(res.u64);
@@ -201,7 +201,7 @@ int nvmf_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val)
 	cmd.prop_get.offset = cpu_to_le32(off);
 
 	ret = __nvme_submit_sync_cmd(ctrl->fabrics_q, &cmd, &res, NULL, 0, 0,
-			NVME_QID_ANY, 0, 0, false);
+			NVME_QID_ANY, 0, 0);
 
 	if (ret >= 0)
 		*val = le64_to_cpu(res.u64);
@@ -247,7 +247,7 @@ int nvmf_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val)
 	cmd.prop_set.value = cpu_to_le64(val);
 
 	ret = __nvme_submit_sync_cmd(ctrl->fabrics_q, &cmd, NULL, NULL, 0, 0,
-			NVME_QID_ANY, 0, 0, false);
+			NVME_QID_ANY, 0, 0);
 	if (unlikely(ret))
 		dev_err(ctrl->device,
 			"Property Set error: %d, offset %#x\n",
@@ -394,7 +394,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 
 	ret = __nvme_submit_sync_cmd(ctrl->fabrics_q, &cmd, &res,
 			data, sizeof(*data), 0, NVME_QID_ANY, 1,
-			BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT, false);
+			BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT);
 	if (ret) {
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
@@ -418,7 +418,6 @@ EXPORT_SYMBOL_GPL(nvmf_connect_admin_queue);
  * @qid:	NVMe I/O queue number for the new I/O connection between
  *		host and target (note qid == 0 is illegal as this is
  *		the Admin queue, per NVMe standard).
- * @poll:	Whether or not to poll for the completion of the connect cmd.
  *
  * This function issues a fabrics-protocol connection
  * of a NVMe I/O queue (via NVMe Fabrics "Connect" command)
@@ -430,7 +429,7 @@ EXPORT_SYMBOL_GPL(nvmf_connect_admin_queue);
  *	> 0: NVMe error status code
  *	< 0: Linux errno error code
  */
-int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid, bool poll)
+int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 {
 	struct nvme_command cmd;
 	struct nvmf_connect_data *data;
@@ -457,7 +456,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid, bool poll)
 
 	ret = __nvme_submit_sync_cmd(ctrl->connect_q, &cmd, &res,
 			data, sizeof(*data), 0, qid, 1,
-			BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT, poll);
+			BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT);
 	if (ret) {
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index c31dad69a773..a146cb903869 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -182,7 +182,7 @@ int nvmf_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val);
 int nvmf_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val);
 int nvmf_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val);
 int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl);
-int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid, bool poll);
+int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid);
 int nvmf_register_transport(struct nvmf_transport_ops *ops);
 void nvmf_unregister_transport(struct nvmf_transport_ops *ops);
 void nvmf_free_options(struct nvmf_ctrl_options *opts);
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 256e87721a01..e3d7b636b961 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2346,7 +2346,7 @@ nvme_fc_connect_io_queues(struct nvme_fc_ctrl *ctrl, u16 qsize)
 					(qsize / 5));
 		if (ret)
 			break;
-		ret = nvmf_connect_io_queue(&ctrl->ctrl, i, false);
+		ret = nvmf_connect_io_queue(&ctrl->ctrl, i);
 		if (ret)
 			break;
 
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1f397ecba16c..499681bc0286 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -658,7 +658,7 @@ int nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 		union nvme_result *result, void *buffer, unsigned bufflen,
 		unsigned timeout, int qid, int at_head,
-		blk_mq_req_flags_t flags, bool poll);
+		blk_mq_req_flags_t flags);
 int nvme_set_features(struct nvme_ctrl *dev, unsigned int fid,
 		      unsigned int dword11, void *buffer, size_t buflen,
 		      u32 *result);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 74bf2c7f2b80..855a33e0d1b1 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -680,11 +680,10 @@ static void nvme_rdma_stop_io_queues(struct nvme_rdma_ctrl *ctrl)
 static int nvme_rdma_start_queue(struct nvme_rdma_ctrl *ctrl, int idx)
 {
 	struct nvme_rdma_queue *queue = &ctrl->queues[idx];
-	bool poll = nvme_rdma_poll_queue(queue);
 	int ret;
 
 	if (idx)
-		ret = nvmf_connect_io_queue(&ctrl->ctrl, idx, poll);
+		ret = nvmf_connect_io_queue(&ctrl->ctrl, idx);
 	else
 		ret = nvmf_connect_admin_queue(&ctrl->ctrl);
 
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 5fc6c568c626..01056722a5e4 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1574,7 +1574,7 @@ static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx)
 	int ret;
 
 	if (idx)
-		ret = nvmf_connect_io_queue(nctrl, idx, false);
+		ret = nvmf_connect_io_queue(nctrl, idx);
 	else
 		ret = nvmf_connect_admin_queue(nctrl);
 
diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index cb30cb942e1d..2a7bf1019952 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -335,7 +335,7 @@ static int nvme_loop_connect_io_queues(struct nvme_loop_ctrl *ctrl)
 	int i, ret;
 
 	for (i = 1; i < ctrl->ctrl.queue_count; i++) {
-		ret = nvmf_connect_io_queue(&ctrl->ctrl, i, false);
+		ret = nvmf_connect_io_queue(&ctrl->ctrl, i);
 		if (ret)
 			return ret;
 		set_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[i].flags);
-- 
2.25.4



From kbusch at kernel.org  Thu Jun 10 14:44:36 2021
From: kbusch at kernel.org (Keith Busch)
Date: Thu, 10 Jun 2021 14:44:36 -0700
Subject: [PATCHv4 3/4] block: return errors from blk_execute_rq()
In-Reply-To: <20210610214437.641245-1-kbusch@kernel.org>
References: <20210610214437.641245-1-kbusch@kernel.org>
Message-ID: <20210610214437.641245-4-kbusch@kernel.org>

The synchronous blk_execute_rq() had not provided a way for its callers
to know if its request was successful or not. Return the blk_status_t
result of the request.

Reviewed-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Ming Lei <ming.lei at redhat.com>
Signed-off-by: Keith Busch <kbusch at kernel.org>
---
 block/blk-exec.c       | 7 +++++--
 include/linux/blkdev.h | 4 +++-
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/block/blk-exec.c b/block/blk-exec.c
index 38f88552aa31..d6cd501c0d34 100644
--- a/block/blk-exec.c
+++ b/block/blk-exec.c
@@ -21,7 +21,7 @@ static void blk_end_sync_rq(struct request *rq, blk_status_t error)
 {
 	struct completion *waiting = rq->end_io_data;
 
-	rq->end_io_data = NULL;
+	rq->end_io_data = (void *)(uintptr_t)error;
 
 	/*
 	 * complete last, if this is a stack request the process (and thus
@@ -85,8 +85,9 @@ static void blk_rq_poll_completion(struct request *rq, struct completion *wait)
  * Description:
  *    Insert a fully prepared request at the back of the I/O scheduler queue
  *    for execution and wait for completion.
+ * Return: The blk_status_t result provided to blk_mq_end_request().
  */
-void blk_execute_rq(struct gendisk *bd_disk, struct request *rq, int at_head)
+blk_status_t blk_execute_rq(struct gendisk *bd_disk, struct request *rq, int at_head)
 {
 	DECLARE_COMPLETION_ONSTACK(wait);
 	unsigned long hang_check;
@@ -103,5 +104,7 @@ void blk_execute_rq(struct gendisk *bd_disk, struct request *rq, int at_head)
 		while (!wait_for_completion_io_timeout(&wait, hang_check * (HZ/2)));
 	else
 		wait_for_completion_io(&wait);
+
+	return (blk_status_t)(uintptr_t)rq->end_io_data;
 }
 EXPORT_SYMBOL(blk_execute_rq);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index d66d0da72529..1261e1fe88bd 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -936,10 +936,12 @@ extern int blk_rq_map_kern(struct request_queue *, struct request *, void *, uns
 extern int blk_rq_map_user_iov(struct request_queue *, struct request *,
 			       struct rq_map_data *, const struct iov_iter *,
 			       gfp_t);
-extern void blk_execute_rq(struct gendisk *, struct request *, int);
 extern void blk_execute_rq_nowait(struct gendisk *,
 				  struct request *, int, rq_end_io_fn *);
 
+blk_status_t blk_execute_rq(struct gendisk *bd_disk, struct request *rq,
+			    int at_head);
+
 /* Helper to convert REQ_OP_XXX to its string format XXX */
 extern const char *blk_op_str(unsigned int op);
 
-- 
2.25.4



From kbusch at kernel.org  Thu Jun 10 14:44:37 2021
From: kbusch at kernel.org (Keith Busch)
Date: Thu, 10 Jun 2021 14:44:37 -0700
Subject: [PATCHv4 4/4] nvme: use return value from blk_execute_rq()
In-Reply-To: <20210610214437.641245-1-kbusch@kernel.org>
References: <20210610214437.641245-1-kbusch@kernel.org>
Message-ID: <20210610214437.641245-5-kbusch@kernel.org>

We don't have an nvme status to report if the driver's .queue_rq()
returns an error without dispatching the requested nvme command. Check
the return value from blk_execute_rq() for all passthrough commands so
the caller may know their command was not successful.

If the command is from the target passthrough interface and fails to
dispatch, synthesize the response back to the host as a internal target
error.

Signed-off-by: Keith Busch <kbusch at kernel.org>
---
v2->v3:

  Combined request dispatch with the status decoding into single
  function

 drivers/nvme/host/core.c       | 35 ++++++++++++++++++++++++++--------
 drivers/nvme/host/ioctl.c      |  6 +-----
 drivers/nvme/host/nvme.h       |  2 +-
 drivers/nvme/target/passthru.c |  8 ++++----
 4 files changed, 33 insertions(+), 18 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 2b9deaffa4e1..f4a690808010 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -609,6 +609,7 @@ EXPORT_SYMBOL_NS_GPL(nvme_put_ns, NVME_TARGET_PASSTHRU);
 
 static inline void nvme_clear_nvme_request(struct request *req)
 {
+	nvme_req(req)->status = 0;
 	nvme_req(req)->retries = 0;
 	nvme_req(req)->flags = 0;
 	req->rq_flags |= RQF_DONTPREP;
@@ -1034,6 +1035,25 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req)
 }
 EXPORT_SYMBOL_GPL(nvme_setup_cmd);
 
+/*
+ * Return values:
+ * 0:  success
+ * >0: nvme controller's cqe status response
+ * <0: kernel error in lieu of controller response
+ */
+static int nvme_execute_rq(struct gendisk *disk, struct request *rq,
+		bool at_head)
+{
+	blk_status_t status;
+
+	status = blk_execute_rq(disk, rq, at_head);
+	if (nvme_req(rq)->flags & NVME_REQ_CANCELLED)
+		return -EINTR;
+	if (nvme_req(rq)->status)
+		return nvme_req(rq)->status;
+	return blk_status_to_errno(status);
+}
+
 /*
  * Returns 0 on success.  If the result is negative, it's a Linux error code;
  * if the result is positive, it's an NVM Express status code
@@ -1062,13 +1082,9 @@ int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 			goto out;
 	}
 
-	blk_execute_rq(NULL, req, at_head);
-	if (result)
+	ret = nvme_execute_rq(NULL, req, at_head);
+	if (result && ret >= 0)
 		*result = nvme_req(req)->result;
-	if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
-		ret = -EINTR;
-	else
-		ret = nvme_req(req)->status;
  out:
 	blk_mq_free_request(req);
 	return ret;
@@ -1156,18 +1172,21 @@ static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
 	}
 }
 
-void nvme_execute_passthru_rq(struct request *rq)
+int nvme_execute_passthru_rq(struct request *rq)
 {
 	struct nvme_command *cmd = nvme_req(rq)->cmd;
 	struct nvme_ctrl *ctrl = nvme_req(rq)->ctrl;
 	struct nvme_ns *ns = rq->q->queuedata;
 	struct gendisk *disk = ns ? ns->disk : NULL;
 	u32 effects;
+	int  ret;
 
 	effects = nvme_passthru_start(ctrl, ns, cmd->common.opcode);
-	blk_execute_rq(disk, rq, 0);
+	ret = nvme_execute_rq(disk, rq, false);
 	if (effects) /* nothing to be done for zero cmd effects */
 		nvme_passthru_end(ctrl, effects);
+
+	return ret;
 }
 EXPORT_SYMBOL_NS_GPL(nvme_execute_passthru_rq, NVME_TARGET_PASSTHRU);
 
diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c
index 2e7780ea0354..85cfad3dd692 100644
--- a/drivers/nvme/host/ioctl.c
+++ b/drivers/nvme/host/ioctl.c
@@ -93,11 +93,7 @@ static int nvme_submit_user_cmd(struct request_queue *q,
 		}
 	}
 
-	nvme_execute_passthru_rq(req);
-	if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
-		ret = -EINTR;
-	else
-		ret = nvme_req(req)->status;
+	ret = nvme_execute_passthru_rq(req);
 	if (result)
 		*result = le64_to_cpu(nvme_req(req)->result.u64);
 	if (meta && !ret && !write) {
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 499681bc0286..7557f3404615 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -871,7 +871,7 @@ static inline void nvme_hwmon_exit(struct nvme_ctrl *ctrl)
 
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
-void nvme_execute_passthru_rq(struct request *rq);
+int nvme_execute_passthru_rq(struct request *rq);
 struct nvme_ctrl *nvme_ctrl_from_file(struct file *file);
 struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid);
 void nvme_put_ns(struct nvme_ns *ns);
diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
index 39b1473f7204..ba28ce42cb55 100644
--- a/drivers/nvme/target/passthru.c
+++ b/drivers/nvme/target/passthru.c
@@ -153,11 +153,10 @@ static void nvmet_passthru_execute_cmd_work(struct work_struct *w)
 {
 	struct nvmet_req *req = container_of(w, struct nvmet_req, p.work);
 	struct request *rq = req->p.rq;
-	u16 status;
+	int status;
 
-	nvme_execute_passthru_rq(rq);
+	status = nvme_execute_passthru_rq(rq);
 
-	status = nvme_req(rq)->status;
 	if (status == NVME_SC_SUCCESS &&
 	    req->cmd->common.opcode == nvme_admin_identify) {
 		switch (req->cmd->identify.cns) {
@@ -168,7 +167,8 @@ static void nvmet_passthru_execute_cmd_work(struct work_struct *w)
 			nvmet_passthru_override_id_ns(req);
 			break;
 		}
-	}
+	} else if (status < 0)
+		status = NVME_SC_INTERNAL;
 
 	req->cqe->result = nvme_req(rq)->result;
 	nvmet_req_complete(req, status);
-- 
2.25.4



From Chaitanya.Kulkarni at wdc.com  Thu Jun 10 18:55:35 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Fri, 11 Jun 2021 01:55:35 +0000
Subject: [PATCHv4 2/4] nvme: use blk_execute_rq() for passthrough commands
References: <20210610214437.641245-1-kbusch@kernel.org>
 <20210610214437.641245-3-kbusch@kernel.org>
Message-ID: <BYAPR04MB4965C9553CF9E9303850B76D86349@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/10/21 14:45, Keith Busch wrote:
> The generic blk_execute_rq() knows how to handle polled completions. Use
> that instead of implementing an nvme specific handler.
>
> Signed-off-by: Keith Busch <kbusch at kernel.org>

Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>




From Chaitanya.Kulkarni at wdc.com  Thu Jun 10 18:57:07 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Fri, 11 Jun 2021 01:57:07 +0000
Subject: [PATCHv4 3/4] block: return errors from blk_execute_rq()
References: <20210610214437.641245-1-kbusch@kernel.org>
 <20210610214437.641245-4-kbusch@kernel.org>
Message-ID: <BYAPR04MB49656B80D8290D909B29074186349@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/10/21 14:45, Keith Busch wrote:
> The synchronous blk_execute_rq() had not provided a way for its callers
> to know if its request was successful or not. Return the blk_status_t
> result of the request.
>
> Reviewed-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Ming Lei <ming.lei at redhat.com>
> Signed-off-by: Keith Busch <kbusch at kernel.org>

Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>




From Chaitanya.Kulkarni at wdc.com  Thu Jun 10 19:00:36 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Fri, 11 Jun 2021 02:00:36 +0000
Subject: [PATCHv4 4/4] nvme: use return value from blk_execute_rq()
References: <20210610214437.641245-1-kbusch@kernel.org>
 <20210610214437.641245-5-kbusch@kernel.org>
Message-ID: <BYAPR04MB4965A4FB6B655ABFAAF72CAB86349@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/10/21 14:45, Keith Busch wrote:
> We don't have an nvme status to report if the driver's .queue_rq()
> returns an error without dispatching the requested nvme command. Check
> the return value from blk_execute_rq() for all passthrough commands so
> the caller may know their command was not successful.
>
> If the command is from the target passthrough interface and fails to
> dispatch, synthesize the response back to the host as a internal target
> error.
>
> Signed-off-by: Keith Busch <kbusch at kernel.org>

Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>




From wubo40 at huawei.com  Thu Jun 10 19:14:08 2021
From: wubo40 at huawei.com (Wu Bo)
Date: Fri, 11 Jun 2021 10:14:08 +0800
Subject: [PATCH V15 4/5] nvmet: add Command Set Identifier support
In-Reply-To: <20210610013252.53874-5-chaitanya.kulkarni@wdc.com>
References: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
 <20210610013252.53874-5-chaitanya.kulkarni@wdc.com>
Message-ID: <923a0c0e-2f5f-e2f7-aa3b-b4feddd754b2@huawei.com>

On 2021/6/10 9:32, Chaitanya Kulkarni wrote:
> NVMe TP 4056 allows controllers to support different command sets.
> NVMeoF target currently only supports namespaces that contain
> traditional logical blocks that may be randomly read and written. In
> some applications there is a value in exposing namespaces that contain
> logical blocks that have special access rules (e.g. sequentially write
> required namespace such as Zoned Namespace (ZNS)).
> 
> In order to support the Zoned Block Devices (ZBD) backend, controllers
> need to have support for ZNS Command Set Identifier (CSI).
> 
> In this preparation patch, we adjust the code such that it can now
> support the default command set identifier. We update the namespace data
> structure to store the CSI value which defaults to NVME_CSI_NVM
> that represents traditional logical blocks namespace type.
> 
> The CSI support is required to implement the ZBD backend for NVMeOF
> with host side NVMe ZNS interface, since ZNS commands belong to
> the different command set than the default one.
> 
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> ---
>   drivers/nvme/target/admin-cmd.c | 75 +++++++++++++++++++++++++++------
>   drivers/nvme/target/core.c      | 28 +++++++++---
>   drivers/nvme/target/nvmet.h     |  1 +
>   include/linux/nvme.h            |  1 +
>   4 files changed, 87 insertions(+), 18 deletions(-)
> 
> diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
> index 3de6a6c99b01..93aaa7479e71 100644
> --- a/drivers/nvme/target/admin-cmd.c
> +++ b/drivers/nvme/target/admin-cmd.c
> @@ -162,15 +162,8 @@ static void nvmet_execute_get_log_page_smart(struct nvmet_req *req)
>   	nvmet_req_complete(req, status);
>   }
>   
> -static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
> +static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log)
>   {
> -	u16 status = NVME_SC_INTERNAL;
> -	struct nvme_effects_log *log;
> -
> -	log = kzalloc(sizeof(*log), GFP_KERNEL);
> -	if (!log)
> -		goto out;
> -
>   	log->acs[nvme_admin_get_log_page]	= cpu_to_le32(1 << 0);
>   	log->acs[nvme_admin_identify]		= cpu_to_le32(1 << 0);
>   	log->acs[nvme_admin_abort_cmd]		= cpu_to_le32(1 << 0);
> @@ -184,9 +177,30 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
>   	log->iocs[nvme_cmd_flush]		= cpu_to_le32(1 << 0);
>   	log->iocs[nvme_cmd_dsm]			= cpu_to_le32(1 << 0);
>   	log->iocs[nvme_cmd_write_zeroes]	= cpu_to_le32(1 << 0);
> +}
>   
> -	status = nvmet_copy_to_sgl(req, 0, log, sizeof(*log));
> +static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
> +{
> +	struct nvme_effects_log *log;
> +	u16 status = NVME_SC_SUCCESS;
> +
> +	log = kzalloc(sizeof(*log), GFP_KERNEL);
> +	if (!log) {
> +		status = NVME_SC_INTERNAL;
> +		goto out;
> +	}
> +
> +	switch (req->cmd->get_log_page.csi) {
> +	case NVME_CSI_NVM:
> +		nvmet_get_cmd_effects_nvm(log);
> +		break;
> +	default:
> +		status = NVME_SC_INVALID_LOG_PAGE;
> +		goto free;
> +	}
>   
> +	status = nvmet_copy_to_sgl(req, 0, log, sizeof(*log));
> +free:
>   	kfree(log);
>   out:
>   	nvmet_req_complete(req, status);
> @@ -613,6 +627,12 @@ static void nvmet_execute_identify_desclist(struct nvmet_req *req)
>   			goto out;
>   	}
>   
> +	status = nvmet_copy_ns_identifier(req, NVME_NIDT_CSI,
> +					  NVME_NIDT_CSI_LEN,
> +					  &req->ns->csi, &off);
> +	if (status)
> +		goto out;
> +
>   	if (sg_zero_buffer(req->sg, req->sg_cnt, NVME_IDENTIFY_DATA_SIZE - off,
>   			off) != NVME_IDENTIFY_DATA_SIZE - off)
>   		status = NVME_SC_INTERNAL | NVME_SC_DNR;
> @@ -621,6 +641,17 @@ static void nvmet_execute_identify_desclist(struct nvmet_req *req)
>   	nvmet_req_complete(req, status);
>   }
>   
> +static bool nvmet_handle_identify_desclist(struct nvmet_req *req)
> +{
> +	switch (req->cmd->identify.csi) {
> +	case NVME_CSI_NVM:
> +		nvmet_execute_identify_desclist(req);
> +		return true;
> +	default:
> +		return false;
> +	}
> +}
> +
>   static void nvmet_execute_identify(struct nvmet_req *req)
>   {
>   	if (!nvmet_check_transfer_len(req, NVME_IDENTIFY_DATA_SIZE))
> @@ -628,13 +659,31 @@ static void nvmet_execute_identify(struct nvmet_req *req)
>   
>   	switch (req->cmd->identify.cns) {
>   	case NVME_ID_CNS_NS:
> -		return nvmet_execute_identify_ns(req);
> +		switch (req->cmd->identify.csi) {
> +		case NVME_CSI_NVM:
> +			return nvmet_execute_identify_ns(req);
> +		default:
> +			break;
> +		}
> +		break;
>   	case NVME_ID_CNS_CTRL:
> -		return nvmet_execute_identify_ctrl(req);
> +		switch (req->cmd->identify.csi) {
> +		case NVME_CSI_NVM:
> +			return nvmet_execute_identify_ctrl(req);
> +		}

Is it missing to add the default branch here ?

> +		break;
>   	case NVME_ID_CNS_NS_ACTIVE_LIST:
> -		return nvmet_execute_identify_nslist(req);
> +		switch (req->cmd->identify.csi) {
> +		case NVME_CSI_NVM:
> +			return nvmet_execute_identify_nslist(req);
> +		default:
> +			break;
> +		}
> +		break;
>   	case NVME_ID_CNS_NS_DESC_LIST:
> -		return nvmet_execute_identify_desclist(req);
> +		if (nvmet_handle_identify_desclist(req) == true)
> +			return;
> +		break;
>   	}
>   
>   	nvmet_req_cns_error_complete(req);
> diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
> index 146909486b8f..ac53a1307ce4 100644
> --- a/drivers/nvme/target/core.c
> +++ b/drivers/nvme/target/core.c
> @@ -692,6 +692,7 @@ struct nvmet_ns *nvmet_ns_alloc(struct nvmet_subsys *subsys, u32 nsid)
>   
>   	uuid_gen(&ns->uuid);
>   	ns->buffered_io = false;
> +	ns->csi = NVME_CSI_NVM;
>   
>   	return ns;
>   }
> @@ -887,10 +888,14 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
>   		return ret;
>   	}
>   
> -	if (req->ns->file)
> -		return nvmet_file_parse_io_cmd(req);
> -
> -	return nvmet_bdev_parse_io_cmd(req);
> +	switch (req->ns->csi) {
> +	case NVME_CSI_NVM:
> +		if (req->ns->file)
> +			return nvmet_file_parse_io_cmd(req);
> +		return nvmet_bdev_parse_io_cmd(req);
> +	default:
> +		return NVME_SC_INVALID_IO_CMD_SET;
> +	}
>   }
>   
>   bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
> @@ -1112,6 +1117,17 @@ static inline u8 nvmet_cc_iocqes(u32 cc)
>   	return (cc >> NVME_CC_IOCQES_SHIFT) & 0xf;
>   }
>   
> +static inline bool nvmet_css_supported(u8 cc_css)
> +{
> +	switch (cc_css <<= NVME_CC_CSS_SHIFT) {
> +	case NVME_CC_CSS_NVM:
> +	case NVME_CC_CSS_CSI:
> +		return true;
> +	default:
> +		return false;
> +	}
> +}
> +
>   static void nvmet_start_ctrl(struct nvmet_ctrl *ctrl)
>   {
>   	lockdep_assert_held(&ctrl->lock);
> @@ -1131,7 +1147,7 @@ static void nvmet_start_ctrl(struct nvmet_ctrl *ctrl)
>   
>   	if (nvmet_cc_mps(ctrl->cc) != 0 ||
>   	    nvmet_cc_ams(ctrl->cc) != 0 ||
> -	    nvmet_cc_css(ctrl->cc) != 0) {
> +	    !nvmet_css_supported(nvmet_cc_css(ctrl->cc))) {
>   		ctrl->csts = NVME_CSTS_CFS;
>   		return;
>   	}
> @@ -1182,6 +1198,8 @@ static void nvmet_init_cap(struct nvmet_ctrl *ctrl)
>   {
>   	/* command sets supported: NVMe command set: */
>   	ctrl->cap = (1ULL << 37);
> +	/* Controller supports one or more I/O Command Sets */
> +	ctrl->cap |= (1ULL << 43);
>   	/* CC.EN timeout in 500msec units: */
>   	ctrl->cap |= (15ULL << 24);
>   	/* maximum queue entries supported: */
> diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
> index d4bcfeb570e5..4ca558b94955 100644
> --- a/drivers/nvme/target/nvmet.h
> +++ b/drivers/nvme/target/nvmet.h
> @@ -83,6 +83,7 @@ struct nvmet_ns {
>   	struct pci_dev		*p2p_dev;
>   	int			pi_type;
>   	int			metadata_size;
> +	u8			csi;
>   };
>   
>   static inline struct nvmet_ns *to_nvmet_ns(struct config_item *item)
> diff --git a/include/linux/nvme.h b/include/linux/nvme.h
> index edcbd60b88b9..c7ba83144d52 100644
> --- a/include/linux/nvme.h
> +++ b/include/linux/nvme.h
> @@ -1504,6 +1504,7 @@ enum {
>   	NVME_SC_NS_WRITE_PROTECTED	= 0x20,
>   	NVME_SC_CMD_INTERRUPTED		= 0x21,
>   	NVME_SC_TRANSIENT_TR_ERR	= 0x22,
> +	NVME_SC_INVALID_IO_CMD_SET	= 0x2C,
>   
>   	NVME_SC_LBA_RANGE		= 0x80,
>   	NVME_SC_CAP_EXCEEDED		= 0x81,
> 



From Chaitanya.Kulkarni at wdc.com  Thu Jun 10 19:35:16 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Fri, 11 Jun 2021 02:35:16 +0000
Subject: [PATCH V15 4/5] nvmet: add Command Set Identifier support
References: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
 <20210610013252.53874-5-chaitanya.kulkarni@wdc.com>
 <923a0c0e-2f5f-e2f7-aa3b-b4feddd754b2@huawei.com>
Message-ID: <BYAPR04MB4965DAB9078B76AC9CACC8CA86349@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/10/21 19:14, Wu Bo wrote:
>>   	case NVME_ID_CNS_CTRL:
>> -		return nvmet_execute_identify_ctrl(req);
>> +		switch (req->cmd->identify.csi) {
>> +		case NVME_CSI_NVM:
>> +			return nvmet_execute_identify_ctrl(req);
>> +		}
> Is it missing to add the default branch here ?
>

maybe for the consistency




From Chaitanya.Kulkarni at wdc.com  Thu Jun 10 23:03:47 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Fri, 11 Jun 2021 06:03:47 +0000
Subject: [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload
References: <BYAPR04MB49652C4B75E38F3716F3C06386539@BYAPR04MB4965.namprd04.prod.outlook.com>
Message-ID: <BYAPR04MB49650D374549472F6A47FE7786349@BYAPR04MB4965.namprd04.prod.outlook.com>

On 5/10/21 17:15, Chaitanya Kulkarni wrote:
> Hi,
>
> * Background :-
> -----------------------------------------------------------------------
>
> Copy offload is a feature that allows file-systems or storage devices
> to be instructed to copy files/logical blocks without requiring
> involvement of the local CPU.
>
> With reference to the RISC-V summit keynote [1] single threaded
> performance is limiting due to Denard scaling and multi-threaded
> performance is slowing down due Moore's law limitations. With the rise
> of SNIA Computation Technical Storage Working Group (TWG) [2],
> offloading computations to the device or over the fabrics is becoming
> popular as there are several solutions available [2]. One of the common
> operation which is popular in the kernel and is not merged yet is Copy
> offload over the fabrics or on to the device.
>
> * Problem :-
> -----------------------------------------------------------------------
>
> The original work which is done by Martin is present here [3]. The
> latest work which is posted by Mikulas [4] is not merged yet. These two
> approaches are totally different from each other. Several storage
> vendors discourage mixing copy offload requests with regular READ/WRITE
> I/O. Also, the fact that the operation fails if a copy request ever
> needs to be split as it traverses the stack it has the unfortunate
> side-effect of preventing copy offload from working in pretty much
> every common deployment configuration out there.
>
> * Current state of the work :-
> -----------------------------------------------------------------------
>
> With [3] being hard to handle arbitrary DM/MD stacking without
> splitting the command in two, one for copying IN and one for copying
> OUT. Which is then demonstrated by the [4] why [3] it is not a suitable
> candidate. Also, with [4] there is an unresolved problem with the
> two-command approach about how to handle changes to the DM layout
> between an IN and OUT operations.
>
> * Why Linux Kernel Storage System needs Copy Offload support now ?
> -----------------------------------------------------------------------
>
> With the rise of the SNIA Computational Storage TWG and solutions [2],
> existing SCSI XCopy support in the protocol, recent advancement in the
> Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer
> DMA support in the Linux Kernel mainly for NVMe devices [7] and
> eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit
> from Copy offload operation.
>
> With this background we have significant number of use-cases which are
> strong candidates waiting for outstanding Linux Kernel Block Layer Copy
> Offload support, so that Linux Kernel Storage subsystem can to address
> previously mentioned problems [1] and allow efficient offloading of the
> data related operations. (Such as move/copy etc.)
>
> For reference following is the list of the use-cases/candidates waiting
> for Copy Offload support :-
>
> 1. SCSI-attached storage arrays.
> 2. Stacking drivers supporting XCopy DM/MD.
> 3. Computational Storage solutions.
> 7. File systems :- Local, NFS and Zonefs.
> 4. Block devices :- Distributed, local, and Zoned devices.
> 5. Peer to Peer DMA support solutions.
> 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF.
>
> * What we will discuss in the proposed session ?
> -----------------------------------------------------------------------
>
> I'd like to propose a session to go over this topic to understand :-
>
> 1. What are the blockers for Copy Offload implementation ?
> 2. Discussion about having a file system interface.
> 3. Discussion about having right system call for user-space.
> 4. What is the right way to move this work forward ?
> 5. How can we help to contribute and move this work forward ?
>
> * Required Participants :-
> -----------------------------------------------------------------------
>
> I'd like to invite file system, block layer, and device drivers
> developers to:-
>
> 1. Share their opinion on the topic.
> 2. Share their experience and any other issues with [4].
> 3. Uncover additional details that are missing from this proposal.
>
> Required attendees :-
>
> Martin K. Petersen
> Jens Axboe
> Christoph Hellwig
> Bart Van Assche
> Zach Brown
> Roland Dreier
> Ric Wheeler
> Trond Myklebust
> Mike Snitzer
> Keith Busch
> Sagi Grimberg
> Hannes Reinecke
> Frederick Knight
> Mikulas Patocka
> Keith Busch
>
> Regards,
> Chaitanya
>
> [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf
> [2] https://www.snia.org/computational
> https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/
>       https://www.eideticom.com/products.html
> https://www.xilinx.com/applications/data-center/computational-storage.html
> [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy
> [4] https://www.spinics.net/lists/linux-block/msg00599.html
> [5] https://lwn.net/Articles/793585/
> [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned-
> namespaces-zns-as-go-to-industry-technology/
> [7] https://github.com/sbates130272/linux-p2pmem
> [8] https://kernel.dk/io_uring.pdf
>
>

Mail server is dropping emails from the mailing list, adding personal
email address.




From dwagner at suse.de  Fri Jun 11 02:08:19 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Fri, 11 Jun 2021 11:08:19 +0200
Subject: [PATCH] Revert "nvme: verify MNAN value if ANA is enabled"
In-Reply-To: <BYAPR04MB4965781915C1D4AFF2830E9586359@BYAPR04MB4965.namprd04.prod.outlook.com>
References: <20210610024539.28875-1-chaitanya.kulkarni@wdc.com>
 <BYAPR04MB49655A9B821FEE29D7A89E3986359@BYAPR04MB4965.namprd04.prod.outlook.com>
 <20210610074546.jyqjalpldii6reg6@beryllium.lan>
 <BYAPR04MB4965781915C1D4AFF2830E9586359@BYAPR04MB4965.namprd04.prod.outlook.com>
Message-ID: <20210611090819.d2ths7e4swfuj2s6@beryllium.lan>

Hi Chaitanay,

On Thu, Jun 10, 2021 at 08:32:56PM +0000, Chaitanya Kulkarni wrote:
> Let us look into the host side issue first then we can move to target side.

Sure

> Consider a scenario where ANA enabled subsys with 0 namespaces on the target
> side. When host issues connect command to such a controller ctrl->mnan
> should be 0 and ctrl->nn should be 0 which should be valid.
> 
> With original check in the code :-
> 
> if (!ctrl->max_namespaces ||
>     ctrl->max_namespaces > le32_to_cpu(id->nn))

In the scenario you describe this is obvious wrong. I suspect we can't
decide if there are namespaces on the target side enabled or not at this
point. Maybe later in the discovery path we could check max_namespace
again. But I don't know if this is the trouble worth.

So yes I agree with your proposed change. In the end it's not important
anyway.

Thanks,
Daniel


From dwagner at suse.de  Fri Jun 11 02:17:42 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Fri, 11 Jun 2021 11:17:42 +0200
Subject: [PATCH] Revert "nvme: verify MNAN value if ANA is enabled"
In-Reply-To: <BYAPR04MB4965AE59EF50A714B12DF42886359@BYAPR04MB4965.namprd04.prod.outlook.com>
References: <20210610024539.28875-1-chaitanya.kulkarni@wdc.com>
 <BYAPR04MB49655A9B821FEE29D7A89E3986359@BYAPR04MB4965.namprd04.prod.outlook.com>
 <20210610074546.jyqjalpldii6reg6@beryllium.lan>
 <BYAPR04MB4965AE59EF50A714B12DF42886359@BYAPR04MB4965.namprd04.prod.outlook.com>
Message-ID: <20210611091742.6iefvwlxjlm6oiae@beryllium.lan>

On Thu, Jun 10, 2021 at 09:01:14PM +0000, Chaitanya Kulkarni wrote:
> commit 1368a1a5e7566d726bf74234d05895c3f0d54690
> Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> Date:   Wed Jun 9 20:07:00 2021 -0700
> 
>     nvme: fix the comparison in the mnan check
>    
>     The existing check for the valid mnan value will result in the error
>     when ctrl->max_namespaces are set to the 1024 from NVMeOF target since
>     !1024 == 0 so it will lead to next comparison 1024 > is->nn which will
>     be always true untill target has 1024 namespaces.

The commit message doesn't make sense to me. NSID is not limited to
1024.

>From the discussion in the other mail, I though the argument is, if
there are no namespaces on the target the MNAN is allowed to be
zero. The original check assumed there are always namespaces. The
proposed fix drops the first half of the specs statement:

  ... then this field shall be set to a non-zero value that is less than
  or equal to the NN value.



From ntsironis at arrikto.com  Fri Jun 11 08:35:30 2021
From: ntsironis at arrikto.com (Nikos Tsironis)
Date: Fri, 11 Jun 2021 18:35:30 +0300
Subject: [LSF/MM/BFP ATTEND] [LSF/MM/BFP TOPIC] Storage: Copy Offload
In-Reply-To: <BYAPR04MB49652C4B75E38F3716F3C06386539@BYAPR04MB4965.namprd04.prod.outlook.com>
References: <BYAPR04MB49652C4B75E38F3716F3C06386539@BYAPR04MB4965.namprd04.prod.outlook.com>
Message-ID: <9b14e6c3-8880-4694-1b7e-c82cacd8c9c0@arrikto.com>

On 5/11/21 3:15 AM, Chaitanya Kulkarni wrote:
> Hi,
> 
> * Background :-
> -----------------------------------------------------------------------
> 
> Copy offload is a feature that allows file-systems or storage devices
> to be instructed to copy files/logical blocks without requiring
> involvement of the local CPU.
> 
> With reference to the RISC-V summit keynote [1] single threaded
> performance is limiting due to Denard scaling and multi-threaded
> performance is slowing down due Moore's law limitations. With the rise
> of SNIA Computation Technical Storage Working Group (TWG) [2],
> offloading computations to the device or over the fabrics is becoming
> popular as there are several solutions available [2]. One of the common
> operation which is popular in the kernel and is not merged yet is Copy
> offload over the fabrics or on to the device.
> 
> * Problem :-
> -----------------------------------------------------------------------
> 
> The original work which is done by Martin is present here [3]. The
> latest work which is posted by Mikulas [4] is not merged yet. These two
> approaches are totally different from each other. Several storage
> vendors discourage mixing copy offload requests with regular READ/WRITE
> I/O. Also, the fact that the operation fails if a copy request ever
> needs to be split as it traverses the stack it has the unfortunate
> side-effect of preventing copy offload from working in pretty much
> every common deployment configuration out there.
> 
> * Current state of the work :-
> -----------------------------------------------------------------------
> 
> With [3] being hard to handle arbitrary DM/MD stacking without
> splitting the command in two, one for copying IN and one for copying
> OUT. Which is then demonstrated by the [4] why [3] it is not a suitable
> candidate. Also, with [4] there is an unresolved problem with the
> two-command approach about how to handle changes to the DM layout
> between an IN and OUT operations.
> 
> * Why Linux Kernel Storage System needs Copy Offload support now ?
> -----------------------------------------------------------------------
> 
> With the rise of the SNIA Computational Storage TWG and solutions [2],
> existing SCSI XCopy support in the protocol, recent advancement in the
> Linux Kernel File System for Zoned devices (Zonefs [5]), Peer to Peer
> DMA support in the Linux Kernel mainly for NVMe devices [7] and
> eventually NVMe Devices and subsystem (NVMe PCIe/NVMeOF) will benefit
> from Copy offload operation.
> 
> With this background we have significant number of use-cases which are
> strong candidates waiting for outstanding Linux Kernel Block Layer Copy
> Offload support, so that Linux Kernel Storage subsystem can to address
> previously mentioned problems [1] and allow efficient offloading of the
> data related operations. (Such as move/copy etc.)
> 
> For reference following is the list of the use-cases/candidates waiting
> for Copy Offload support :-
> 
> 1. SCSI-attached storage arrays.
> 2. Stacking drivers supporting XCopy DM/MD.
> 3. Computational Storage solutions.
> 7. File systems :- Local, NFS and Zonefs.
> 4. Block devices :- Distributed, local, and Zoned devices.
> 5. Peer to Peer DMA support solutions.
> 6. Potentially NVMe subsystem both NVMe PCIe and NVMeOF.
> 
> * What we will discuss in the proposed session ?
> -----------------------------------------------------------------------
> 
> I'd like to propose a session to go over this topic to understand :-
> 
> 1. What are the blockers for Copy Offload implementation ?
> 2. Discussion about having a file system interface.
> 3. Discussion about having right system call for user-space.
> 4. What is the right way to move this work forward ?
> 5. How can we help to contribute and move this work forward ?
> 
> * Required Participants :-
> -----------------------------------------------------------------------
> 
> I'd like to invite file system, block layer, and device drivers
> developers to:-
> 
> 1. Share their opinion on the topic.
> 2. Share their experience and any other issues with [4].
> 3. Uncover additional details that are missing from this proposal.
> 
> Required attendees :-
> 
> Martin K. Petersen
> Jens Axboe
> Christoph Hellwig
> Bart Van Assche
> Zach Brown
> Roland Dreier
> Ric Wheeler
> Trond Myklebust
> Mike Snitzer
> Keith Busch
> Sagi Grimberg
> Hannes Reinecke
> Frederick Knight
> Mikulas Patocka
> Keith Busch
> 
> Regards,
> Chaitanya
> 
> [1]https://content.riscv.org/wp-content/uploads/2018/12/A-New-Golden-Age-for-Computer-Architecture-History-Challenges-and-Opportunities-David-Patterson-.pdf
> [2] https://www.snia.org/computational
> https://www.napatech.com/support/resources/solution-descriptions/napatech-smartnic-solution-for-hardware-offload/
>        https://www.eideticom.com/products.html
> https://www.xilinx.com/applications/data-center/computational-storage.html
> [3] git://git.kernel.org/pub/scm/linux/kernel/git/mkp/linux.git xcopy
> [4] https://www.spinics.net/lists/linux-block/msg00599.html
> [5] https://lwn.net/Articles/793585/
> [6] https://nvmexpress.org/new-nvmetm-specification-defines-zoned-
> namespaces-zns-as-go-to-industry-technology/
> [7] https://github.com/sbates130272/linux-p2pmem
> [8] https://kernel.dk/io_uring.pdf
> 

I would like to participate in this discussion too.

Thanks,
Nikos


From Chaitanya.Kulkarni at wdc.com  Sat Jun 12 12:18:16 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sat, 12 Jun 2021 19:18:16 +0000
Subject: [PATCH] Revert "nvme: verify MNAN value if ANA is enabled"
References: <20210610024539.28875-1-chaitanya.kulkarni@wdc.com>
 <BYAPR04MB49655A9B821FEE29D7A89E3986359@BYAPR04MB4965.namprd04.prod.outlook.com>
 <20210610074546.jyqjalpldii6reg6@beryllium.lan>
 <BYAPR04MB4965AE59EF50A714B12DF42886359@BYAPR04MB4965.namprd04.prod.outlook.com>
 <20210611091742.6iefvwlxjlm6oiae@beryllium.lan>
Message-ID: <BYAPR04MB496572D850722F58672395B886339@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/11/21 02:17, Daniel Wagner wrote:
> On Thu, Jun 10, 2021 at 09:01:14PM +0000, Chaitanya Kulkarni wrote:
>> commit 1368a1a5e7566d726bf74234d05895c3f0d54690
>> Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
>> Date:   Wed Jun 9 20:07:00 2021 -0700
>>
>>     nvme: fix the comparison in the mnan check
>>    
>>     The existing check for the valid mnan value will result in the error
>>     when ctrl->max_namespaces are set to the 1024 from NVMeOF target since
>>     !1024 == 0 so it will lead to next comparison 1024 > is->nn which will
>>     be always true untill target has 1024 namespaces.
> The commit message doesn't make sense to me. NSID is not limited to
> 1024.
>
> From the discussion in the other mail, I though the argument is, if
> there are no namespaces on the target the MNAN is allowed to be
> zero. The original check assumed there are always namespaces. The
> proposed fix drops the first half of the specs statement:
>
>   ... then this field shall be set to a non-zero value that is less than
>   or equal to the NN value.
>
>

But that will require any controller to have at least one namespace, with
current implementation we don't enforce that behavior, maybe then we should
check that in the discovery or id controller path if ANA log page is
supported ?




From Amit.Engel at Dell.com  Sun Jun 13 01:35:56 2021
From: Amit.Engel at Dell.com (Engel, Amit)
Date: Sun, 13 Jun 2021 08:35:56 +0000
Subject: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at
 0000000000000230
In-Reply-To: <172e3128-d1dc-ca9a-c679-752b621e956f@grimberg.me>
References: <CO1PR19MB48850A829FEF374C7FC156EFEE3E9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <CO1PR19MB4885314DCBFEB376C7305C98EE3D9@CO1PR19MB4885.namprd19.prod.outlook.com>
 <e40414a9-41ef-f82c-7d1c-2583d8251c26@grimberg.me>
 <CO1PR19MB48850B44ED23C179A4E4F541EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
 <b9c7e6cc-fea5-b8ce-af07-7d004116e3e3@grimberg.me>
 <CO1PR19MB48859C3254E4F3EA3282EDC9EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
 <a76ecf8e-15d0-1ebc-aaf2-c93fee591602@grimberg.me>
 <CO1PR19MB48859E1CCD596402D4007458EE369@CO1PR19MB4885.namprd19.prod.outlook.com>
 <172e3128-d1dc-ca9a-c679-752b621e956f@grimberg.me>
Message-ID: <CO1PR19MB4885C0354F0EF57BB14057A0EE329@CO1PR19MB4885.namprd19.prod.outlook.com>

Hi Sagi, after revisit the upstream code, I agree with your analysis.
We are trying now to run rhel8.3 nvme host with the missing mutex_lock commit.

Anyway, I applied a patch that uses the same mutex_lock also for start_queue failure case.

In this case, nvme_tcp_start_queue calls __nvme_tcp_stop_queue that should be protected by the same mutex_lock.

Thank you for your help
Amit

-----Original Message-----
From: Sagi Grimberg <sagi at grimberg.me> 
Sent: Thursday, June 10, 2021 11:03 PM
To: Engel, Amit; linux-nvme at lists.infradead.org
Cc: Anner, Ran; Grupi, Elad
Subject: Re: nvme_tcp BUG: unable to handle kernel NULL pointer dereference at 0000000000000230


[EXTERNAL EMAIL] 


> Correct, free_queue is being called (sock->sk becomes NULL) before 
> restore_sock_calls
> 
> When restore_sock_calls is called, we fail on 'write_lock_bh(&sock->sk->sk_callback_lock)'
> 
> NULL pointer dereference at 0x230 ? 560 decimal
> crash> struct sock -o
> struct sock {
>     [0] struct sock_common __sk_common;
>     ?
>     ...
>     ?
>     [560] rwlock_t sk_callback_lock;
> 
> stop queue in ctx2 does not really do anything since 'NVME_TCP_Q_LIVE' bit is already cleared (by ctx1).
> can you please explain how stop the queue before free helps to serialize ctx1 ?

What I understood from your description is:
1. ctx1 calls stop_queue - calls kernel_sock_shutdown 2. ctx1 gets to restore_sock_calls (just before) 3. ctx2 is triggered from state_change - scheduling err_work 4. ctx2 does stop_queues 5. ctx2 calls destroy_queues -> there does sock_release 6. ctx1 does frwd progress and access an already freed sk

Hence with the mutex protection, ctx2 will be serialized on step (4) until ctx2 releases the mutex and hence cannot get to step (5) but only after ctx1 releases the mutex, in step (6).

But maybe I'm not interpreting this correctly?

From chaitanya.kulkarni at wdc.com  Sun Jun 13 13:02:49 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 13:02:49 -0700
Subject: [PATCH 0/3] nvmet: use right MNAN value for controllers
Message-ID: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>

Hi,

In current implementation we report MNAN value as NVMET_MAX_NAMESPACES.

For Spec regarding MNAN value:-

If the controller supports Asymmetric Namespace Access Reporting, then
this field shall be set to a _non-zero_ value that is less than or equal
to the NN value.

The first patch sets identify controller MNAN value to subsys->max_nsid
that is always less than or equal to the NN.

The second patch prevents creating the controller from subsys if subsys
has 0 namespaces since it will report 0 (invalid) MNAN value.

The third patch prevents reporting such subsys in the dicovery log page.

-ck

Chaitanya Kulkarni (3):
  nvmet: use subsys->max_nsid to report MNAN
  nvmet: prevent creating ctrl with no namespace
  nvmet: don't report subsys with 0 namespaces

 drivers/nvme/target/admin-cmd.c |  2 +-
 drivers/nvme/target/core.c      |  9 +++++++++
 drivers/nvme/target/discovery.c | 13 +++++++++++++
 3 files changed, 23 insertions(+), 1 deletion(-)

-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 13:02:50 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 13:02:50 -0700
Subject: [PATCH 1/3] nvmet: use subsys->max_nsid to report MNAN
In-Reply-To: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
References: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210613200252.9167-2-chaitanya.kulkarni@wdc.com>

For Spec regarding MNAN value:-

If the controller supports Asymmetric Namespace Access Reporting, then
this field shall be set to a non-zero value that is less than or equal
to the NN value.

Instead of using NVMET_MAX_NAMESPACES use subsys->max_nsid to set the
MNAN value.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index cd60a8184d04..a8ec377bb68d 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -394,7 +394,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
 
 	id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
-	id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
+	id->mnan = cpu_to_le32(ctrl->subsys->max_nsid);
 	id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
 			NVME_CTRL_ONCS_WRITE_ZEROES);
 
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 13:02:51 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 13:02:51 -0700
Subject: [PATCH 2/3] nvmet: prevent creating ctrl with no namespace
In-Reply-To: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
References: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210613200252.9167-3-chaitanya.kulkarni@wdc.com>

>From Spec :-

If the controller supports Asymmetric Namespace Access Reporting, then
MNAN shall be set to a non-zero value that is less than or equal
to the NN value.

In current implementation if subsys has 0 namespaces then we report
wrong MNAN value for the controller if we allow controller creation.

Prevent new controller creation if the subsys associated with the new
controller has no namespaces.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/core.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 146909486b8f..26d93ac9bd2e 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1325,6 +1325,15 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
 		goto out;
 	}
 
+	mutex_lock(&subsys->lock);
+	if (!subsys->max_nsid) {
+		mutex_unlock(&subsys->lock);
+		pr_err("invalid mnan value for the ana enabled controller\n");
+		status = NVME_SC_INTERNAL;
+		goto out_put_subsystem;
+	}
+	mutex_unlock(&subsys->lock);
+
 	down_read(&nvmet_config_sem);
 	if (!nvmet_host_allowed(subsys, hostnqn)) {
 		pr_info("connect by host %s for subsystem %s not allowed\n",
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 13:02:52 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 13:02:52 -0700
Subject: [PATCH 3/3] nvmet: don't report subsys with 0 namespaces
In-Reply-To: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
References: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210613200252.9167-4-chaitanya.kulkarni@wdc.com>

Skip reporting the subsys from the discovery log page which has no
namespaces.

This is needed because we don't allow creating a controller from such
subsystem from previous patch.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/discovery.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/nvme/target/discovery.c b/drivers/nvme/target/discovery.c
index 7aa62bc6ae84..03a34ad3d59b 100644
--- a/drivers/nvme/target/discovery.c
+++ b/drivers/nvme/target/discovery.c
@@ -158,6 +158,17 @@ static size_t discovery_log_entries(struct nvmet_req *req)
 	return entries;
 }
 
+static bool nvmet_disc_get_log_report_subsys(struct nvmet_subsys *s)
+{
+	bool skip;
+
+	mutex_lock(&s->lock);
+	/* skip reporting the subsys with 0 MNAN */
+	skip = s->max_nsid ? true : false;
+	mutex_unlock(&s->lock);
+	return skip;
+}
+
 static void nvmet_execute_disc_get_log_page(struct nvmet_req *req)
 {
 	const int entry_size = sizeof(struct nvmf_disc_rsp_page_entry);
@@ -210,6 +221,8 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req)
 
 		if (!nvmet_host_allowed(p->subsys, ctrl->hostnqn))
 			continue;
+		if (!nvmet_disc_get_log_report_subsys(p->subsys))
+			continue;
 
 		nvmet_set_disc_traddr(req, req->port, traddr);
 		nvmet_format_discovery_entry(hdr, req->port,
-- 
2.22.1



From Chaitanya.Kulkarni at wdc.com  Sun Jun 13 13:06:02 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 20:06:02 +0000
Subject: [PATCH] Revert "nvme: verify MNAN value if ANA is enabled"
References: <20210610024539.28875-1-chaitanya.kulkarni@wdc.com>
 <BYAPR04MB49655A9B821FEE29D7A89E3986359@BYAPR04MB4965.namprd04.prod.outlook.com>
 <20210610074546.jyqjalpldii6reg6@beryllium.lan>
 <BYAPR04MB4965AE59EF50A714B12DF42886359@BYAPR04MB4965.namprd04.prod.outlook.com>
 <20210611091742.6iefvwlxjlm6oiae@beryllium.lan>
Message-ID: <BYAPR04MB496510FC8546E5DBE51D765F86329@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/11/21 02:17, Daniel Wagner wrote:
> On Thu, Jun 10, 2021 at 09:01:14PM +0000, Chaitanya Kulkarni wrote:
>> commit 1368a1a5e7566d726bf74234d05895c3f0d54690
>> Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
>> Date:   Wed Jun 9 20:07:00 2021 -0700
>>
>>     nvme: fix the comparison in the mnan check
>>    
>>     The existing check for the valid mnan value will result in the error
>>     when ctrl->max_namespaces are set to the 1024 from NVMeOF target since
>>     !1024 == 0 so it will lead to next comparison 1024 > is->nn which will
>>     be always true untill target has 1024 namespaces.
> The commit message doesn't make sense to me. NSID is not limited to
> 1024.
>
> From the discussion in the other mail, I though the argument is, if
> there are no namespaces on the target the MNAN is allowed to be
> zero. The original check assumed there are always namespaces. The
> proposed fix drops the first half of the specs statement:
>
>   ... then this field shall be set to a non-zero value that is less than
>   or equal to the NN value.
>
>


I think we can keep the code as it is on the host side and fix the
target side.

I've sent out a small series to fix that, see if that makes sense.




From Chaitanya.Kulkarni at wdc.com  Sun Jun 13 14:04:47 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 21:04:47 +0000
Subject: [PATCH 0/3] nvmet: use right MNAN value for controllers
References: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
Message-ID: <BYAPR04MB4965B8BD0B2E0795AA4A84F786329@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/13/21 13:03, Chaitanya Kulkarni wrote:
> Hi,
>
> In current implementation we report MNAN value as NVMET_MAX_NAMESPACES.
>
> For Spec regarding MNAN value:-
>
> If the controller supports Asymmetric Namespace Access Reporting, then
> this field shall be set to a _non-zero_ value that is less than or equal
> to the NN value.
>
> The first patch sets identify controller MNAN value to subsys->max_nsid
> that is always less than or equal to the NN.
>
> The second patch prevents creating the controller from subsys if subsys
> has 0 namespaces since it will report 0 (invalid) MNAN value.
>
> The third patch prevents reporting such subsys in the dicovery log page.
>
> -ck


Please ignore this as some of the block tests are failing I need to figure
out whether any of those needs a fix or something is wrong in this series,
sorry for the noise.




From chaitanya.kulkarni at wdc.com  Sun Jun 13 18:37:53 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 18:37:53 -0700
Subject: [PATCH 0/1] nvmet: use subsys->max_nsid to report MNAN value
Message-ID: <20210614013754.37940-1-chaitanya.kulkarni@wdc.com>

Hi,

Instaed of using the NVMET_MAX_NAMESPACS use subsys->nax_nsid to set
the MNAN value since nvmet controller supports the ANA Reporting.

Without this patch we get following error with connect command :-

[337976.409142] nvme nvme1: Invalid MNAN value 1024

I've ran blocktests [1] and they pass on the top of this patch on
nvme-5.14.

-ck

Chaitanya Kulkarni (1):
  nvmet: use subsys->max_nsid to report MNAN

 drivers/nvme/target/admin-cmd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

# git log -1
commit 353d099f43f8372010bdfa16a9b6ef1cc0233b6f (HEAD -> nvme-5.14)
Author: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Date:   Sat Jun 12 15:10:43 2021 -0700

    nvmet: use subsys->max_nsid to report MNAN
    
    For Spec regarding MNAN value:-
    
    If the controller supports Asymmetric Namespace Access Reporting, then
    this field shall be set to a non-zero value that is less than or equal
    to the NN value.
    
    Instead of using NVMET_MAX_NAMESPACES use subsys->max_nsid to set the
    MNAN value.
    
    Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
# git format-patch HEAD~1 --signoff 
0001-nvmet-use-subsys-max_nsid-to-report-MNAN.patch
# ./compile_nvme.sh 
+ umount /mnt/nvme0n1
+ clear_dmesg
umount: /mnt/nvme0n1: not mounted
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ git apply ./all-fixes.diff
+ sleep 1
+ git diff
diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index cb30cb942e1d..a7f7b4dd8d37 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -692,7 +692,8 @@ static struct nvmf_transport_ops nvme_loop_transport = {
        .name           = "loop",
        .module         = THIS_MODULE,
        .create_ctrl    = nvme_loop_create_ctrl,
-       .allowed_opts   = NVMF_OPT_TRADDR,
+       .allowed_opts   = NVMF_OPT_TRADDR | NVMF_OPT_CTRL_LOSS_TMO,
+
 };
 
 static int __init nvme_loop_init_module(void)
+ sleep 1
++ nproc
+ make -j 64 M=drivers/nvme/ modules
  CC [M]  drivers/nvme//target/loop.o
  LD [M]  drivers/nvme//target/nvme-loop.o
  MODPOST drivers/nvme//Module.symvers
  LD [M]  drivers/nvme//target/nvme-loop.ko
+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/host/
++ uname -r
+ TARGET_DEST=/lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/host/ /lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/target//
/lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/host/:
total 9.1M
-rw-r--r--. 1 root root 4.0M Jun 13 14:23 nvme-core.ko
-rw-r--r--. 1 root root 639K Jun 13 14:23 nvme-fabrics.ko
-rw-r--r--. 1 root root 1.2M Jun 13 14:23 nvme-fc.ko
-rw-r--r--. 1 root root 1.1M Jun 13 14:23 nvme.ko
-rw-r--r--. 1 root root 1.2M Jun 13 14:23 nvme-rdma.ko
-rw-r--r--. 1 root root 1.1M Jun 13 14:23 nvme-tcp.ko

/lib/modules/5.13.0-rc3nvme+/kernel/drivers/nvme/target//:
total 8.3M
-rw-r--r--. 1 root root 718K Jun 13 14:23 nvme-fcloop.ko
-rw-r--r--. 1 root root 623K Jun 13 14:23 nvme-loop.ko
-rw-r--r--. 1 root root 1.1M Jun 13 14:23 nvmet-fc.ko
-rw-r--r--. 1 root root 4.0M Jun 13 14:23 nvmet.ko
-rw-r--r--. 1 root root 1.1M Jun 13 14:23 nvmet-rdma.ko
-rw-r--r--. 1 root root 868K Jun 13 14:23 nvmet-tcp.ko
+ modprobe nvme
+ git co drivers/nvme/target/loop.c
Updated 1 path from the index
# cdblktests 
 blktests (master) # ./check nvme-
./check: no group or test named nvme-
 blktests (master) # ./check nvme
nvme/002 (create many subsystems and test discovery)         [passed]
    runtime  41.158s  ...  40.884s
nvme/003 (test if we're sending keep-alives to a discovery controller) [passed]
    runtime  10.194s  ...  10.179s
nvme/004 (test nvme and nvmet UUID NS descriptors)           [passed]
    runtime  1.780s  ...  1.790s
nvme/005 (reset local loopback target)                       [passed]
    runtime  2.249s  ...  2.229s
nvme/006 (create an NVMeOF target with a block device-backed ns) [passed]
    runtime  0.156s  ...  0.157s
nvme/007 (create an NVMeOF target with a file-backed ns)     [passed]
    runtime  0.107s  ...  0.100s
nvme/008 (create an NVMeOF host with a block device-backed ns) [passed]
    runtime  1.794s  ...  1.810s
nvme/009 (create an NVMeOF host with a file-backed ns)       [passed]
    runtime  1.745s  ...  1.758s
nvme/010 (run data verification fio job on NVMeOF block device-backed ns) [passed]
    runtime  42.438s  ...  36.312s
nvme/011 (run data verification fio job on NVMeOF file-backed ns) [passed]
    runtime  325.840s  ...  305.214s
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]
    runtime  9.217s  ...  9.089s
nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed ns) [passed]
    runtime  25.653s  ...  23.342s
nvme/014 (flush a NVMeOF block device-backed ns)             [passed]
    runtime  21.644s  ...  20.553s
nvme/015 (unit test for NVMe flush for file backed ns)       [passed]
    runtime  21.054s  ...  20.377s
nvme/016 (create/delete many NVMeOF block device-backed ns and test discovery) [passed]
    runtime  21.955s  ...  21.895s
nvme/017 (create/delete many file-ns and test discovery)     [passed]
    runtime  22.402s  ...  21.234s
nvme/018 (unit test NVMe-oF out of range access on a file backend) [passed]
    runtime  1.782s  ...  1.755s
nvme/019 (test NVMe DSM Discard command on NVMeOF block-device ns) [passed]
    runtime  2.332s  ...  1.785s
nvme/020 (test NVMe DSM Discard command on NVMeOF file-backed ns) [passed]
    runtime  1.952s  ...  1.747s
nvme/021 (test NVMe list command on NVMeOF file-backed ns)   [passed]
    runtime  1.966s  ...  1.790s
nvme/022 (test NVMe reset command on NVMeOF file-backed ns)  [passed]
    runtime  2.183s  ...  2.192s
nvme/023 (test NVMe smart-log command on NVMeOF block-device ns) [passed]
    runtime  1.831s  ...  1.828s
nvme/024 (test NVMe smart-log command on NVMeOF file-backed ns) [passed]
    runtime  1.748s  ...  1.734s
nvme/025 (test NVMe effects-log command on NVMeOF file-backed ns) [passed]
    runtime  1.749s  ...  1.748s
nvme/026 (test NVMe ns-descs command on NVMeOF file-backed ns) [passed]
    runtime  1.859s  ...  1.745s
nvme/027 (test NVMe ns-rescan command on NVMeOF file-backed ns) [passed]
    runtime  1.777s  ...  1.745s
nvme/028 (test NVMe list-subsys command on NVMeOF file-backed ns) [passed]
    runtime  1.764s  ...  1.740s
nvme/029 (test userspace IO via nvme-cli read/write interface) [passed]
    runtime  2.277s  ...  2.245s
nvme/030 (ensure the discovery generation counter is updated appropriately) [passed]
    runtime  0.422s  ...  0.414s
nvme/031 (test deletion of NVMeOF controllers immediately after setup) [passed]
    runtime  5.982s  ...  6.045s
nvme/038 (test deletion of NVMeOF subsystem without enabling) [passed]
    runtime  0.060s  ...  0.063s


-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 18:37:54 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 18:37:54 -0700
Subject: [PATCH 1/1] nvmet: use subsys->max_nsid to report MNAN
In-Reply-To: <20210614013754.37940-1-chaitanya.kulkarni@wdc.com>
References: <20210614013754.37940-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210614013754.37940-2-chaitanya.kulkarni@wdc.com>

For Spec regarding MNAN value:-

If the controller supports Asymmetric Namespace Access Reporting, then
this field shall be set to a non-zero value that is less than or equal
to the NN value.

Instead of using NVMET_MAX_NAMESPACES use subsys->max_nsid to set the
MNAN value.

Without this patch it results in the following error :- 

[337976.409142] nvme nvme1: Invalid MNAN value 1024

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index cd60a8184d04..a8ec377bb68d 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -394,7 +394,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
 
 	id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
-	id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
+	id->mnan = cpu_to_le32(ctrl->subsys->max_nsid);
 	id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
 			NVME_CTRL_ONCS_WRITE_ZEROES);
 
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 18:58:45 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 18:58:45 -0700
Subject: [PATCH 0/6] nvmet: small cleanup
Message-ID: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>

Hi,

This has cleanup for removing local variables which are only used once
or not used to do anything meaningful along with type mismatch for
subsys->max_nsid.

-ck

Chaitanya Kulkarni (6):
  nvmet: use req->cmd directly in bdev-ns fast path
  nvmet: use req->cmd directly in file-ns fast path
  nvmet: use u32 for nvmet_subsys max_nsid
  nvmet: use u32 type for the local variable nsid
  nvmet: use nvme status value directly
  nvmet: remove local variable

 drivers/nvme/target/core.c        | 30 ++++++++++--------------------
 drivers/nvme/target/io-cmd-bdev.c |  4 +---
 drivers/nvme/target/io-cmd-file.c |  4 +---
 drivers/nvme/target/nvmet.h       |  2 +-
 4 files changed, 13 insertions(+), 27 deletions(-)

-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 18:58:46 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 18:58:46 -0700
Subject: [PATCH 1/6] nvmet: use req->cmd directly in bdev-ns fast path
In-Reply-To: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
References: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210614015851.53925-2-chaitanya.kulkarni@wdc.com>

The function nvmet_bdev_parse_io_cmd() is called from the fast path.
The local variable to that function cmd is only used once.

Remove the local variable and use req->cmd directly.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/io-cmd-bdev.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index f673679d258a..5d998e5873d3 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -429,9 +429,7 @@ static void nvmet_bdev_execute_write_zeroes(struct nvmet_req *req)
 
 u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req)
 {
-	struct nvme_command *cmd = req->cmd;
-
-	switch (cmd->common.opcode) {
+	switch (req->cmd->common.opcode) {
 	case nvme_cmd_read:
 	case nvme_cmd_write:
 		req->execute = nvmet_bdev_execute_rw;
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 18:58:47 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 18:58:47 -0700
Subject: [PATCH 2/6] nvmet: use req->cmd directly in file-ns fast path
In-Reply-To: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
References: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210614015851.53925-3-chaitanya.kulkarni@wdc.com>

The function nvmet_file_parse_io_cmd() is called from the fast path. The
local variable to that function cmd is only used once.

Remove the local variable and use req->cmd directly.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/io-cmd-file.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c
index 7fdbdc496597..1dd1a0fe2e81 100644
--- a/drivers/nvme/target/io-cmd-file.c
+++ b/drivers/nvme/target/io-cmd-file.c
@@ -385,9 +385,7 @@ static void nvmet_file_execute_write_zeroes(struct nvmet_req *req)
 
 u16 nvmet_file_parse_io_cmd(struct nvmet_req *req)
 {
-	struct nvme_command *cmd = req->cmd;
-
-	switch (cmd->common.opcode) {
+	switch (req->cmd->common.opcode) {
 	case nvme_cmd_read:
 	case nvme_cmd_write:
 		req->execute = nvmet_file_execute_rw;
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 18:58:48 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 18:58:48 -0700
Subject: [PATCH 3/6] nvmet: use u32 for nvmet_subsys max_nsid
In-Reply-To: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
References: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210614015851.53925-4-chaitanya.kulkarni@wdc.com>

Use u32 type for the nsid_max member of the nvmet_subsys structure.
This avoids the type confusion when updating the subsys->nax_nsid from
ns->nsid. This also matches the nvmet_ns->nsid member.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/nvmet.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index bd0a0b91d843..3468f25cb4b7 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -218,7 +218,7 @@ struct nvmet_subsys {
 
 	struct xarray		namespaces;
 	unsigned int		nr_namespaces;
-	unsigned int		max_nsid;
+	u32			max_nsid;
 	u16			cntlid_min;
 	u16			cntlid_max;
 
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 18:58:49 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 18:58:49 -0700
Subject: [PATCH 4/6] nvmet: use u32 type for the local variable nsid
In-Reply-To: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
References: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210614015851.53925-5-chaitanya.kulkarni@wdc.com>

In function nvmet_max_nsid() we calculate the max nsid by iterating
over the XArray and store it in the variable nsid that has type of
unsigned long.

Since the value of this function is stored into the subsys->max_nsid
which is of type u32, change the local variable nsid type and the return
type of the same function to u32.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/core.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 146909486b8f..8494a132da35 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -122,11 +122,11 @@ u16 nvmet_zero_sgl(struct nvmet_req *req, off_t off, size_t len)
 	return 0;
 }
 
-static unsigned int nvmet_max_nsid(struct nvmet_subsys *subsys)
+static u32 nvmet_max_nsid(struct nvmet_subsys *subsys)
 {
-	unsigned long nsid = 0;
 	struct nvmet_ns *cur;
 	unsigned long idx;
+	u32 nsid = 0;
 
 	xa_for_each(&subsys->namespaces, idx, cur)
 		nsid = cur->nsid;
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 18:58:50 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 18:58:50 -0700
Subject: [PATCH 5/6] nvmet: use nvme status value directly
In-Reply-To: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
References: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210614015851.53925-6-chaitanya.kulkarni@wdc.com>

There is no point in keeping the status variable that is used only once
in the function nvmet_async_events_failall().

Remove the variable and use the value directly.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/core.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 8494a132da35..45a5b273b525 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -141,14 +141,13 @@ static u32 nvmet_async_event_result(struct nvmet_async_event *aen)
 
 static void nvmet_async_events_failall(struct nvmet_ctrl *ctrl)
 {
-	u16 status = NVME_SC_INTERNAL | NVME_SC_DNR;
 	struct nvmet_req *req;
 
 	mutex_lock(&ctrl->lock);
 	while (ctrl->nr_async_event_cmds) {
 		req = ctrl->async_event_cmds[--ctrl->nr_async_event_cmds];
 		mutex_unlock(&ctrl->lock);
-		nvmet_req_complete(req, status);
+		nvmet_req_complete(req, NVME_SC_INTERNAL | NVME_SC_DNR);
 		mutex_lock(&ctrl->lock);
 	}
 	mutex_unlock(&ctrl->lock);
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 18:58:51 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 18:58:51 -0700
Subject: [PATCH 6/6] nvmet: remove local variable
In-Reply-To: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
References: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210614015851.53925-7-chaitanya.kulkarni@wdc.com>

In function errno_to_nvme_status() we store the value of the NVMe
status into the local variable and don't do anything useful with that
but just return.

Remove the local variable and return the value directly from switch.
This also removed extra break statements.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/core.c | 23 +++++++----------------
 1 file changed, 7 insertions(+), 16 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 45a5b273b525..c8708dcaeaa5 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -43,43 +43,34 @@ DECLARE_RWSEM(nvmet_ana_sem);
 
 inline u16 errno_to_nvme_status(struct nvmet_req *req, int errno)
 {
-	u16 status;
-
 	switch (errno) {
 	case 0:
-		status = NVME_SC_SUCCESS;
-		break;
+		return NVME_SC_SUCCESS;
 	case -ENOSPC:
 		req->error_loc = offsetof(struct nvme_rw_command, length);
-		status = NVME_SC_CAP_EXCEEDED | NVME_SC_DNR;
-		break;
+		return NVME_SC_CAP_EXCEEDED | NVME_SC_DNR;
 	case -EREMOTEIO:
 		req->error_loc = offsetof(struct nvme_rw_command, slba);
-		status = NVME_SC_LBA_RANGE | NVME_SC_DNR;
-		break;
+		return  NVME_SC_LBA_RANGE | NVME_SC_DNR;
 	case -EOPNOTSUPP:
 		req->error_loc = offsetof(struct nvme_common_command, opcode);
 		switch (req->cmd->common.opcode) {
 		case nvme_cmd_dsm:
 		case nvme_cmd_write_zeroes:
-			status = NVME_SC_ONCS_NOT_SUPPORTED | NVME_SC_DNR;
-			break;
+			return NVME_SC_ONCS_NOT_SUPPORTED | NVME_SC_DNR;
 		default:
-			status = NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
+			return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
 		}
 		break;
 	case -ENODATA:
 		req->error_loc = offsetof(struct nvme_rw_command, nsid);
-		status = NVME_SC_ACCESS_DENIED;
-		break;
+		return NVME_SC_ACCESS_DENIED;
 	case -EIO:
 		fallthrough;
 	default:
 		req->error_loc = offsetof(struct nvme_common_command, opcode);
-		status = NVME_SC_INTERNAL | NVME_SC_DNR;
+		return NVME_SC_INTERNAL | NVME_SC_DNR;
 	}
-
-	return status;
 }
 
 u16 nvmet_report_invalid_opcode(struct nvmet_req *req)
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 19:49:59 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 19:49:59 -0700
Subject: [PATCH RFC] nvme: move sysfs code into separate file
Message-ID: <20210614024959.11925-1-chaitanya.kulkarni@wdc.com>

Move sysfs related code into it's separate file since to trim down the
host/core.c.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/Makefile |   2 +-
 drivers/nvme/host/core.c   | 440 +-----------------------------------
 drivers/nvme/host/nvme.h   |   4 +
 drivers/nvme/host/sysfs.c  | 441 +++++++++++++++++++++++++++++++++++++
 4 files changed, 449 insertions(+), 438 deletions(-)
 create mode 100644 drivers/nvme/host/sysfs.c

diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index cbc509784b2e..a89f15194082 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -9,7 +9,7 @@ obj-$(CONFIG_NVME_RDMA)			+= nvme-rdma.o
 obj-$(CONFIG_NVME_FC)			+= nvme-fc.o
 obj-$(CONFIG_NVME_TCP)			+= nvme-tcp.o
 
-nvme-core-y				:= core.o ioctl.o
+nvme-core-y				:= core.o ioctl.o sysfs.o
 nvme-core-$(CONFIG_TRACING)		+= trace.o
 nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_NVM)			+= lightnvm.o
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index cb1a76a3d42a..42945af7059c 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -217,7 +217,7 @@ int nvme_reset_ctrl_sync(struct nvme_ctrl *ctrl)
 	return ret;
 }
 
-static void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl)
+void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl)
 {
 	dev_info(ctrl->device,
 		 "Removing ctrl: NQN \"%s\"\n", ctrl->opts->subsysnqn);
@@ -247,18 +247,6 @@ int nvme_delete_ctrl(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_delete_ctrl);
 
-static void nvme_delete_ctrl_sync(struct nvme_ctrl *ctrl)
-{
-	/*
-	 * Keep a reference until nvme_do_delete_ctrl() complete,
-	 * since ->delete_ctrl can free the controller.
-	 */
-	nvme_get_ctrl(ctrl);
-	if (nvme_change_ctrl_state(ctrl, NVME_CTRL_DELETING))
-		nvme_do_delete_ctrl(ctrl);
-	nvme_put_ctrl(ctrl);
-}
-
 static blk_status_t nvme_error_status(u16 status)
 {
 	switch (status & 0x7ff) {
@@ -2073,7 +2061,7 @@ static int nvme_report_zones(struct gendisk *disk, sector_t sector,
 #define nvme_report_zones	NULL
 #endif /* CONFIG_BLK_DEV_ZONED */
 
-static const struct block_device_operations nvme_bdev_ops = {
+const struct block_device_operations nvme_bdev_ops = {
 	.owner		= THIS_MODULE,
 	.ioctl		= nvme_ioctl,
 	.open		= nvme_open,
@@ -3078,428 +3066,6 @@ static const struct file_operations nvme_dev_fops = {
 	.compat_ioctl	= compat_ptr_ioctl,
 };
 
-static ssize_t nvme_sysfs_reset(struct device *dev,
-				struct device_attribute *attr, const char *buf,
-				size_t count)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-	int ret;
-
-	ret = nvme_reset_ctrl_sync(ctrl);
-	if (ret < 0)
-		return ret;
-	return count;
-}
-static DEVICE_ATTR(reset_controller, S_IWUSR, NULL, nvme_sysfs_reset);
-
-static ssize_t nvme_sysfs_rescan(struct device *dev,
-				struct device_attribute *attr, const char *buf,
-				size_t count)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-
-	nvme_queue_scan(ctrl);
-	return count;
-}
-static DEVICE_ATTR(rescan_controller, S_IWUSR, NULL, nvme_sysfs_rescan);
-
-static inline struct nvme_ns_head *dev_to_ns_head(struct device *dev)
-{
-	struct gendisk *disk = dev_to_disk(dev);
-
-	if (disk->fops == &nvme_bdev_ops)
-		return nvme_get_ns_from_dev(dev)->head;
-	else
-		return disk->private_data;
-}
-
-static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
-		char *buf)
-{
-	struct nvme_ns_head *head = dev_to_ns_head(dev);
-	struct nvme_ns_ids *ids = &head->ids;
-	struct nvme_subsystem *subsys = head->subsys;
-	int serial_len = sizeof(subsys->serial);
-	int model_len = sizeof(subsys->model);
-
-	if (!uuid_is_null(&ids->uuid))
-		return sysfs_emit(buf, "uuid.%pU\n", &ids->uuid);
-
-	if (memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
-		return sysfs_emit(buf, "eui.%16phN\n", ids->nguid);
-
-	if (memchr_inv(ids->eui64, 0, sizeof(ids->eui64)))
-		return sysfs_emit(buf, "eui.%8phN\n", ids->eui64);
-
-	while (serial_len > 0 && (subsys->serial[serial_len - 1] == ' ' ||
-				  subsys->serial[serial_len - 1] == '\0'))
-		serial_len--;
-	while (model_len > 0 && (subsys->model[model_len - 1] == ' ' ||
-				 subsys->model[model_len - 1] == '\0'))
-		model_len--;
-
-	return sysfs_emit(buf, "nvme.%04x-%*phN-%*phN-%08x\n", subsys->vendor_id,
-		serial_len, subsys->serial, model_len, subsys->model,
-		head->ns_id);
-}
-static DEVICE_ATTR_RO(wwid);
-
-static ssize_t nguid_show(struct device *dev, struct device_attribute *attr,
-		char *buf)
-{
-	return sysfs_emit(buf, "%pU\n", dev_to_ns_head(dev)->ids.nguid);
-}
-static DEVICE_ATTR_RO(nguid);
-
-static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
-		char *buf)
-{
-	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
-
-	/* For backward compatibility expose the NGUID to userspace if
-	 * we have no UUID set
-	 */
-	if (uuid_is_null(&ids->uuid)) {
-		printk_ratelimited(KERN_WARNING
-				   "No UUID available providing old NGUID\n");
-		return sysfs_emit(buf, "%pU\n", ids->nguid);
-	}
-	return sysfs_emit(buf, "%pU\n", &ids->uuid);
-}
-static DEVICE_ATTR_RO(uuid);
-
-static ssize_t eui_show(struct device *dev, struct device_attribute *attr,
-		char *buf)
-{
-	return sysfs_emit(buf, "%8ph\n", dev_to_ns_head(dev)->ids.eui64);
-}
-static DEVICE_ATTR_RO(eui);
-
-static ssize_t nsid_show(struct device *dev, struct device_attribute *attr,
-		char *buf)
-{
-	return sysfs_emit(buf, "%d\n", dev_to_ns_head(dev)->ns_id);
-}
-static DEVICE_ATTR_RO(nsid);
-
-static struct attribute *nvme_ns_id_attrs[] = {
-	&dev_attr_wwid.attr,
-	&dev_attr_uuid.attr,
-	&dev_attr_nguid.attr,
-	&dev_attr_eui.attr,
-	&dev_attr_nsid.attr,
-#ifdef CONFIG_NVME_MULTIPATH
-	&dev_attr_ana_grpid.attr,
-	&dev_attr_ana_state.attr,
-#endif
-	NULL,
-};
-
-static umode_t nvme_ns_id_attrs_are_visible(struct kobject *kobj,
-		struct attribute *a, int n)
-{
-	struct device *dev = container_of(kobj, struct device, kobj);
-	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
-
-	if (a == &dev_attr_uuid.attr) {
-		if (uuid_is_null(&ids->uuid) &&
-		    !memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
-			return 0;
-	}
-	if (a == &dev_attr_nguid.attr) {
-		if (!memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
-			return 0;
-	}
-	if (a == &dev_attr_eui.attr) {
-		if (!memchr_inv(ids->eui64, 0, sizeof(ids->eui64)))
-			return 0;
-	}
-#ifdef CONFIG_NVME_MULTIPATH
-	if (a == &dev_attr_ana_grpid.attr || a == &dev_attr_ana_state.attr) {
-		if (dev_to_disk(dev)->fops != &nvme_bdev_ops) /* per-path attr */
-			return 0;
-		if (!nvme_ctrl_use_ana(nvme_get_ns_from_dev(dev)->ctrl))
-			return 0;
-	}
-#endif
-	return a->mode;
-}
-
-static const struct attribute_group nvme_ns_id_attr_group = {
-	.attrs		= nvme_ns_id_attrs,
-	.is_visible	= nvme_ns_id_attrs_are_visible,
-};
-
-const struct attribute_group *nvme_ns_id_attr_groups[] = {
-	&nvme_ns_id_attr_group,
-#ifdef CONFIG_NVM
-	&nvme_nvm_attr_group,
-#endif
-	NULL,
-};
-
-#define nvme_show_str_function(field)						\
-static ssize_t  field##_show(struct device *dev,				\
-			    struct device_attribute *attr, char *buf)		\
-{										\
-        struct nvme_ctrl *ctrl = dev_get_drvdata(dev);				\
-        return sysfs_emit(buf, "%.*s\n",					\
-		(int)sizeof(ctrl->subsys->field), ctrl->subsys->field);		\
-}										\
-static DEVICE_ATTR(field, S_IRUGO, field##_show, NULL);
-
-nvme_show_str_function(model);
-nvme_show_str_function(serial);
-nvme_show_str_function(firmware_rev);
-
-#define nvme_show_int_function(field)						\
-static ssize_t  field##_show(struct device *dev,				\
-			    struct device_attribute *attr, char *buf)		\
-{										\
-        struct nvme_ctrl *ctrl = dev_get_drvdata(dev);				\
-        return sysfs_emit(buf, "%d\n", ctrl->field);				\
-}										\
-static DEVICE_ATTR(field, S_IRUGO, field##_show, NULL);
-
-nvme_show_int_function(cntlid);
-nvme_show_int_function(numa_node);
-nvme_show_int_function(queue_count);
-nvme_show_int_function(sqsize);
-nvme_show_int_function(kato);
-
-static ssize_t nvme_sysfs_delete(struct device *dev,
-				struct device_attribute *attr, const char *buf,
-				size_t count)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-
-	if (device_remove_file_self(dev, attr))
-		nvme_delete_ctrl_sync(ctrl);
-	return count;
-}
-static DEVICE_ATTR(delete_controller, S_IWUSR, NULL, nvme_sysfs_delete);
-
-static ssize_t nvme_sysfs_show_transport(struct device *dev,
-					 struct device_attribute *attr,
-					 char *buf)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-
-	return sysfs_emit(buf, "%s\n", ctrl->ops->name);
-}
-static DEVICE_ATTR(transport, S_IRUGO, nvme_sysfs_show_transport, NULL);
-
-static ssize_t nvme_sysfs_show_state(struct device *dev,
-				     struct device_attribute *attr,
-				     char *buf)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-	static const char *const state_name[] = {
-		[NVME_CTRL_NEW]		= "new",
-		[NVME_CTRL_LIVE]	= "live",
-		[NVME_CTRL_RESETTING]	= "resetting",
-		[NVME_CTRL_CONNECTING]	= "connecting",
-		[NVME_CTRL_DELETING]	= "deleting",
-		[NVME_CTRL_DELETING_NOIO]= "deleting (no IO)",
-		[NVME_CTRL_DEAD]	= "dead",
-	};
-
-	if ((unsigned)ctrl->state < ARRAY_SIZE(state_name) &&
-	    state_name[ctrl->state])
-		return sysfs_emit(buf, "%s\n", state_name[ctrl->state]);
-
-	return sysfs_emit(buf, "unknown state\n");
-}
-
-static DEVICE_ATTR(state, S_IRUGO, nvme_sysfs_show_state, NULL);
-
-static ssize_t nvme_sysfs_show_subsysnqn(struct device *dev,
-					 struct device_attribute *attr,
-					 char *buf)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-
-	return sysfs_emit(buf, "%s\n", ctrl->subsys->subnqn);
-}
-static DEVICE_ATTR(subsysnqn, S_IRUGO, nvme_sysfs_show_subsysnqn, NULL);
-
-static ssize_t nvme_sysfs_show_hostnqn(struct device *dev,
-					struct device_attribute *attr,
-					char *buf)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-
-	return sysfs_emit(buf, "%s\n", ctrl->opts->host->nqn);
-}
-static DEVICE_ATTR(hostnqn, S_IRUGO, nvme_sysfs_show_hostnqn, NULL);
-
-static ssize_t nvme_sysfs_show_hostid(struct device *dev,
-					struct device_attribute *attr,
-					char *buf)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-
-	return sysfs_emit(buf, "%pU\n", &ctrl->opts->host->id);
-}
-static DEVICE_ATTR(hostid, S_IRUGO, nvme_sysfs_show_hostid, NULL);
-
-static ssize_t nvme_sysfs_show_address(struct device *dev,
-					 struct device_attribute *attr,
-					 char *buf)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-
-	return ctrl->ops->get_address(ctrl, buf, PAGE_SIZE);
-}
-static DEVICE_ATTR(address, S_IRUGO, nvme_sysfs_show_address, NULL);
-
-static ssize_t nvme_ctrl_loss_tmo_show(struct device *dev,
-		struct device_attribute *attr, char *buf)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-	struct nvmf_ctrl_options *opts = ctrl->opts;
-
-	if (ctrl->opts->max_reconnects == -1)
-		return sysfs_emit(buf, "off\n");
-	return sysfs_emit(buf, "%d\n",
-			  opts->max_reconnects * opts->reconnect_delay);
-}
-
-static ssize_t nvme_ctrl_loss_tmo_store(struct device *dev,
-		struct device_attribute *attr, const char *buf, size_t count)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-	struct nvmf_ctrl_options *opts = ctrl->opts;
-	int ctrl_loss_tmo, err;
-
-	err = kstrtoint(buf, 10, &ctrl_loss_tmo);
-	if (err)
-		return -EINVAL;
-
-	if (ctrl_loss_tmo < 0)
-		opts->max_reconnects = -1;
-	else
-		opts->max_reconnects = DIV_ROUND_UP(ctrl_loss_tmo,
-						opts->reconnect_delay);
-	return count;
-}
-static DEVICE_ATTR(ctrl_loss_tmo, S_IRUGO | S_IWUSR,
-	nvme_ctrl_loss_tmo_show, nvme_ctrl_loss_tmo_store);
-
-static ssize_t nvme_ctrl_reconnect_delay_show(struct device *dev,
-		struct device_attribute *attr, char *buf)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-
-	if (ctrl->opts->reconnect_delay == -1)
-		return sysfs_emit(buf, "off\n");
-	return sysfs_emit(buf, "%d\n", ctrl->opts->reconnect_delay);
-}
-
-static ssize_t nvme_ctrl_reconnect_delay_store(struct device *dev,
-		struct device_attribute *attr, const char *buf, size_t count)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-	unsigned int v;
-	int err;
-
-	err = kstrtou32(buf, 10, &v);
-	if (err)
-		return err;
-
-	ctrl->opts->reconnect_delay = v;
-	return count;
-}
-static DEVICE_ATTR(reconnect_delay, S_IRUGO | S_IWUSR,
-	nvme_ctrl_reconnect_delay_show, nvme_ctrl_reconnect_delay_store);
-
-static ssize_t nvme_ctrl_fast_io_fail_tmo_show(struct device *dev,
-		struct device_attribute *attr, char *buf)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-
-	if (ctrl->opts->fast_io_fail_tmo == -1)
-		return sysfs_emit(buf, "off\n");
-	return sysfs_emit(buf, "%d\n", ctrl->opts->fast_io_fail_tmo);
-}
-
-static ssize_t nvme_ctrl_fast_io_fail_tmo_store(struct device *dev,
-		struct device_attribute *attr, const char *buf, size_t count)
-{
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-	struct nvmf_ctrl_options *opts = ctrl->opts;
-	int fast_io_fail_tmo, err;
-
-	err = kstrtoint(buf, 10, &fast_io_fail_tmo);
-	if (err)
-		return -EINVAL;
-
-	if (fast_io_fail_tmo < 0)
-		opts->fast_io_fail_tmo = -1;
-	else
-		opts->fast_io_fail_tmo = fast_io_fail_tmo;
-	return count;
-}
-static DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
-	nvme_ctrl_fast_io_fail_tmo_show, nvme_ctrl_fast_io_fail_tmo_store);
-
-static struct attribute *nvme_dev_attrs[] = {
-	&dev_attr_reset_controller.attr,
-	&dev_attr_rescan_controller.attr,
-	&dev_attr_model.attr,
-	&dev_attr_serial.attr,
-	&dev_attr_firmware_rev.attr,
-	&dev_attr_cntlid.attr,
-	&dev_attr_delete_controller.attr,
-	&dev_attr_transport.attr,
-	&dev_attr_subsysnqn.attr,
-	&dev_attr_address.attr,
-	&dev_attr_state.attr,
-	&dev_attr_numa_node.attr,
-	&dev_attr_queue_count.attr,
-	&dev_attr_sqsize.attr,
-	&dev_attr_hostnqn.attr,
-	&dev_attr_hostid.attr,
-	&dev_attr_ctrl_loss_tmo.attr,
-	&dev_attr_reconnect_delay.attr,
-	&dev_attr_fast_io_fail_tmo.attr,
-	&dev_attr_kato.attr,
-	NULL
-};
-
-static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
-		struct attribute *a, int n)
-{
-	struct device *dev = container_of(kobj, struct device, kobj);
-	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
-
-	if (a == &dev_attr_delete_controller.attr && !ctrl->ops->delete_ctrl)
-		return 0;
-	if (a == &dev_attr_address.attr && !ctrl->ops->get_address)
-		return 0;
-	if (a == &dev_attr_hostnqn.attr && !ctrl->opts)
-		return 0;
-	if (a == &dev_attr_hostid.attr && !ctrl->opts)
-		return 0;
-	if (a == &dev_attr_ctrl_loss_tmo.attr && !ctrl->opts)
-		return 0;
-	if (a == &dev_attr_reconnect_delay.attr && !ctrl->opts)
-		return 0;
-	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
-		return 0;
-
-	return a->mode;
-}
-
-static const struct attribute_group nvme_dev_attrs_group = {
-	.attrs		= nvme_dev_attrs,
-	.is_visible	= nvme_dev_attrs_are_visible,
-};
-
-static const struct attribute_group *nvme_dev_attr_groups[] = {
-	&nvme_dev_attrs_group,
-	NULL,
-};
 
 static struct nvme_ns_head *nvme_find_ns_head(struct nvme_subsystem *subsys,
 		unsigned nsid)
@@ -4417,7 +3983,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 			ctrl->instance);
 	ctrl->device->class = nvme_class;
 	ctrl->device->parent = ctrl->dev;
-	ctrl->device->groups = nvme_dev_attr_groups;
+	ctrl->device->groups = nvme_get_device_attr_groups();
 	ctrl->device->release = nvme_free_ctrl;
 	dev_set_drvdata(ctrl->device, ctrl);
 	ret = dev_set_name(ctrl->device, "nvme%d", ctrl->instance);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1f397ecba16c..cc8052a6c196 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -36,6 +36,9 @@ extern unsigned int admin_timeout;
 #define  NVME_INLINE_METADATA_SG_CNT  1
 #endif
 
+const struct attribute_group **nvme_get_device_attr_groups(void);
+extern const struct block_device_operations nvme_bdev_ops;
+
 /*
  * Default to a 4K page size, with the intention to update this
  * path in the future to accommodate architectures with differing
@@ -670,6 +673,7 @@ void nvme_stop_keep_alive(struct nvme_ctrl *ctrl);
 int nvme_reset_ctrl(struct nvme_ctrl *ctrl);
 int nvme_reset_ctrl_sync(struct nvme_ctrl *ctrl);
 int nvme_try_sched_reset(struct nvme_ctrl *ctrl);
+void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl);
 int nvme_delete_ctrl(struct nvme_ctrl *ctrl);
 void nvme_queue_scan(struct nvme_ctrl *ctrl);
 int nvme_get_log(struct nvme_ctrl *ctrl, u32 nsid, u8 log_page, u8 lsp, u8 csi,
diff --git a/drivers/nvme/host/sysfs.c b/drivers/nvme/host/sysfs.c
new file mode 100644
index 000000000000..9cb46012bfa3
--- /dev/null
+++ b/drivers/nvme/host/sysfs.c
@@ -0,0 +1,441 @@
+#include "nvme.h"
+#include "fabrics.h"
+
+static ssize_t nvme_sysfs_reset(struct device *dev,
+				struct device_attribute *attr, const char *buf,
+				size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	int ret;
+
+	ret = nvme_reset_ctrl_sync(ctrl);
+	if (ret < 0)
+		return ret;
+	return count;
+}
+static DEVICE_ATTR(reset_controller, S_IWUSR, NULL, nvme_sysfs_reset);
+
+static ssize_t nvme_sysfs_rescan(struct device *dev,
+				struct device_attribute *attr, const char *buf,
+				size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	nvme_queue_scan(ctrl);
+	return count;
+}
+static DEVICE_ATTR(rescan_controller, S_IWUSR, NULL, nvme_sysfs_rescan);
+
+static inline struct nvme_ns_head *dev_to_ns_head(struct device *dev)
+{
+	struct gendisk *disk = dev_to_disk(dev);
+
+	if (disk->fops == &nvme_bdev_ops)
+		return nvme_get_ns_from_dev(dev)->head;
+	else
+		return disk->private_data;
+}
+
+static ssize_t wwid_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	struct nvme_ns_head *head = dev_to_ns_head(dev);
+	struct nvme_ns_ids *ids = &head->ids;
+	struct nvme_subsystem *subsys = head->subsys;
+	int serial_len = sizeof(subsys->serial);
+	int model_len = sizeof(subsys->model);
+
+	if (!uuid_is_null(&ids->uuid))
+		return sysfs_emit(buf, "uuid.%pU\n", &ids->uuid);
+
+	if (memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
+		return sysfs_emit(buf, "eui.%16phN\n", ids->nguid);
+
+	if (memchr_inv(ids->eui64, 0, sizeof(ids->eui64)))
+		return sysfs_emit(buf, "eui.%8phN\n", ids->eui64);
+
+	while (serial_len > 0 && (subsys->serial[serial_len - 1] == ' ' ||
+				  subsys->serial[serial_len - 1] == '\0'))
+		serial_len--;
+	while (model_len > 0 && (subsys->model[model_len - 1] == ' ' ||
+				 subsys->model[model_len - 1] == '\0'))
+		model_len--;
+
+	return sysfs_emit(buf, "nvme.%04x-%*phN-%*phN-%08x\n", subsys->vendor_id,
+		serial_len, subsys->serial, model_len, subsys->model,
+		head->ns_id);
+}
+static DEVICE_ATTR_RO(wwid);
+
+static ssize_t nguid_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	return sysfs_emit(buf, "%pU\n", dev_to_ns_head(dev)->ids.nguid);
+}
+static DEVICE_ATTR_RO(nguid);
+
+static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
+
+	/*
+	 * For backward compatibility expose the NGUID to userspace if
+	 * we have no UUID set
+	 */
+	if (uuid_is_null(&ids->uuid)) {
+		printk_ratelimited(KERN_WARNING
+				   "No UUID available providing old NGUID\n");
+		return sysfs_emit(buf, "%pU\n", ids->nguid);
+	}
+	return sysfs_emit(buf, "%pU\n", &ids->uuid);
+}
+static DEVICE_ATTR_RO(uuid);
+
+static ssize_t eui_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	return sysfs_emit(buf, "%8ph\n", dev_to_ns_head(dev)->ids.eui64);
+}
+static DEVICE_ATTR_RO(eui);
+
+static ssize_t nsid_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	return sysfs_emit(buf, "%d\n", dev_to_ns_head(dev)->ns_id);
+}
+static DEVICE_ATTR_RO(nsid);
+
+static struct attribute *nvme_ns_id_attrs[] = {
+	&dev_attr_wwid.attr,
+	&dev_attr_uuid.attr,
+	&dev_attr_nguid.attr,
+	&dev_attr_eui.attr,
+	&dev_attr_nsid.attr,
+#ifdef CONFIG_NVME_MULTIPATH
+	&dev_attr_ana_grpid.attr,
+	&dev_attr_ana_state.attr,
+#endif
+	NULL,
+};
+
+static umode_t nvme_ns_id_attrs_are_visible(struct kobject *kobj,
+		struct attribute *a, int n)
+{
+	struct device *dev = container_of(kobj, struct device, kobj);
+	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
+
+	if (a == &dev_attr_uuid.attr) {
+		if (uuid_is_null(&ids->uuid) &&
+		    !memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
+			return 0;
+	}
+	if (a == &dev_attr_nguid.attr) {
+		if (!memchr_inv(ids->nguid, 0, sizeof(ids->nguid)))
+			return 0;
+	}
+	if (a == &dev_attr_eui.attr) {
+		if (!memchr_inv(ids->eui64, 0, sizeof(ids->eui64)))
+			return 0;
+	}
+#ifdef CONFIG_NVME_MULTIPATH
+	if (a == &dev_attr_ana_grpid.attr || a == &dev_attr_ana_state.attr) {
+		if (dev_to_disk(dev)->fops != &nvme_bdev_ops) /* per-path attr */
+			return 0;
+		if (!nvme_ctrl_use_ana(nvme_get_ns_from_dev(dev)->ctrl))
+			return 0;
+	}
+#endif
+	return a->mode;
+}
+
+static const struct attribute_group nvme_ns_id_attr_group = {
+	.attrs		= nvme_ns_id_attrs,
+	.is_visible	= nvme_ns_id_attrs_are_visible,
+};
+
+const struct attribute_group *nvme_ns_id_attr_groups[] = {
+	&nvme_ns_id_attr_group,
+#ifdef CONFIG_NVM
+	&nvme_nvm_attr_group,
+#endif
+	NULL,
+};
+
+#define nvme_show_str_function(field)						\
+static ssize_t  field##_show(struct device *dev,				\
+			    struct device_attribute *attr, char *buf)		\
+{										\
+        struct nvme_ctrl *ctrl = dev_get_drvdata(dev);				\
+        return sysfs_emit(buf, "%.*s\n",					\
+		(int)sizeof(ctrl->subsys->field), ctrl->subsys->field);		\
+}										\
+static DEVICE_ATTR(field, S_IRUGO, field##_show, NULL);
+
+nvme_show_str_function(model);
+nvme_show_str_function(serial);
+nvme_show_str_function(firmware_rev);
+
+#define nvme_show_int_function(field)						\
+static ssize_t  field##_show(struct device *dev,				\
+			    struct device_attribute *attr, char *buf)		\
+{										\
+        struct nvme_ctrl *ctrl = dev_get_drvdata(dev);				\
+        return sysfs_emit(buf, "%d\n", ctrl->field);				\
+}										\
+static DEVICE_ATTR(field, S_IRUGO, field##_show, NULL);
+
+nvme_show_int_function(cntlid);
+nvme_show_int_function(numa_node);
+nvme_show_int_function(queue_count);
+nvme_show_int_function(sqsize);
+nvme_show_int_function(kato);
+
+static ssize_t nvme_sysfs_delete(struct device *dev,
+				struct device_attribute *attr, const char *buf,
+				size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	if (device_remove_file_self(dev, attr)) {
+
+		/*
+		 * Keep a reference until nvme_do_delete_ctrl() complete,
+		 * since ->delete_ctrl can free the controller. This deletes
+		 * ctrl synchronously.
+		 */
+		nvme_get_ctrl(ctrl);
+		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_DELETING))
+			nvme_do_delete_ctrl(ctrl);
+		nvme_put_ctrl(ctrl);
+	}
+	return count;
+}
+static DEVICE_ATTR(delete_controller, S_IWUSR, NULL, nvme_sysfs_delete);
+
+static ssize_t nvme_sysfs_show_transport(struct device *dev,
+					 struct device_attribute *attr,
+					 char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	return sysfs_emit(buf, "%s\n", ctrl->ops->name);
+}
+static DEVICE_ATTR(transport, S_IRUGO, nvme_sysfs_show_transport, NULL);
+
+static ssize_t nvme_sysfs_show_state(struct device *dev,
+				     struct device_attribute *attr,
+				     char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	static const char *const state_name[] = {
+		[NVME_CTRL_NEW]		= "new",
+		[NVME_CTRL_LIVE]	= "live",
+		[NVME_CTRL_RESETTING]	= "resetting",
+		[NVME_CTRL_CONNECTING]	= "connecting",
+		[NVME_CTRL_DELETING]	= "deleting",
+		[NVME_CTRL_DELETING_NOIO]= "deleting (no IO)",
+		[NVME_CTRL_DEAD]	= "dead",
+	};
+
+	if ((unsigned)ctrl->state < ARRAY_SIZE(state_name) &&
+	    state_name[ctrl->state])
+		return sysfs_emit(buf, "%s\n", state_name[ctrl->state]);
+
+	return sysfs_emit(buf, "unknown state\n");
+}
+
+static DEVICE_ATTR(state, S_IRUGO, nvme_sysfs_show_state, NULL);
+
+static ssize_t nvme_sysfs_show_subsysnqn(struct device *dev,
+					 struct device_attribute *attr,
+					 char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	return sysfs_emit(buf, "%s\n", ctrl->subsys->subnqn);
+}
+static DEVICE_ATTR(subsysnqn, S_IRUGO, nvme_sysfs_show_subsysnqn, NULL);
+
+static ssize_t nvme_sysfs_show_hostnqn(struct device *dev,
+					struct device_attribute *attr,
+					char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	return sysfs_emit(buf, "%s\n", ctrl->opts->host->nqn);
+}
+static DEVICE_ATTR(hostnqn, S_IRUGO, nvme_sysfs_show_hostnqn, NULL);
+
+static ssize_t nvme_sysfs_show_hostid(struct device *dev,
+					struct device_attribute *attr,
+					char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	return sysfs_emit(buf, "%pU\n", &ctrl->opts->host->id);
+}
+static DEVICE_ATTR(hostid, S_IRUGO, nvme_sysfs_show_hostid, NULL);
+
+static ssize_t nvme_sysfs_show_address(struct device *dev,
+					 struct device_attribute *attr,
+					 char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	return ctrl->ops->get_address(ctrl, buf, PAGE_SIZE);
+}
+static DEVICE_ATTR(address, S_IRUGO, nvme_sysfs_show_address, NULL);
+
+static ssize_t nvme_ctrl_loss_tmo_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (ctrl->opts->max_reconnects == -1)
+		return sysfs_emit(buf, "off\n");
+	return sysfs_emit(buf, "%d\n",
+			  opts->max_reconnects * opts->reconnect_delay);
+}
+
+static ssize_t nvme_ctrl_loss_tmo_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	int ctrl_loss_tmo, err;
+
+	err = kstrtoint(buf, 10, &ctrl_loss_tmo);
+	if (err)
+		return -EINVAL;
+
+	if (ctrl_loss_tmo < 0)
+		opts->max_reconnects = -1;
+	else
+		opts->max_reconnects = DIV_ROUND_UP(ctrl_loss_tmo,
+						opts->reconnect_delay);
+	return count;
+}
+static DEVICE_ATTR(ctrl_loss_tmo, S_IRUGO | S_IWUSR,
+	nvme_ctrl_loss_tmo_show, nvme_ctrl_loss_tmo_store);
+
+static ssize_t nvme_ctrl_reconnect_delay_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	if (ctrl->opts->reconnect_delay == -1)
+		return sysfs_emit(buf, "off\n");
+	return sysfs_emit(buf, "%d\n", ctrl->opts->reconnect_delay);
+}
+
+static ssize_t nvme_ctrl_reconnect_delay_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	unsigned int v;
+	int err;
+
+	err = kstrtou32(buf, 10, &v);
+	if (err)
+		return err;
+
+	ctrl->opts->reconnect_delay = v;
+	return count;
+}
+static DEVICE_ATTR(reconnect_delay, S_IRUGO | S_IWUSR,
+	nvme_ctrl_reconnect_delay_show, nvme_ctrl_reconnect_delay_store);
+
+static ssize_t nvme_ctrl_fast_io_fail_tmo_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	if (ctrl->opts->fast_io_fail_tmo == -1)
+		return sysfs_emit(buf, "off\n");
+	return sysfs_emit(buf, "%d\n", ctrl->opts->fast_io_fail_tmo);
+}
+
+static ssize_t nvme_ctrl_fast_io_fail_tmo_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	int fast_io_fail_tmo, err;
+
+	err = kstrtoint(buf, 10, &fast_io_fail_tmo);
+	if (err)
+		return -EINVAL;
+
+	if (fast_io_fail_tmo < 0)
+		opts->fast_io_fail_tmo = -1;
+	else
+		opts->fast_io_fail_tmo = fast_io_fail_tmo;
+	return count;
+}
+static DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
+	nvme_ctrl_fast_io_fail_tmo_show, nvme_ctrl_fast_io_fail_tmo_store);
+
+static struct attribute *nvme_dev_attrs[] = {
+	&dev_attr_reset_controller.attr,
+	&dev_attr_rescan_controller.attr,
+	&dev_attr_model.attr,
+	&dev_attr_serial.attr,
+	&dev_attr_firmware_rev.attr,
+	&dev_attr_cntlid.attr,
+	&dev_attr_delete_controller.attr,
+	&dev_attr_transport.attr,
+	&dev_attr_subsysnqn.attr,
+	&dev_attr_address.attr,
+	&dev_attr_state.attr,
+	&dev_attr_numa_node.attr,
+	&dev_attr_queue_count.attr,
+	&dev_attr_sqsize.attr,
+	&dev_attr_hostnqn.attr,
+	&dev_attr_hostid.attr,
+	&dev_attr_ctrl_loss_tmo.attr,
+	&dev_attr_reconnect_delay.attr,
+	&dev_attr_fast_io_fail_tmo.attr,
+	&dev_attr_kato.attr,
+	NULL
+};
+
+static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
+		struct attribute *a, int n)
+{
+	struct device *dev = container_of(kobj, struct device, kobj);
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	if (a == &dev_attr_delete_controller.attr && !ctrl->ops->delete_ctrl)
+		return 0;
+	if (a == &dev_attr_address.attr && !ctrl->ops->get_address)
+		return 0;
+	if (a == &dev_attr_hostnqn.attr && !ctrl->opts)
+		return 0;
+	if (a == &dev_attr_hostid.attr && !ctrl->opts)
+		return 0;
+	if (a == &dev_attr_ctrl_loss_tmo.attr && !ctrl->opts)
+		return 0;
+	if (a == &dev_attr_reconnect_delay.attr && !ctrl->opts)
+		return 0;
+	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
+		return 0;
+
+	return a->mode;
+}
+
+static const struct attribute_group nvme_dev_attrs_group = {
+	.attrs		= nvme_dev_attrs,
+	.is_visible	= nvme_dev_attrs_are_visible,
+};
+
+static const struct attribute_group *nvme_dev_attr_groups[] = {
+	&nvme_dev_attrs_group,
+	NULL,
+};
+
+const struct attribute_group **nvme_get_device_attr_groups(void)
+{
+	return nvme_dev_attr_groups;
+}
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 13 20:20:46 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 13 Jun 2021 20:20:46 -0700
Subject: [PATCH] nvme: move pr code into separate file
Message-ID: <20210614032046.18678-1-chaitanya.kulkarni@wdc.com>

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/Makefile |   2 +-
 drivers/nvme/host/core.c   | 121 -----------------------------------
 drivers/nvme/host/nvme.h   |   2 +
 drivers/nvme/host/pr.c     | 126 +++++++++++++++++++++++++++++++++++++
 4 files changed, 129 insertions(+), 122 deletions(-)
 create mode 100644 drivers/nvme/host/pr.c

diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index cbc509784b2e..8771f27fb49a 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -9,7 +9,7 @@ obj-$(CONFIG_NVME_RDMA)			+= nvme-rdma.o
 obj-$(CONFIG_NVME_FC)			+= nvme-fc.o
 obj-$(CONFIG_NVME_TCP)			+= nvme-tcp.o
 
-nvme-core-y				:= core.o ioctl.o
+nvme-core-y				:= core.o ioctl.o pr.o
 nvme-core-$(CONFIG_TRACING)		+= trace.o
 nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_NVM)			+= lightnvm.o
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index cb1a76a3d42a..6eedb78ca9ab 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -16,11 +16,9 @@
 #include <linux/list_sort.h>
 #include <linux/slab.h>
 #include <linux/types.h>
-#include <linux/pr.h>
 #include <linux/ptrace.h>
 #include <linux/nvme_ioctl.h>
 #include <linux/pm_qos.h>
-#include <asm/unaligned.h>
 
 #include "nvme.h"
 #include "fabrics.h"
@@ -1921,125 +1919,6 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id)
 	return ret;
 }
 
-static char nvme_pr_type(enum pr_type type)
-{
-	switch (type) {
-	case PR_WRITE_EXCLUSIVE:
-		return 1;
-	case PR_EXCLUSIVE_ACCESS:
-		return 2;
-	case PR_WRITE_EXCLUSIVE_REG_ONLY:
-		return 3;
-	case PR_EXCLUSIVE_ACCESS_REG_ONLY:
-		return 4;
-	case PR_WRITE_EXCLUSIVE_ALL_REGS:
-		return 5;
-	case PR_EXCLUSIVE_ACCESS_ALL_REGS:
-		return 6;
-	default:
-		return 0;
-	}
-};
-
-static int nvme_send_ns_head_pr_command(struct block_device *bdev,
-		struct nvme_command *c, u8 data[16])
-{
-	struct nvme_ns_head *head = bdev->bd_disk->private_data;
-	int srcu_idx = srcu_read_lock(&head->srcu);
-	struct nvme_ns *ns = nvme_find_path(head);
-	int ret = -EWOULDBLOCK;
-
-	if (ns) {
-		c->common.nsid = cpu_to_le32(ns->head->ns_id);
-		ret = nvme_submit_sync_cmd(ns->queue, c, data, 16);
-	}
-	srcu_read_unlock(&head->srcu, srcu_idx);
-	return ret;
-}
-	
-static int nvme_send_ns_pr_command(struct nvme_ns *ns, struct nvme_command *c,
-		u8 data[16])
-{
-	c->common.nsid = cpu_to_le32(ns->head->ns_id);
-	return nvme_submit_sync_cmd(ns->queue, c, data, 16);
-}
-
-static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
-				u64 key, u64 sa_key, u8 op)
-{
-	struct nvme_command c;
-	u8 data[16] = { 0, };
-
-	put_unaligned_le64(key, &data[0]);
-	put_unaligned_le64(sa_key, &data[8]);
-
-	memset(&c, 0, sizeof(c));
-	c.common.opcode = op;
-	c.common.cdw10 = cpu_to_le32(cdw10);
-
-	if (IS_ENABLED(CONFIG_NVME_MULTIPATH) &&
-	    bdev->bd_disk->fops == &nvme_ns_head_ops)
-		return nvme_send_ns_head_pr_command(bdev, &c, data);
-	return nvme_send_ns_pr_command(bdev->bd_disk->private_data, &c, data);
-}
-
-static int nvme_pr_register(struct block_device *bdev, u64 old,
-		u64 new, unsigned flags)
-{
-	u32 cdw10;
-
-	if (flags & ~PR_FL_IGNORE_KEY)
-		return -EOPNOTSUPP;
-
-	cdw10 = old ? 2 : 0;
-	cdw10 |= (flags & PR_FL_IGNORE_KEY) ? 1 << 3 : 0;
-	cdw10 |= (1 << 30) | (1 << 31); /* PTPL=1 */
-	return nvme_pr_command(bdev, cdw10, old, new, nvme_cmd_resv_register);
-}
-
-static int nvme_pr_reserve(struct block_device *bdev, u64 key,
-		enum pr_type type, unsigned flags)
-{
-	u32 cdw10;
-
-	if (flags & ~PR_FL_IGNORE_KEY)
-		return -EOPNOTSUPP;
-
-	cdw10 = nvme_pr_type(type) << 8;
-	cdw10 |= ((flags & PR_FL_IGNORE_KEY) ? 1 << 3 : 0);
-	return nvme_pr_command(bdev, cdw10, key, 0, nvme_cmd_resv_acquire);
-}
-
-static int nvme_pr_preempt(struct block_device *bdev, u64 old, u64 new,
-		enum pr_type type, bool abort)
-{
-	u32 cdw10 = nvme_pr_type(type) << 8 | (abort ? 2 : 1);
-
-	return nvme_pr_command(bdev, cdw10, old, new, nvme_cmd_resv_acquire);
-}
-
-static int nvme_pr_clear(struct block_device *bdev, u64 key)
-{
-	u32 cdw10 = 1 | (key ? 1 << 3 : 0);
-
-	return nvme_pr_command(bdev, cdw10, key, 0, nvme_cmd_resv_register);
-}
-
-static int nvme_pr_release(struct block_device *bdev, u64 key, enum pr_type type)
-{
-	u32 cdw10 = nvme_pr_type(type) << 8 | (key ? 1 << 3 : 0);
-
-	return nvme_pr_command(bdev, cdw10, key, 0, nvme_cmd_resv_release);
-}
-
-const struct pr_ops nvme_pr_ops = {
-	.pr_register	= nvme_pr_register,
-	.pr_reserve	= nvme_pr_reserve,
-	.pr_release	= nvme_pr_release,
-	.pr_preempt	= nvme_pr_preempt,
-	.pr_clear	= nvme_pr_clear,
-};
-
 #ifdef CONFIG_BLK_SED_OPAL
 int nvme_sec_submit(void *data, u16 spsp, u8 secp, void *buffer, size_t len,
 		bool send)
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1f397ecba16c..db90b57c4336 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -53,6 +53,8 @@ enum {
 	NVME_NS_LIGHTNVM	= 1,
 };
 
+extern const struct pr_ops nvme_pr_ops;
+
 /*
  * List of workarounds for devices that required behavior not specified in
  * the standard.
diff --git a/drivers/nvme/host/pr.c b/drivers/nvme/host/pr.c
new file mode 100644
index 000000000000..64a3c84d3ef4
--- /dev/null
+++ b/drivers/nvme/host/pr.c
@@ -0,0 +1,126 @@
+#include <linux/blkdev.h>
+#include <linux/pr.h>
+#include <asm/unaligned.h>
+
+#include "nvme.h"
+#include "fabrics.h"
+
+static char nvme_pr_type(enum pr_type type)
+{
+	switch (type) {
+	case PR_WRITE_EXCLUSIVE:
+		return 1;
+	case PR_EXCLUSIVE_ACCESS:
+		return 2;
+	case PR_WRITE_EXCLUSIVE_REG_ONLY:
+		return 3;
+	case PR_EXCLUSIVE_ACCESS_REG_ONLY:
+		return 4;
+	case PR_WRITE_EXCLUSIVE_ALL_REGS:
+		return 5;
+	case PR_EXCLUSIVE_ACCESS_ALL_REGS:
+		return 6;
+	default:
+		return 0;
+	}
+}
+
+static int nvme_send_ns_head_pr_command(struct block_device *bdev,
+		struct nvme_command *c, u8 data[16])
+{
+	struct nvme_ns_head *head = bdev->bd_disk->private_data;
+	int srcu_idx = srcu_read_lock(&head->srcu);
+	struct nvme_ns *ns = nvme_find_path(head);
+	int ret = -EWOULDBLOCK;
+
+	if (ns) {
+		c->common.nsid = cpu_to_le32(ns->head->ns_id);
+		ret = nvme_submit_sync_cmd(ns->queue, c, data, 16);
+	}
+	srcu_read_unlock(&head->srcu, srcu_idx);
+	return ret;
+}
+	
+static int nvme_send_ns_pr_command(struct nvme_ns *ns, struct nvme_command *c,
+		u8 data[16])
+{
+	c->common.nsid = cpu_to_le32(ns->head->ns_id);
+	return nvme_submit_sync_cmd(ns->queue, c, data, 16);
+}
+
+static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
+				u64 key, u64 sa_key, u8 op)
+{
+	struct nvme_command c;
+	u8 data[16] = { 0, };
+
+	put_unaligned_le64(key, &data[0]);
+	put_unaligned_le64(sa_key, &data[8]);
+
+	memset(&c, 0, sizeof(c));
+	c.common.opcode = op;
+	c.common.cdw10 = cpu_to_le32(cdw10);
+
+	if (IS_ENABLED(CONFIG_NVME_MULTIPATH) &&
+	    bdev->bd_disk->fops == &nvme_ns_head_ops)
+		return nvme_send_ns_head_pr_command(bdev, &c, data);
+	return nvme_send_ns_pr_command(bdev->bd_disk->private_data, &c, data);
+}
+
+static int nvme_pr_register(struct block_device *bdev, u64 old,
+		u64 new, unsigned flags)
+{
+	u32 cdw10;
+
+	if (flags & ~PR_FL_IGNORE_KEY)
+		return -EOPNOTSUPP;
+
+	cdw10 = old ? 2 : 0;
+	cdw10 |= (flags & PR_FL_IGNORE_KEY) ? 1 << 3 : 0;
+	cdw10 |= (1 << 30) | (1 << 31); /* PTPL=1 */
+	return nvme_pr_command(bdev, cdw10, old, new, nvme_cmd_resv_register);
+}
+
+static int nvme_pr_reserve(struct block_device *bdev, u64 key,
+		enum pr_type type, unsigned flags)
+{
+	u32 cdw10;
+
+	if (flags & ~PR_FL_IGNORE_KEY)
+		return -EOPNOTSUPP;
+
+	cdw10 = nvme_pr_type(type) << 8;
+	cdw10 |= ((flags & PR_FL_IGNORE_KEY) ? 1 << 3 : 0);
+	return nvme_pr_command(bdev, cdw10, key, 0, nvme_cmd_resv_acquire);
+}
+
+static int nvme_pr_preempt(struct block_device *bdev, u64 old, u64 new,
+		enum pr_type type, bool abort)
+{
+	u32 cdw10 = nvme_pr_type(type) << 8 | (abort ? 2 : 1);
+
+	return nvme_pr_command(bdev, cdw10, old, new, nvme_cmd_resv_acquire);
+}
+
+static int nvme_pr_clear(struct block_device *bdev, u64 key)
+{
+	u32 cdw10 = 1 | (key ? 1 << 3 : 0);
+
+	return nvme_pr_command(bdev, cdw10, key, 0, nvme_cmd_resv_register);
+}
+
+static int nvme_pr_release(struct block_device *bdev, u64 key, enum pr_type type)
+{
+	u32 cdw10 = nvme_pr_type(type) << 8 | (key ? 1 << 3 : 0);
+
+	return nvme_pr_command(bdev, cdw10, key, 0, nvme_cmd_resv_release);
+}
+
+const struct pr_ops nvme_pr_ops = {
+	.pr_register	= nvme_pr_register,
+	.pr_reserve	= nvme_pr_reserve,
+	.pr_release	= nvme_pr_release,
+	.pr_preempt	= nvme_pr_preempt,
+	.pr_clear	= nvme_pr_clear,
+};
+
-- 
2.22.1



From Chaitanya.Kulkarni at wdc.com  Sun Jun 13 20:25:38 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 14 Jun 2021 03:25:38 +0000
Subject: [PATCH  RFC] nvme: move pr code into separate file
References: <20210614032046.18678-1-chaitanya.kulkarni@wdc.com>
Message-ID: <BYAPR04MB4965F9348EEBA142C5050BB086319@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/13/21 20:20, Chaitanya Kulkarni wrote:
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
 OOPs forgot to add RFC in the subject.


From hch at lst.de  Sun Jun 13 22:49:43 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 14 Jun 2021 07:49:43 +0200
Subject: [PATCHv4 2/4] nvme: use blk_execute_rq() for passthrough commands
In-Reply-To: <20210610214437.641245-3-kbusch@kernel.org>
References: <20210610214437.641245-1-kbusch@kernel.org>
 <20210610214437.641245-3-kbusch@kernel.org>
Message-ID: <20210614054943.GA27731@lst.de>

Looks good,

Reviewed-by: Christoph Hellwig <hch at lst.de>


From hch at lst.de  Sun Jun 13 22:50:19 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 14 Jun 2021 07:50:19 +0200
Subject: [PATCHv4 4/4] nvme: use return value from blk_execute_rq()
In-Reply-To: <20210610214437.641245-5-kbusch@kernel.org>
References: <20210610214437.641245-1-kbusch@kernel.org>
 <20210610214437.641245-5-kbusch@kernel.org>
Message-ID: <20210614055019.GB27731@lst.de>

Looks good,

Reviewed-by: Christoph Hellwig <hch at lst.de>


From hch at lst.de  Sun Jun 13 23:36:29 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 14 Jun 2021 08:36:29 +0200
Subject: [PATCH 1/3] nvmet: use subsys->max_nsid to report MNAN
In-Reply-To: <20210613200252.9167-2-chaitanya.kulkarni@wdc.com>
References: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
 <20210613200252.9167-2-chaitanya.kulkarni@wdc.com>
Message-ID: <20210614063629.GA29129@lst.de>

On Sun, Jun 13, 2021 at 01:02:50PM -0700, Chaitanya Kulkarni wrote:
> For Spec regarding MNAN value:-
> 
> If the controller supports Asymmetric Namespace Access Reporting, then
> this field shall be set to a non-zero value that is less than or equal
> to the NN value.
> 
> Instead of using NVMET_MAX_NAMESPACES use subsys->max_nsid to set the
> MNAN value.

I think this is a rather bad idea as it is supposed to be a relatively
static number.  I think the right fix is to stop playing games with a
dyanmic NN and always report UINT_MAX in nn.


From dwagner at suse.de  Mon Jun 14 04:21:26 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Mon, 14 Jun 2021 13:21:26 +0200
Subject: [PATCH 1/1] nvmet: use subsys->max_nsid to report MNAN
In-Reply-To: <20210614013754.37940-2-chaitanya.kulkarni@wdc.com>
References: <20210614013754.37940-1-chaitanya.kulkarni@wdc.com>
 <20210614013754.37940-2-chaitanya.kulkarni@wdc.com>
Message-ID: <20210614112126.aybtimgev7okdkvu@beryllium.lan>

On Sun, Jun 13, 2021 at 06:37:54PM -0700, Chaitanya Kulkarni wrote:
> For Spec regarding MNAN value:-
> 
> If the controller supports Asymmetric Namespace Access Reporting, then
> this field shall be set to a non-zero value that is less than or equal
> to the NN value.
> 
> Instead of using NVMET_MAX_NAMESPACES use subsys->max_nsid to set the
> MNAN value.
> 
> Without this patch it results in the following error :- 
> 
> [337976.409142] nvme nvme1: Invalid MNAN value 1024

Is this not the very same patch on which Christoph commented?


From geert+renesas at glider.be  Mon Jun 14 07:16:07 2021
From: geert+renesas at glider.be (Geert Uytterhoeven)
Date: Mon, 14 Jun 2021 16:16:07 +0200
Subject: [PATCH trivial] nvme: Grammar s/each namespaces/each namespace/
Message-ID: <d195ebc405033c1de6ca01937a906afc97f285f7.1623680137.git.geert+renesas@glider.be>

Fix a singular/plural mismatch in the CONFIG_NVME_MULTIPATH help text.

Signed-off-by: Geert Uytterhoeven <geert+renesas at glider.be>
---
 drivers/nvme/host/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index f76cc4690bfc37bc..26ddd34ae45b76fc 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -21,7 +21,7 @@ config NVME_MULTIPATH
 	help
 	   This option enables support for multipath access to NVMe
 	   subsystems.  If this option is enabled only a single
-	   /dev/nvmeXnY device will show up for each NVMe namespaces,
+	   /dev/nvmeXnY device will show up for each NVMe namespace,
 	   even if it is accessible through multiple controllers.
 
 config NVME_HWMON
-- 
2.25.1



From mkalderon at marvell.com  Mon Jun 14 07:44:51 2021
From: mkalderon at marvell.com (Michal Kalderon)
Date: Mon, 14 Jun 2021 14:44:51 +0000
Subject: [EXT] Re: BUG: scheduling while atomic when
 nvmet_rdma_queue_response fails in posting a request
In-Reply-To: <4a031bc0-fba3-12d6-428f-a378aba1897e@grimberg.me>
References: <CH0PR18MB4129247C2B9AE092E3C59740A1209@CH0PR18MB4129.namprd18.prod.outlook.com>
 <e059fc90-81e7-dbe8-86f4-87b4bdbd5bb0@grimberg.me>
 <20210608184134.GA339600@dhcp-10-100-145-180.wdc.com>
 <4a031bc0-fba3-12d6-428f-a378aba1897e@grimberg.me>
Message-ID: <CH0PR18MB412996A6716DE9AA114C09B0A1319@CH0PR18MB4129.namprd18.prod.outlook.com>

> From: Sagi Grimberg <sagi at grimberg.me>
> Sent: Wednesday, June 9, 2021 3:04 AM
> 
> ----------------------------------------------------------------------
> 
> >> diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
> >> index 7d607f435e36..6d2eea322779 100644
> >> --- a/drivers/nvme/target/rdma.c
> >> +++ b/drivers/nvme/target/rdma.c
> >> @@ -16,6 +16,7 @@
> >>   #include <linux/wait.h>
> >>   #include <linux/inet.h>
> >>   #include <asm/unaligned.h>
> >> +#include <linux/async.h>
> >>
> >>   #include <rdma/ib_verbs.h>
> >>   #include <rdma/rdma_cm.h>
> >> @@ -712,6 +713,12 @@ static void nvmet_rdma_send_done(struct ib_cq
> *cq,
> >> struct ib_wc *wc)
> >>          }
> >>   }
> >>
> >> +static void nvmet_rdma_async_release_rsp(void *data, async_cookie_t
> cookie)
> >> +{
> >> +       struct nvmet_rdma_rsp *rsp = data;
> >> +       nvmet_rdma_release_rsp(rsp);
> >> +}
> >> +
> >>   static void nvmet_rdma_queue_response(struct nvmet_req *req)
> >>   {
> >>          struct nvmet_rdma_rsp *rsp =
> >> @@ -745,7 +752,12 @@ static void nvmet_rdma_queue_response(struct
> nvmet_req
> >> *req)
> >>
> >>          if (unlikely(ib_post_send(cm_id->qp, first_wr, NULL))) {
> >>                  pr_err("sending cmd response failed\n");
> >> -               nvmet_rdma_release_rsp(rsp);
> >> +               /*
> >> +                * We might be in atomic context, hence release
> >> +                * the rsp in async context in case we need to
> >> +                * process the wr_wait_list.
> >> +                */
> >> +               async_schedule(nvmet_rdma_async_release_rsp, rsp);
> >>          }
> >>   }
> >
> > Just FYI, async_schedule() has conditions where it may execute your
> > callback synchronously. Your suggestion is probably fine for testing,
> > but it sounds like you require something that can guarantee a non-atomic
> > context for nvmet_rdma_release_rsp().
> 
> OK, it seems that the issue is that we are submitting I/O in atomic
> context. This should be more appropriate...

Thanks Sagi, this seems to work. I'm still hitting some other issues where in some cases reconnect fails, but I'm 
Collecting more info. 

> 
> --
> diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
> index 7d607f435e36..16f2f5a84ae7 100644
> --- a/drivers/nvme/target/rdma.c
> +++ b/drivers/nvme/target/rdma.c
> @@ -102,6 +102,7 @@ struct nvmet_rdma_queue {
> 
>          struct work_struct      release_work;
>          struct list_head        rsp_wait_list;
> +       struct work_struct      wr_wait_work;
>          struct list_head        rsp_wr_wait_list;
>          spinlock_t              rsp_wr_wait_lock;
> 
> @@ -517,8 +518,10 @@ static int nvmet_rdma_post_recv(struct
> nvmet_rdma_device *ndev,
>          return ret;
>   }
> 
> -static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue
> *queue)
> +static void nvmet_rdma_process_wr_wait_list(struct work_struct *w)
>   {
> +       struct nvmet_rdma_queue *queue =
> +               container_of(w, struct nvmet_rdma_queue, wr_wait_work);
>          spin_lock(&queue->rsp_wr_wait_lock);
>          while (!list_empty(&queue->rsp_wr_wait_list)) {
>                  struct nvmet_rdma_rsp *rsp;
> @@ -677,7 +680,7 @@ static void nvmet_rdma_release_rsp(struct
> nvmet_rdma_rsp *rsp)
>                  nvmet_req_free_sgls(&rsp->req);
> 
>          if (unlikely(!list_empty_careful(&queue->rsp_wr_wait_list)))
> -               nvmet_rdma_process_wr_wait_list(queue);
> +               schedule_work(&queue->wr_wait_work);
> 
>          nvmet_rdma_put_rsp(rsp);
>   }
> @@ -1446,6 +1449,7 @@ nvmet_rdma_alloc_queue(struct
> nvmet_rdma_device *ndev,
>           * inside a CM callback would trigger a deadlock. (great API
> design..)
>           */
>          INIT_WORK(&queue->release_work,
> nvmet_rdma_release_queue_work);
> +       INIT_WORK(&queue->wr_wait_work,
> nvmet_rdma_process_wr_wait_list);
>          queue->dev = ndev;
>          queue->cm_id = cm_id;
>          queue->port = port->nport;
> --
Thanks,?

Tested-by: Michal Kalderon?<michal.kalderon at marvell.com>



From sagi at grimberg.me  Mon Jun 14 09:44:42 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 14 Jun 2021 09:44:42 -0700
Subject: [EXT] Re: BUG: scheduling while atomic when
 nvmet_rdma_queue_response fails in posting a request
In-Reply-To: <CH0PR18MB412996A6716DE9AA114C09B0A1319@CH0PR18MB4129.namprd18.prod.outlook.com>
References: <CH0PR18MB4129247C2B9AE092E3C59740A1209@CH0PR18MB4129.namprd18.prod.outlook.com>
 <e059fc90-81e7-dbe8-86f4-87b4bdbd5bb0@grimberg.me>
 <20210608184134.GA339600@dhcp-10-100-145-180.wdc.com>
 <4a031bc0-fba3-12d6-428f-a378aba1897e@grimberg.me>
 <CH0PR18MB412996A6716DE9AA114C09B0A1319@CH0PR18MB4129.namprd18.prod.outlook.com>
Message-ID: <4b77d39b-00ba-317a-658b-b00be6cb3e20@grimberg.me>


>> OK, it seems that the issue is that we are submitting I/O in atomic
>> context. This should be more appropriate...
> 
> Thanks Sagi, this seems to work. I'm still hitting some other issues where in some cases reconnect fails, but I'm
> Collecting more info.

Same type of failures?


From mkalderon at marvell.com  Mon Jun 14 11:14:55 2021
From: mkalderon at marvell.com (Michal Kalderon)
Date: Mon, 14 Jun 2021 18:14:55 +0000
Subject: [EXT] Re: BUG: scheduling while atomic when
 nvmet_rdma_queue_response fails in posting a request
In-Reply-To: <4b77d39b-00ba-317a-658b-b00be6cb3e20@grimberg.me>
References: <CH0PR18MB4129247C2B9AE092E3C59740A1209@CH0PR18MB4129.namprd18.prod.outlook.com>
 <e059fc90-81e7-dbe8-86f4-87b4bdbd5bb0@grimberg.me>
 <20210608184134.GA339600@dhcp-10-100-145-180.wdc.com>
 <4a031bc0-fba3-12d6-428f-a378aba1897e@grimberg.me>
 <CH0PR18MB412996A6716DE9AA114C09B0A1319@CH0PR18MB4129.namprd18.prod.outlook.com>
 <4b77d39b-00ba-317a-658b-b00be6cb3e20@grimberg.me>
Message-ID: <CH0PR18MB4129DEE104D73DEB0BD9B01EA1319@CH0PR18MB4129.namprd18.prod.outlook.com>

> From: Sagi Grimberg <sagi at grimberg.me>
> Sent: Monday, June 14, 2021 7:45 PM
> 
> 
> >> OK, it seems that the issue is that we are submitting I/O in atomic
> >> context. This should be more appropriate...
> >
> > Thanks Sagi, this seems to work. I'm still hitting some other issues where in
> some cases reconnect fails, but I'm
> > Collecting more info.
> 
> Same type of failures?
No, something else. 
After recovery completes, I'm getting the following errors on initiator side without any messages on target: 
[14678.618025] nvme nvme2: Connect rejected: status -104 (reset by remote host).
[14678.619350] nvme nvme2: rdma connection establishment failed (-104)
[14678.622274] nvme nvme2: Failed reconnect attempt 6
[14678.623623] nvme nvme2: Reconnecting in 10 seconds...
[14751.304247] nvme nvme2: I/O 0 QID 0 timeout
[14751.305749] nvme nvme2: Connect command failed, error wo/DNR bit: 881
[14751.307240] nvme nvme2: failed to connect queue: 0 ret=881
[14751.310497] nvme nvme2: Failed reconnect attempt 7
[14751.312174] nvme nvme2: Reconnecting in 10 seconds...
[14825.032645] nvme nvme2: I/O 1 QID 0 timeout



From Chaitanya.Kulkarni at wdc.com  Mon Jun 14 12:16:07 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 14 Jun 2021 19:16:07 +0000
Subject: [PATCH 1/1] nvmet: use subsys->max_nsid to report MNAN
References: <20210614013754.37940-1-chaitanya.kulkarni@wdc.com>
 <20210614013754.37940-2-chaitanya.kulkarni@wdc.com>
 <20210614112126.aybtimgev7okdkvu@beryllium.lan>
Message-ID: <BYAPR04MB496580065E9548EEFCBA581386319@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/14/21 04:21, Daniel Wagner wrote:
> On Sun, Jun 13, 2021 at 06:37:54PM -0700, Chaitanya Kulkarni wrote:
>> For Spec regarding MNAN value:-
>>
>> If the controller supports Asymmetric Namespace Access Reporting, then
>> this field shall be set to a non-zero value that is less than or equal
>> to the NN value.
>>
>> Instead of using NVMET_MAX_NAMESPACES use subsys->max_nsid to set the
>> MNAN value.
>>
>> Without this patch it results in the following error :- 
>>
>> [337976.409142] nvme nvme1: Invalid MNAN value 1024
> Is this not the very same patch on which Christoph commented?
>

Yes without the rest of the patches.




From chaitanya.kulkarni at wdc.com  Mon Jun 14 12:47:12 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 14 Jun 2021 12:47:12 -0700
Subject: [PATCH] nvmet: use U32_MAX value to report NN
Message-ID: <20210614194712.622-1-chaitanya.kulkarni@wdc.com>

For Spec regarding MNAN value:-

If the controller supports Asymmetric Namespace Access Reporting, then
this field shall be set to a non-zero value that is less than or equal
to the NN value.

Instead of using subsys->max_nsid that gets calculated dynamically,
use U32_MAX value to report NN. This way we will maintain the MNAN value
spec compliant.

Without this patch, code results in the following error :-

[337976.409142] nvme nvme1: Invalid MNAN value 1024

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index cd60a8184d04..c8c4a7ba4958 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -393,7 +393,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	/* no enforcement soft-limit for maxcmd - pick arbitrary high value */
 	id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
 
-	id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
+	id->nn = cpu_to_le32(U32_MAX);
 	id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
 	id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
 			NVME_CTRL_ONCS_WRITE_ZEROES);
-- 
2.22.1



From kbusch at kernel.org  Mon Jun 14 14:18:40 2021
From: kbusch at kernel.org (Keith Busch)
Date: Mon, 14 Jun 2021 14:18:40 -0700
Subject: [PATCH] nvmet: use U32_MAX value to report NN
In-Reply-To: <20210614194712.622-1-chaitanya.kulkarni@wdc.com>
References: <20210614194712.622-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210614211840.GA646237@dhcp-10-100-145-180.wdc.com>

On Mon, Jun 14, 2021 at 12:47:12PM -0700, Chaitanya Kulkarni wrote:
> @@ -393,7 +393,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
>  	/* no enforcement soft-limit for maxcmd - pick arbitrary high value */
>  	id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
>  
> -	id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
> +	id->nn = cpu_to_le32(U32_MAX);

The NN field has to have the highest valid NSID, and FFFFFFFFh is not
valid. That's reserved for the broadcast.


From Chaitanya.Kulkarni at wdc.com  Mon Jun 14 14:19:18 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 14 Jun 2021 21:19:18 +0000
Subject: [PATCH 1/3] nvmet: use subsys->max_nsid to report MNAN
References: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
 <20210613200252.9167-2-chaitanya.kulkarni@wdc.com>
 <20210614063629.GA29129@lst.de>
Message-ID: <BYAPR04MB49651D977E8A8593309CE56E86319@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/13/21 23:36, Christoph Hellwig wrote:
> On Sun, Jun 13, 2021 at 01:02:50PM -0700, Chaitanya Kulkarni wrote:
>> For Spec regarding MNAN value:-
>>
>> If the controller supports Asymmetric Namespace Access Reporting, then
>> this field shall be set to a non-zero value that is less than or equal
>> to the NN value.
>>
>> Instead of using NVMET_MAX_NAMESPACES use subsys->max_nsid to set the
>> MNAN value.
> I think this is a rather bad idea as it is supposed to be a relatively
> static number.  I think the right fix is to stop playing games with a
> dyanmic NN and always report UINT_MAX in nn.
>

Sent out the patch with NN set to U32_MAX.



From Chaitanya.Kulkarni at wdc.com  Mon Jun 14 14:23:07 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 14 Jun 2021 21:23:07 +0000
Subject: [PATCH] nvmet: use U32_MAX value to report NN
References: <20210614194712.622-1-chaitanya.kulkarni@wdc.com>
 <20210614211840.GA646237@dhcp-10-100-145-180.wdc.com>
Message-ID: <BYAPR04MB496590CDFF6E7589895BE0F186319@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/14/21 14:19, Keith Busch wrote:
> On Mon, Jun 14, 2021 at 12:47:12PM -0700, Chaitanya Kulkarni wrote:
>> @@ -393,7 +393,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
>>  	/* no enforcement soft-limit for maxcmd - pick arbitrary high value */
>>  	id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
>>  
>> -	id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
>> +	id->nn = cpu_to_le32(U32_MAX);
> The NN field has to have the highest valid NSID, and FFFFFFFFh is not
> valid. That's reserved for the broadcast.
>

In that case we have to keep the dynamic id setting for the MNAN and
may have to keep the original proposed fix :-

 drivers/nvme/target/admin-cmd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/target/admin-cmd.c
b/drivers/nvme/target/admin-cmd.c
index cd60a8184d04..a8ec377bb68d 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -394,7 +394,7 @@ static void nvmet_execute_identify_ctrl(struct
nvmet_req *req)
        id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
 
        id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
-       id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
+       id->mnan = cpu_to_le32(ctrl->subsys->max_nsid);
        id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
                        NVME_CTRL_ONCS_WRITE_ZEROES);
 
-- 
2.22.1




From hch at lst.de  Tue Jun 15 06:10:34 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:34 +0200
Subject: [PATCH 16/16] nvme-multipath: enable polled I/O
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-17-hch@lst.de>

Set the poll queue flag to enable polling, given that the multipath
node just dispatches the bios to a lower queue.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 drivers/nvme/host/multipath.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index f77b774699de..48ea649d6c24 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -450,6 +450,15 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
 
 	blk_queue_flag_set(QUEUE_FLAG_NONROT, head->disk->queue);
 	blk_queue_flag_set(QUEUE_FLAG_NOWAIT, head->disk->queue);
+	/*
+	 * This assumes all controllers that refer to a namespace either
+	 * support poll queues or not.  That is not a strict guarantee,
+	 * but if the assumption is wrong the effect is only suboptimal
+	 * performance but not correctness problem.
+	 */
+	if (ctrl->tagset->nr_maps > HCTX_TYPE_POLL &&
+	    ctrl->tagset->map[HCTX_TYPE_POLL].nr_queues)
+		blk_queue_flag_set(QUEUE_FLAG_POLL, head->disk->queue);
 
 	/* set to a default value of 512 until the disk is validated */
 	blk_queue_logical_block_size(head->disk->queue, 512);
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:33 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:33 +0200
Subject: [PATCH 15/16] nvme-multipath: set QUEUE_FLAG_NOWAIT
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-16-hch@lst.de>

The nvme multipathing code just dispatches bios to one of the blk-mq
based paths and never blocks on its own, so set QUEUE_FLAG_NOWAIT
to support REQ_NOWAIT bios.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 drivers/nvme/host/multipath.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index c1b4d627c05b..f77b774699de 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -449,6 +449,8 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
 			ctrl->subsys->instance, head->instance);
 
 	blk_queue_flag_set(QUEUE_FLAG_NONROT, head->disk->queue);
+	blk_queue_flag_set(QUEUE_FLAG_NOWAIT, head->disk->queue);
+
 	/* set to a default value of 512 until the disk is validated */
 	blk_queue_logical_block_size(head->disk->queue, 512);
 	blk_set_stacking_limits(&head->disk->queue->limits);
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:32 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:32 +0200
Subject: [PATCH 14/16] block: don't allow writing to the poll queue attribute
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-15-hch@lst.de>

The poll attribute is a historic artefact from before when we had
explicit poll queues that require driver specific configuration.
Just print a warning when writing to the attribute.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 block/blk-sysfs.c | 23 ++++-------------------
 1 file changed, 4 insertions(+), 19 deletions(-)

diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index f89e2fc3963b..f78e73ca6091 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -428,26 +428,11 @@ static ssize_t queue_poll_show(struct request_queue *q, char *page)
 static ssize_t queue_poll_store(struct request_queue *q, const char *page,
 				size_t count)
 {
-	unsigned long poll_on;
-	ssize_t ret;
-
-	if (!q->tag_set || q->tag_set->nr_maps <= HCTX_TYPE_POLL ||
-	    !q->tag_set->map[HCTX_TYPE_POLL].nr_queues)
+	if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
 		return -EINVAL;
-
-	ret = queue_var_store(&poll_on, page, count);
-	if (ret < 0)
-		return ret;
-
-	if (poll_on) {
-		blk_queue_flag_set(QUEUE_FLAG_POLL, q);
-	} else {
-		blk_mq_freeze_queue(q);
-		blk_queue_flag_clear(QUEUE_FLAG_POLL, q);
-		blk_mq_unfreeze_queue(q);
-	}
-
-	return ret;
+	pr_info_ratelimited("writes to the poll attribute are ignored.\n");
+	pr_info_ratelimited("please use driver specific parameters instead.\n");
+	return count;
 }
 
 static ssize_t queue_io_timeout_show(struct request_queue *q, char *page)
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:31 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:31 +0200
Subject: [PATCH 13/16] block: switch polling to be bio based
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-14-hch@lst.de>

Replace the blk_poll interface that requires the caller to keep a queue
and cookie from the submissions with polling based on the bio.

Polling for the bio itself leads to a few advantages:

 - the cookie construction can made entirely private in blk-mq.c
 - the caller does not need to remember the request_queue and cookie
   separately and thus sidesteps their lifetime issues
 - keeping the device and the cookie inside the bio allows to trivially
   support polling BIOs remapping by stacking drivers
 - a lot of code to propagate the cookie back up the submission path can
   be removed entirely.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 arch/m68k/emu/nfblock.c             |   3 +-
 arch/xtensa/platforms/iss/simdisk.c |   3 +-
 block/bio.c                         |   1 +
 block/blk-core.c                    | 127 +++++++++++++++++++++-------
 block/blk-mq.c                      |  73 +++++-----------
 block/blk-mq.h                      |   2 +
 drivers/block/brd.c                 |  12 ++-
 drivers/block/drbd/drbd_int.h       |   2 +-
 drivers/block/drbd/drbd_req.c       |   3 +-
 drivers/block/n64cart.c             |  12 ++-
 drivers/block/null_blk/main.c       |   3 +-
 drivers/block/pktcdvd.c             |   7 +-
 drivers/block/ps3vram.c             |   6 +-
 drivers/block/rsxx/dev.c            |   7 +-
 drivers/block/zram/zram_drv.c       |  10 +--
 drivers/lightnvm/pblk-init.c        |   6 +-
 drivers/md/bcache/request.c         |  13 ++-
 drivers/md/bcache/request.h         |   4 +-
 drivers/md/dm.c                     |  28 +++---
 drivers/md/md.c                     |  10 +--
 drivers/nvdimm/blk.c                |   5 +-
 drivers/nvdimm/btt.c                |   5 +-
 drivers/nvdimm/pmem.c               |   3 +-
 drivers/nvme/host/core.c            |   2 +-
 drivers/nvme/host/multipath.c       |   6 +-
 drivers/s390/block/dcssblk.c        |   7 +-
 drivers/s390/block/xpram.c          |   5 +-
 fs/block_dev.c                      |  25 ++----
 fs/btrfs/inode.c                    |   8 +-
 fs/ext4/file.c                      |   2 +-
 fs/gfs2/file.c                      |   4 +-
 fs/iomap/direct-io.c                |  39 +++------
 fs/xfs/xfs_file.c                   |   2 +-
 fs/zonefs/super.c                   |   2 +-
 include/linux/bio.h                 |   2 +-
 include/linux/blk-mq.h              |  15 +---
 include/linux/blk_types.h           |  12 ++-
 include/linux/blkdev.h              |   8 +-
 include/linux/fs.h                  |   6 +-
 include/linux/iomap.h               |   3 +-
 mm/page_io.c                        |   8 +-
 41 files changed, 230 insertions(+), 271 deletions(-)

diff --git a/arch/m68k/emu/nfblock.c b/arch/m68k/emu/nfblock.c
index 9a8394e96388..4ef457ba5220 100644
--- a/arch/m68k/emu/nfblock.c
+++ b/arch/m68k/emu/nfblock.c
@@ -58,7 +58,7 @@ struct nfhd_device {
 	struct gendisk *disk;
 };
 
-static blk_qc_t nfhd_submit_bio(struct bio *bio)
+static void nfhd_submit_bio(struct bio *bio)
 {
 	struct nfhd_device *dev = bio->bi_bdev->bd_disk->private_data;
 	struct bio_vec bvec;
@@ -76,7 +76,6 @@ static blk_qc_t nfhd_submit_bio(struct bio *bio)
 		sec += len;
 	}
 	bio_endio(bio);
-	return BLK_QC_T_NONE;
 }
 
 static int nfhd_getgeo(struct block_device *bdev, struct hd_geometry *geo)
diff --git a/arch/xtensa/platforms/iss/simdisk.c b/arch/xtensa/platforms/iss/simdisk.c
index 3cdfa00738e0..ddd1fe3db474 100644
--- a/arch/xtensa/platforms/iss/simdisk.c
+++ b/arch/xtensa/platforms/iss/simdisk.c
@@ -100,7 +100,7 @@ static void simdisk_transfer(struct simdisk *dev, unsigned long sector,
 	spin_unlock(&dev->lock);
 }
 
-static blk_qc_t simdisk_submit_bio(struct bio *bio)
+static void simdisk_submit_bio(struct bio *bio)
 {
 	struct simdisk *dev = bio->bi_bdev->bd_disk->private_data;
 	struct bio_vec bvec;
@@ -118,7 +118,6 @@ static blk_qc_t simdisk_submit_bio(struct bio *bio)
 	}
 
 	bio_endio(bio);
-	return BLK_QC_T_NONE;
 }
 
 static int simdisk_open(struct block_device *bdev, fmode_t mode)
diff --git a/block/bio.c b/block/bio.c
index de5505d7018e..986908cc99d4 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -250,6 +250,7 @@ void bio_init(struct bio *bio, struct bio_vec *table,
 	memset(bio, 0, sizeof(*bio));
 	atomic_set(&bio->__bi_remaining, 1);
 	atomic_set(&bio->__bi_cnt, 1);
+	bio->bi_cookie = BLK_QC_T_NONE;
 
 	bio->bi_io_vec = table;
 	bio->bi_max_vecs = max_vecs;
diff --git a/block/blk-core.c b/block/blk-core.c
index 858a76858cb0..531176578221 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -909,18 +909,18 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio)
 	return false;
 }
 
-static blk_qc_t __submit_bio(struct bio *bio)
+static void __submit_bio(struct bio *bio)
 {
 	struct gendisk *disk = bio->bi_bdev->bd_disk;
-	blk_qc_t ret = BLK_QC_T_NONE;
 
 	if (blk_crypto_bio_prep(&bio)) {
-		if (!disk->fops->submit_bio)
-			return blk_mq_submit_bio(bio);
-		ret = disk->fops->submit_bio(bio);
+		if (!disk->fops->submit_bio) {
+			blk_mq_submit_bio(bio);
+			return;
+		}
+		disk->fops->submit_bio(bio);
 	}
 	blk_queue_exit(disk->queue);
-	return ret;
 }
 
 /*
@@ -942,10 +942,9 @@ static blk_qc_t __submit_bio(struct bio *bio)
  * bio_list_on_stack[1] contains bios that were submitted before the current
  *	->submit_bio_bio, but that haven't been processed yet.
  */
-static blk_qc_t __submit_bio_noacct(struct bio *bio)
+static void __submit_bio_noacct(struct bio *bio)
 {
 	struct bio_list bio_list_on_stack[2];
-	blk_qc_t ret = BLK_QC_T_NONE;
 
 	BUG_ON(bio->bi_next);
 
@@ -965,7 +964,7 @@ static blk_qc_t __submit_bio_noacct(struct bio *bio)
 		bio_list_on_stack[1] = bio_list_on_stack[0];
 		bio_list_init(&bio_list_on_stack[0]);
 
-		ret = __submit_bio(bio);
+		__submit_bio(bio);
 
 		/*
 		 * Sort new bios into those for a lower level and those for the
@@ -988,13 +987,11 @@ static blk_qc_t __submit_bio_noacct(struct bio *bio)
 	} while ((bio = bio_list_pop(&bio_list_on_stack[0])));
 
 	current->bio_list = NULL;
-	return ret;
 }
 
-static blk_qc_t __submit_bio_noacct_mq(struct bio *bio)
+static void __submit_bio_noacct_mq(struct bio *bio)
 {
 	struct bio_list bio_list[2] = { };
-	blk_qc_t ret = BLK_QC_T_NONE;
 
 	current->bio_list = bio_list;
 
@@ -1006,15 +1003,13 @@ static blk_qc_t __submit_bio_noacct_mq(struct bio *bio)
 
 		if (!blk_crypto_bio_prep(&bio)) {
 			blk_queue_exit(disk->queue);
-			ret = BLK_QC_T_NONE;
 			continue;
 		}
 
-		ret = blk_mq_submit_bio(bio);
+		blk_mq_submit_bio(bio);
 	} while ((bio = bio_list_pop(&bio_list[0])));
 
 	current->bio_list = NULL;
-	return ret;
 }
 
 /**
@@ -1026,10 +1021,10 @@ static blk_qc_t __submit_bio_noacct_mq(struct bio *bio)
  * systems and other upper level users of the block layer should use
  * submit_bio() instead.
  */
-blk_qc_t submit_bio_noacct(struct bio *bio)
+void submit_bio_noacct(struct bio *bio)
 {
 	if (!submit_bio_checks(bio))
-		return BLK_QC_T_NONE;
+		return;
 
 	/*
 	 * We only want one ->submit_bio to be active at a time, else stack
@@ -1037,14 +1032,12 @@ blk_qc_t submit_bio_noacct(struct bio *bio)
 	 * to collect a list of requests submited by a ->submit_bio method while
 	 * it is active, and then process them after it returned.
 	 */
-	if (current->bio_list) {
+	if (current->bio_list)
 		bio_list_add(&current->bio_list[0], bio);
-		return BLK_QC_T_NONE;
-	}
-
-	if (!bio->bi_bdev->bd_disk->fops->submit_bio)
-		return __submit_bio_noacct_mq(bio);
-	return __submit_bio_noacct(bio);
+	else if (!bio->bi_bdev->bd_disk->fops->submit_bio)
+		__submit_bio_noacct_mq(bio);
+	else
+		__submit_bio_noacct(bio);
 }
 EXPORT_SYMBOL(submit_bio_noacct);
 
@@ -1061,10 +1054,10 @@ EXPORT_SYMBOL(submit_bio_noacct);
  * in @bio.  The bio must NOT be touched by thecaller until ->bi_end_io() has
  * been called.
  */
-blk_qc_t submit_bio(struct bio *bio)
+void submit_bio(struct bio *bio)
 {
 	if (blkcg_punt_bio_submit(bio))
-		return BLK_QC_T_NONE;
+		return;
 
 	/*
 	 * If it's a regular read/write or a barrier with data attached,
@@ -1096,19 +1089,91 @@ blk_qc_t submit_bio(struct bio *bio)
 	if (unlikely(bio_op(bio) == REQ_OP_READ &&
 	    bio_flagged(bio, BIO_WORKINGSET))) {
 		unsigned long pflags;
-		blk_qc_t ret;
 
 		psi_memstall_enter(&pflags);
-		ret = submit_bio_noacct(bio);
+		submit_bio_noacct(bio);
 		psi_memstall_leave(&pflags);
-
-		return ret;
+		return;
 	}
 
-	return submit_bio_noacct(bio);
+	submit_bio_noacct(bio);
 }
 EXPORT_SYMBOL(submit_bio);
 
+/**
+ * bio_poll - poll for BIO completions
+ * @bio: bio to poll for
+ * @flags: BLK_POLL_* flags that control the behavior
+ *
+ * Poll for completions on queue associated with the bio. Returns number of
+ * completed entries found.
+ *
+ * Note: the caller must either be the context that submitted @bio, or
+ * be in a RCU critical section to prevent freeing of @bio.
+ */
+int bio_poll(struct bio *bio, unsigned int flags)
+{
+	struct request_queue *q = bio->bi_bdev->bd_disk->queue;
+	blk_qc_t cookie = READ_ONCE(bio->bi_cookie);
+	int ret;
+
+	if (cookie == BLK_QC_T_NONE ||
+	    !test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
+		return 0;
+
+	if (current->plug)
+		blk_flush_plug_list(current->plug, false);
+
+	if (blk_queue_enter(q, BLK_MQ_REQ_NOWAIT))
+		return 0;
+	if (WARN_ON_ONCE(!queue_is_mq(q)))
+		ret = 0;	/* not yet implemented, should not happen */
+	else
+		ret = blk_mq_poll(q, cookie, flags);
+	blk_queue_exit(q);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(bio_poll);
+
+/*
+ * Helper to implement file_operations.iopoll.  Requires the bio to be stored
+ * in iocb->private, and cleared before freeing the bio.
+ */
+int iocb_bio_iopoll(struct kiocb *kiocb, unsigned int flags)
+{
+	struct bio *bio;
+	int ret = 0;
+
+	/*
+	 * Note: the bio cache only uses SLAB_TYPESAFE_BY_RCU, so bio can
+	 * point to a freshly allocated bio at this point.  If that happens
+	 * we have a few cases to consider:
+	 *
+	 *  1) the bio is beeing initialized and bi_bdev is NULL.  We can just
+	 *     simply nothing in this case
+	 *  2) the bio points to a not poll enabled device.  bio_poll will catch
+	 *     this and return 0
+	 *  3) the bio points to a poll capable device, including but not
+	 *     limited to the one that the original bio pointed to.  In this
+	 *     case we will call into the actual poll method and poll for I/O,
+	 *     even if we don't need to, but it won't cause harm either.
+	 *
+	 * For cases 2) and 3) above the RCU grace period ensures that bi_bdev
+	 * is still allocated. Because partitions hold a reference to the whole
+	 * device bdev and thus disk, the disk is also still valid.  Grabbing
+	 * a reference to the queue in bio_poll() ensures the hctxs and requests
+	 * are still valid as well.
+	 */
+	rcu_read_lock();
+	bio = READ_ONCE(kiocb->private);
+	if (bio && bio->bi_bdev)
+		ret = bio_poll(bio, flags);
+	rcu_read_unlock();
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iocb_bio_iopoll);
+
 /**
  * blk_cloned_rq_check_limits - Helper function to check a cloned request
  *                              for the new queue limits
diff --git a/block/blk-mq.c b/block/blk-mq.c
index a0cc1eedc15a..df5dc3b756f5 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -63,6 +63,9 @@ static int blk_mq_poll_stats_bkt(const struct request *rq)
 	return bucket;
 }
 
+#define BLK_QC_T_SHIFT		16
+#define BLK_QC_T_INTERNAL	(1U << 31)
+
 static inline struct blk_mq_hw_ctx *blk_qc_to_hctx(struct request_queue *q,
 		blk_qc_t qc)
 {
@@ -79,6 +82,13 @@ static inline struct request *blk_qc_to_rq(struct blk_mq_hw_ctx *hctx,
 	return blk_mq_tag_to_rq(hctx->tags, tag);
 }
 
+static inline blk_qc_t blk_rq_to_qc(struct request *rq)
+{
+	return (rq->mq_hctx->queue_num << BLK_QC_T_SHIFT) |
+		(rq->tag != -1 ?
+		 rq->tag : (rq->internal_tag | BLK_QC_T_INTERNAL));
+}
+
 /*
  * Check if any of the ctx, dispatch list or elevator
  * have pending work in this hardware queue.
@@ -747,6 +757,8 @@ void blk_mq_start_request(struct request *rq)
 	if (blk_integrity_rq(rq) && req_op(rq) == REQ_OP_WRITE)
 		q->integrity.profile->prepare_fn(rq);
 #endif
+	if (rq->bio && rq->bio->bi_opf & REQ_POLLED)
+	        WRITE_ONCE(rq->bio->bi_cookie, blk_rq_to_qc(rq));
 }
 EXPORT_SYMBOL(blk_mq_start_request);
 
@@ -1996,19 +2008,15 @@ static void blk_mq_bio_to_request(struct request *rq, struct bio *bio,
 }
 
 static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx,
-					    struct request *rq,
-					    blk_qc_t *cookie, bool last)
+					    struct request *rq, bool last)
 {
 	struct request_queue *q = rq->q;
 	struct blk_mq_queue_data bd = {
 		.rq = rq,
 		.last = last,
 	};
-	blk_qc_t new_cookie;
 	blk_status_t ret;
 
-	new_cookie = request_to_qc_t(hctx, rq);
-
 	/*
 	 * For OK queue, we are done. For error, caller may kill it.
 	 * Any other error (busy), just add it to our list as we
@@ -2018,7 +2026,6 @@ static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx,
 	switch (ret) {
 	case BLK_STS_OK:
 		blk_mq_update_dispatch_busy(hctx, false);
-		*cookie = new_cookie;
 		break;
 	case BLK_STS_RESOURCE:
 	case BLK_STS_DEV_RESOURCE:
@@ -2027,7 +2034,6 @@ static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx,
 		break;
 	default:
 		blk_mq_update_dispatch_busy(hctx, false);
-		*cookie = BLK_QC_T_NONE;
 		break;
 	}
 
@@ -2036,7 +2042,6 @@ static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx,
 
 static blk_status_t __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx,
 						struct request *rq,
-						blk_qc_t *cookie,
 						bool bypass_insert, bool last)
 {
 	struct request_queue *q = rq->q;
@@ -2070,7 +2075,7 @@ static blk_status_t __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx,
 		goto insert;
 	}
 
-	return __blk_mq_issue_directly(hctx, rq, cookie, last);
+	return __blk_mq_issue_directly(hctx, rq, last);
 insert:
 	if (bypass_insert)
 		return BLK_STS_RESOURCE;
@@ -2084,7 +2089,6 @@ static blk_status_t __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx,
  * blk_mq_try_issue_directly - Try to send a request directly to device driver.
  * @hctx: Pointer of the associated hardware queue.
  * @rq: Pointer to request to be sent.
- * @cookie: Request queue cookie.
  *
  * If the device has enough resources to accept a new request now, send the
  * request directly to device driver. Else, insert at hctx->dispatch queue, so
@@ -2092,7 +2096,7 @@ static blk_status_t __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx,
  * queue have higher priority.
  */
 static void blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx,
-		struct request *rq, blk_qc_t *cookie)
+		struct request *rq)
 {
 	blk_status_t ret;
 	int srcu_idx;
@@ -2101,7 +2105,7 @@ static void blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx,
 
 	hctx_lock(hctx, &srcu_idx);
 
-	ret = __blk_mq_try_issue_directly(hctx, rq, cookie, false, true);
+	ret = __blk_mq_try_issue_directly(hctx, rq, false, true);
 	if (ret == BLK_STS_RESOURCE || ret == BLK_STS_DEV_RESOURCE)
 		blk_mq_request_bypass_insert(rq, false, true);
 	else if (ret != BLK_STS_OK)
@@ -2114,11 +2118,10 @@ blk_status_t blk_mq_request_issue_directly(struct request *rq, bool last)
 {
 	blk_status_t ret;
 	int srcu_idx;
-	blk_qc_t unused_cookie;
 	struct blk_mq_hw_ctx *hctx = rq->mq_hctx;
 
 	hctx_lock(hctx, &srcu_idx);
-	ret = __blk_mq_try_issue_directly(hctx, rq, &unused_cookie, true, last);
+	ret = __blk_mq_try_issue_directly(hctx, rq, true, last);
 	hctx_unlock(hctx, srcu_idx);
 
 	return ret;
@@ -2186,10 +2189,8 @@ static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq)
  *
  * It will not queue the request if there is an error with the bio, or at the
  * request creation.
- *
- * Returns: Request queue cookie.
  */
-blk_qc_t blk_mq_submit_bio(struct bio *bio)
+void blk_mq_submit_bio(struct bio *bio)
 {
 	struct request_queue *q = bio->bi_bdev->bd_disk->queue;
 	const int is_sync = op_is_sync(bio->bi_opf);
@@ -2201,9 +2202,7 @@ blk_qc_t blk_mq_submit_bio(struct bio *bio)
 	struct blk_plug *plug;
 	struct request *same_queue_rq = NULL;
 	unsigned int nr_segs;
-	blk_qc_t cookie;
 	blk_status_t ret;
-	bool hipri;
 
 	blk_queue_bounce(q, &bio);
 	__blk_queue_split(&bio, &nr_segs);
@@ -2220,8 +2219,6 @@ blk_qc_t blk_mq_submit_bio(struct bio *bio)
 
 	rq_qos_throttle(q, bio);
 
-	hipri = bio->bi_opf & REQ_POLLED;
-
 	data.cmd_flags = bio->bi_opf;
 	rq = __blk_mq_alloc_request(&data);
 	if (unlikely(!rq)) {
@@ -2235,8 +2232,6 @@ blk_qc_t blk_mq_submit_bio(struct bio *bio)
 
 	rq_qos_track(q, rq, bio);
 
-	cookie = request_to_qc_t(data.hctx, rq);
-
 	blk_mq_bio_to_request(rq, bio, nr_segs);
 
 	ret = blk_crypto_init_request(rq);
@@ -2244,7 +2239,7 @@ blk_qc_t blk_mq_submit_bio(struct bio *bio)
 		bio->bi_status = ret;
 		bio_endio(bio);
 		blk_mq_free_request(rq);
-		return BLK_QC_T_NONE;
+		return;
 	}
 
 	plug = blk_mq_plug(q, bio);
@@ -2300,8 +2295,7 @@ blk_qc_t blk_mq_submit_bio(struct bio *bio)
 		if (same_queue_rq) {
 			data.hctx = same_queue_rq->mq_hctx;
 			trace_block_unplug(q, 1, true);
-			blk_mq_try_issue_directly(data.hctx, same_queue_rq,
-					&cookie);
+			blk_mq_try_issue_directly(data.hctx, same_queue_rq);
 		}
 	} else if ((q->nr_hw_queues > 1 && is_sync) ||
 			!data.hctx->dispatch_busy) {
@@ -2309,18 +2303,15 @@ blk_qc_t blk_mq_submit_bio(struct bio *bio)
 		 * There is no scheduler and we can try to send directly
 		 * to the hardware.
 		 */
-		blk_mq_try_issue_directly(data.hctx, rq, &cookie);
+		blk_mq_try_issue_directly(data.hctx, rq);
 	} else {
 		/* Default case. */
 		blk_mq_sched_insert_request(rq, false, true, true);
 	}
 
-	if (!hipri)
-		return BLK_QC_T_NONE;
-	return cookie;
+	return;
 queue_exit:
 	blk_queue_exit(q);
-	return BLK_QC_T_NONE;
 }
 
 static size_t order_to_size(unsigned int order)
@@ -3983,25 +3974,8 @@ static int blk_mq_poll_classic(struct request_queue *q, blk_qc_t cookie,
 	return 0;
 }
 
-/**
- * blk_poll - poll for IO completions
- * @q:  the queue
- * @cookie: cookie passed back at IO submission time
- * @flags: BLK_POLL_* flags that control the behavior
- *
- * Description:
- *    Poll for completions on the passed in queue. Returns number of
- *    completed entries found.
- */
-int blk_poll(struct request_queue *q, blk_qc_t cookie, unsigned int flags)
+int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, unsigned int flags)
 {
-	if (cookie == BLK_QC_T_NONE ||
-	    !test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
-		return 0;
-
-	if (current->plug)
-		blk_flush_plug_list(current->plug, false);
-
 	if (!(flags & BLK_POLL_NOSLEEP) &&
 	    q->poll_nsec != BLK_MQ_POLL_CLASSIC) {
 		if (blk_mq_poll_hybrid(q, cookie))
@@ -4009,7 +3983,6 @@ int blk_poll(struct request_queue *q, blk_qc_t cookie, unsigned int flags)
 	}
 	return blk_mq_poll_classic(q, cookie, flags);
 }
-EXPORT_SYMBOL_GPL(blk_poll);
 
 unsigned int blk_mq_rq_cpu(struct request *rq)
 {
diff --git a/block/blk-mq.h b/block/blk-mq.h
index 8444e00e8d25..66d1a628a2a1 100644
--- a/block/blk-mq.h
+++ b/block/blk-mq.h
@@ -37,6 +37,8 @@ struct blk_mq_ctx {
 	struct kobject		kobj;
 } ____cacheline_aligned_in_smp;
 
+void blk_mq_submit_bio(struct bio *bio);
+int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, unsigned int flags);
 void blk_mq_exit_queue(struct request_queue *q);
 int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr);
 void blk_mq_wake_waiters(struct request_queue *q);
diff --git a/drivers/block/brd.c b/drivers/block/brd.c
index 95694113e38e..274e2634ea58 100644
--- a/drivers/block/brd.c
+++ b/drivers/block/brd.c
@@ -285,7 +285,7 @@ static int brd_do_bvec(struct brd_device *brd, struct page *page,
 	return err;
 }
 
-static blk_qc_t brd_submit_bio(struct bio *bio)
+static void brd_submit_bio(struct bio *bio)
 {
 	struct brd_device *brd = bio->bi_bdev->bd_disk->private_data;
 	sector_t sector = bio->bi_iter.bi_sector;
@@ -302,16 +302,14 @@ static blk_qc_t brd_submit_bio(struct bio *bio)
 
 		err = brd_do_bvec(brd, bvec.bv_page, len, bvec.bv_offset,
 				  bio_op(bio), sector);
-		if (err)
-			goto io_error;
+		if (err) {
+			bio_io_error(bio);
+			return;
+		}
 		sector += len >> SECTOR_SHIFT;
 	}
 
 	bio_endio(bio);
-	return BLK_QC_T_NONE;
-io_error:
-	bio_io_error(bio);
-	return BLK_QC_T_NONE;
 }
 
 static int brd_rw_page(struct block_device *bdev, sector_t sector,
diff --git a/drivers/block/drbd/drbd_int.h b/drivers/block/drbd/drbd_int.h
index 5d9181382ce1..6674a0b88341 100644
--- a/drivers/block/drbd/drbd_int.h
+++ b/drivers/block/drbd/drbd_int.h
@@ -1448,7 +1448,7 @@ extern void conn_free_crypto(struct drbd_connection *connection);
 /* drbd_req */
 extern void do_submit(struct work_struct *ws);
 extern void __drbd_make_request(struct drbd_device *, struct bio *);
-extern blk_qc_t drbd_submit_bio(struct bio *bio);
+void drbd_submit_bio(struct bio *bio);
 extern int drbd_read_remote(struct drbd_device *device, struct drbd_request *req);
 extern int is_valid_ar_handle(struct drbd_request *, sector_t);
 
diff --git a/drivers/block/drbd/drbd_req.c b/drivers/block/drbd/drbd_req.c
index 13beb98a7c5a..0a2b3cb7e0bd 100644
--- a/drivers/block/drbd/drbd_req.c
+++ b/drivers/block/drbd/drbd_req.c
@@ -1597,7 +1597,7 @@ void do_submit(struct work_struct *ws)
 	}
 }
 
-blk_qc_t drbd_submit_bio(struct bio *bio)
+void drbd_submit_bio(struct bio *bio)
 {
 	struct drbd_device *device = bio->bi_bdev->bd_disk->private_data;
 
@@ -1610,7 +1610,6 @@ blk_qc_t drbd_submit_bio(struct bio *bio)
 
 	inc_ap_bio(device);
 	__drbd_make_request(device, bio);
-	return BLK_QC_T_NONE;
 }
 
 static bool net_timeout_reached(struct drbd_request *net_req,
diff --git a/drivers/block/n64cart.c b/drivers/block/n64cart.c
index 7b4dd10af9ec..f95fd4f04f28 100644
--- a/drivers/block/n64cart.c
+++ b/drivers/block/n64cart.c
@@ -84,7 +84,7 @@ static bool n64cart_do_bvec(struct device *dev, struct bio_vec *bv, u32 pos)
 	return true;
 }
 
-static blk_qc_t n64cart_submit_bio(struct bio *bio)
+static void n64cart_submit_bio(struct bio *bio)
 {
 	struct bio_vec bvec;
 	struct bvec_iter iter;
@@ -92,16 +92,14 @@ static blk_qc_t n64cart_submit_bio(struct bio *bio)
 	u32 pos = bio->bi_iter.bi_sector << SECTOR_SHIFT;
 
 	bio_for_each_segment(bvec, bio, iter) {
-		if (!n64cart_do_bvec(dev, &bvec, pos))
-			goto io_error;
+		if (!n64cart_do_bvec(dev, &bvec, pos)) {
+			bio_io_error(bio);
+			return;
+		}
 		pos += bvec.bv_len;
 	}
 
 	bio_endio(bio);
-	return BLK_QC_T_NONE;
-io_error:
-	bio_io_error(bio);
-	return BLK_QC_T_NONE;
 }
 
 static const struct block_device_operations n64cart_fops = {
diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c
index 3b320b005aa8..7ce063871a5d 100644
--- a/drivers/block/null_blk/main.c
+++ b/drivers/block/null_blk/main.c
@@ -1426,7 +1426,7 @@ static struct nullb_queue *nullb_to_queue(struct nullb *nullb)
 	return &nullb->queues[index];
 }
 
-static blk_qc_t null_submit_bio(struct bio *bio)
+static void null_submit_bio(struct bio *bio)
 {
 	sector_t sector = bio->bi_iter.bi_sector;
 	sector_t nr_sectors = bio_sectors(bio);
@@ -1438,7 +1438,6 @@ static blk_qc_t null_submit_bio(struct bio *bio)
 	cmd->bio = bio;
 
 	null_handle_cmd(cmd, sector, nr_sectors, bio_op(bio));
-	return BLK_QC_T_NONE;
 }
 
 static bool should_timeout_request(struct request *rq)
diff --git a/drivers/block/pktcdvd.c b/drivers/block/pktcdvd.c
index f69b5c69c2a6..5bfb26f90b6d 100644
--- a/drivers/block/pktcdvd.c
+++ b/drivers/block/pktcdvd.c
@@ -2402,7 +2402,7 @@ static void pkt_make_request_write(struct request_queue *q, struct bio *bio)
 	}
 }
 
-static blk_qc_t pkt_submit_bio(struct bio *bio)
+static void pkt_submit_bio(struct bio *bio)
 {
 	struct pktcdvd_device *pd;
 	char b[BDEVNAME_SIZE];
@@ -2425,7 +2425,7 @@ static blk_qc_t pkt_submit_bio(struct bio *bio)
 	 */
 	if (bio_data_dir(bio) == READ) {
 		pkt_make_request_read(pd, bio);
-		return BLK_QC_T_NONE;
+		return;
 	}
 
 	if (!test_bit(PACKET_WRITABLE, &pd->flags)) {
@@ -2457,10 +2457,9 @@ static blk_qc_t pkt_submit_bio(struct bio *bio)
 		pkt_make_request_write(bio->bi_bdev->bd_disk->queue, split);
 	} while (split != bio);
 
-	return BLK_QC_T_NONE;
+	return;
 end_io:
 	bio_io_error(bio);
-	return BLK_QC_T_NONE;
 }
 
 static void pkt_init_queue(struct pktcdvd_device *pd)
diff --git a/drivers/block/ps3vram.c b/drivers/block/ps3vram.c
index 7fbf469651c4..507520cab9c3 100644
--- a/drivers/block/ps3vram.c
+++ b/drivers/block/ps3vram.c
@@ -578,7 +578,7 @@ static struct bio *ps3vram_do_bio(struct ps3_system_bus_device *dev,
 	return next;
 }
 
-static blk_qc_t ps3vram_submit_bio(struct bio *bio)
+static void ps3vram_submit_bio(struct bio *bio)
 {
 	struct ps3_system_bus_device *dev = bio->bi_bdev->bd_disk->private_data;
 	struct ps3vram_priv *priv = ps3_system_bus_get_drvdata(dev);
@@ -594,13 +594,11 @@ static blk_qc_t ps3vram_submit_bio(struct bio *bio)
 	spin_unlock_irq(&priv->lock);
 
 	if (busy)
-		return BLK_QC_T_NONE;
+		return;
 
 	do {
 		bio = ps3vram_do_bio(dev, bio);
 	} while (bio);
-
-	return BLK_QC_T_NONE;
 }
 
 static const struct block_device_operations ps3vram_fops = {
diff --git a/drivers/block/rsxx/dev.c b/drivers/block/rsxx/dev.c
index 1cc40b0ea761..268252380e88 100644
--- a/drivers/block/rsxx/dev.c
+++ b/drivers/block/rsxx/dev.c
@@ -50,7 +50,7 @@ struct rsxx_bio_meta {
 
 static struct kmem_cache *bio_meta_pool;
 
-static blk_qc_t rsxx_submit_bio(struct bio *bio);
+static void rsxx_submit_bio(struct bio *bio);
 
 /*----------------- Block Device Operations -----------------*/
 static int rsxx_blkdev_ioctl(struct block_device *bdev,
@@ -120,7 +120,7 @@ static void bio_dma_done_cb(struct rsxx_cardinfo *card,
 	}
 }
 
-static blk_qc_t rsxx_submit_bio(struct bio *bio)
+static void rsxx_submit_bio(struct bio *bio)
 {
 	struct rsxx_cardinfo *card = bio->bi_bdev->bd_disk->private_data;
 	struct rsxx_bio_meta *bio_meta;
@@ -169,7 +169,7 @@ static blk_qc_t rsxx_submit_bio(struct bio *bio)
 	if (st)
 		goto queue_err;
 
-	return BLK_QC_T_NONE;
+	return;
 
 queue_err:
 	kmem_cache_free(bio_meta_pool, bio_meta);
@@ -177,7 +177,6 @@ static blk_qc_t rsxx_submit_bio(struct bio *bio)
 	if (st)
 		bio->bi_status = st;
 	bio_endio(bio);
-	return BLK_QC_T_NONE;
 }
 
 /*----------------- Device Setup -------------------*/
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index fcaf2750f68f..a68297fb51a2 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -1598,22 +1598,18 @@ static void __zram_make_request(struct zram *zram, struct bio *bio)
 /*
  * Handler function for all zram I/O requests.
  */
-static blk_qc_t zram_submit_bio(struct bio *bio)
+static void zram_submit_bio(struct bio *bio)
 {
 	struct zram *zram = bio->bi_bdev->bd_disk->private_data;
 
 	if (!valid_io_request(zram, bio->bi_iter.bi_sector,
 					bio->bi_iter.bi_size)) {
 		atomic64_inc(&zram->stats.invalid_io);
-		goto error;
+		bio_io_error(bio);
+		return;
 	}
 
 	__zram_make_request(zram, bio);
-	return BLK_QC_T_NONE;
-
-error:
-	bio_io_error(bio);
-	return BLK_QC_T_NONE;
 }
 
 static void zram_slot_free_notify(struct block_device *bdev,
diff --git a/drivers/lightnvm/pblk-init.c b/drivers/lightnvm/pblk-init.c
index 5924f09c217b..6716003d4cb6 100644
--- a/drivers/lightnvm/pblk-init.c
+++ b/drivers/lightnvm/pblk-init.c
@@ -47,7 +47,7 @@ static struct pblk_global_caches pblk_caches = {
 
 struct bio_set pblk_bio_set;
 
-static blk_qc_t pblk_submit_bio(struct bio *bio)
+static void pblk_submit_bio(struct bio *bio)
 {
 	struct pblk *pblk = bio->bi_bdev->bd_disk->queue->queuedata;
 
@@ -55,7 +55,7 @@ static blk_qc_t pblk_submit_bio(struct bio *bio)
 		pblk_discard(pblk, bio);
 		if (!(bio->bi_opf & REQ_PREFLUSH)) {
 			bio_endio(bio);
-			return BLK_QC_T_NONE;
+			return;
 		}
 	}
 
@@ -75,8 +75,6 @@ static blk_qc_t pblk_submit_bio(struct bio *bio)
 
 		pblk_write_to_cache(pblk, bio, PBLK_IOTYPE_USER);
 	}
-
-	return BLK_QC_T_NONE;
 }
 
 static const struct block_device_operations pblk_bops = {
diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c
index 29c231758293..b549147a567f 100644
--- a/drivers/md/bcache/request.c
+++ b/drivers/md/bcache/request.c
@@ -1169,7 +1169,7 @@ static void quit_max_writeback_rate(struct cache_set *c,
 
 /* Cached devices - read & write stuff */
 
-blk_qc_t cached_dev_submit_bio(struct bio *bio)
+void cached_dev_submit_bio(struct bio *bio)
 {
 	struct search *s;
 	struct block_device *orig_bdev = bio->bi_bdev;
@@ -1182,7 +1182,7 @@ blk_qc_t cached_dev_submit_bio(struct bio *bio)
 		     dc->io_disable)) {
 		bio->bi_status = BLK_STS_IOERR;
 		bio_endio(bio);
-		return BLK_QC_T_NONE;
+		return;
 	}
 
 	if (likely(d->c)) {
@@ -1228,8 +1228,6 @@ blk_qc_t cached_dev_submit_bio(struct bio *bio)
 	} else
 		/* I/O request sent to backing device */
 		detached_dev_do_request(d, bio, orig_bdev, start_time);
-
-	return BLK_QC_T_NONE;
 }
 
 static int cached_dev_ioctl(struct bcache_device *d, fmode_t mode,
@@ -1279,7 +1277,7 @@ static void flash_dev_nodata(struct closure *cl)
 	continue_at(cl, search_free, NULL);
 }
 
-blk_qc_t flash_dev_submit_bio(struct bio *bio)
+void flash_dev_submit_bio(struct bio *bio)
 {
 	struct search *s;
 	struct closure *cl;
@@ -1288,7 +1286,7 @@ blk_qc_t flash_dev_submit_bio(struct bio *bio)
 	if (unlikely(d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags))) {
 		bio->bi_status = BLK_STS_IOERR;
 		bio_endio(bio);
-		return BLK_QC_T_NONE;
+		return;
 	}
 
 	s = search_alloc(bio, d, bio->bi_bdev, bio_start_io_acct(bio));
@@ -1304,7 +1302,7 @@ blk_qc_t flash_dev_submit_bio(struct bio *bio)
 		continue_at_nobarrier(&s->cl,
 				      flash_dev_nodata,
 				      bcache_wq);
-		return BLK_QC_T_NONE;
+		return;
 	} else if (bio_data_dir(bio)) {
 		bch_keybuf_check_overlapping(&s->iop.c->moving_gc_keys,
 					&KEY(d->id, bio->bi_iter.bi_sector, 0),
@@ -1320,7 +1318,6 @@ blk_qc_t flash_dev_submit_bio(struct bio *bio)
 	}
 
 	continue_at(cl, search_free, NULL);
-	return BLK_QC_T_NONE;
 }
 
 static int flash_dev_ioctl(struct bcache_device *d, fmode_t mode,
diff --git a/drivers/md/bcache/request.h b/drivers/md/bcache/request.h
index 82b38366a95d..38ab4856eaab 100644
--- a/drivers/md/bcache/request.h
+++ b/drivers/md/bcache/request.h
@@ -37,10 +37,10 @@ unsigned int bch_get_congested(const struct cache_set *c);
 void bch_data_insert(struct closure *cl);
 
 void bch_cached_dev_request_init(struct cached_dev *dc);
-blk_qc_t cached_dev_submit_bio(struct bio *bio);
+void cached_dev_submit_bio(struct bio *bio);
 
 void bch_flash_dev_request_init(struct bcache_device *d);
-blk_qc_t flash_dev_submit_bio(struct bio *bio);
+void flash_dev_submit_bio(struct bio *bio);
 
 extern struct kmem_cache *bch_search_cache;
 
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 3c7c2d257018..363f12a285ce 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1292,14 +1292,13 @@ static noinline void __set_swap_bios_limit(struct mapped_device *md, int latch)
 	mutex_unlock(&md->swap_bios_lock);
 }
 
-static blk_qc_t __map_bio(struct dm_target_io *tio)
+static void __map_bio(struct dm_target_io *tio)
 {
 	int r;
 	sector_t sector;
 	struct bio *clone = &tio->clone;
 	struct dm_io *io = tio->io;
 	struct dm_target *ti = tio->ti;
-	blk_qc_t ret = BLK_QC_T_NONE;
 
 	clone->bi_end_io = clone_endio;
 
@@ -1326,7 +1325,7 @@ static blk_qc_t __map_bio(struct dm_target_io *tio)
 	case DM_MAPIO_REMAPPED:
 		/* the bio has been remapped so dispatch it */
 		trace_block_bio_remap(clone, bio_dev(io->orig_bio), sector);
-		ret = submit_bio_noacct(clone);
+		submit_bio_noacct(clone);
 		break;
 	case DM_MAPIO_KILL:
 		if (unlikely(swap_bios_limit(ti, clone))) {
@@ -1348,8 +1347,6 @@ static blk_qc_t __map_bio(struct dm_target_io *tio)
 		DMWARN("unimplemented target map return value: %d", r);
 		BUG();
 	}
-
-	return ret;
 }
 
 static void bio_setup_sector(struct bio *bio, sector_t sector, unsigned len)
@@ -1436,7 +1433,7 @@ static void alloc_multiple_bios(struct bio_list *blist, struct clone_info *ci,
 	}
 }
 
-static blk_qc_t __clone_and_map_simple_bio(struct clone_info *ci,
+static void __clone_and_map_simple_bio(struct clone_info *ci,
 					   struct dm_target_io *tio, unsigned *len)
 {
 	struct bio *clone = &tio->clone;
@@ -1446,8 +1443,7 @@ static blk_qc_t __clone_and_map_simple_bio(struct clone_info *ci,
 	__bio_clone_fast(clone, ci->bio);
 	if (len)
 		bio_setup_sector(clone, ci->sector, *len);
-
-	return __map_bio(tio);
+	__map_bio(tio);
 }
 
 static void __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti,
@@ -1461,7 +1457,7 @@ static void __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti,
 
 	while ((bio = bio_list_pop(&blist))) {
 		tio = container_of(bio, struct dm_target_io, clone);
-		(void) __clone_and_map_simple_bio(ci, tio, len);
+		__clone_and_map_simple_bio(ci, tio, len);
 	}
 }
 
@@ -1505,7 +1501,7 @@ static int __clone_and_map_data_bio(struct clone_info *ci, struct dm_target *ti,
 		free_tio(tio);
 		return r;
 	}
-	(void) __map_bio(tio);
+	__map_bio(tio);
 
 	return 0;
 }
@@ -1620,11 +1616,10 @@ static void init_clone_info(struct clone_info *ci, struct mapped_device *md,
 /*
  * Entry point to split a bio into clones and submit them to the targets.
  */
-static blk_qc_t __split_and_process_bio(struct mapped_device *md,
+static void __split_and_process_bio(struct mapped_device *md,
 					struct dm_table *map, struct bio *bio)
 {
 	struct clone_info ci;
-	blk_qc_t ret = BLK_QC_T_NONE;
 	int error = 0;
 
 	init_clone_info(&ci, md, map, bio);
@@ -1667,19 +1662,17 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md,
 
 			bio_chain(b, bio);
 			trace_block_split(b, bio->bi_iter.bi_sector);
-			ret = submit_bio_noacct(bio);
+			submit_bio_noacct(bio);
 		}
 	}
 
 	/* drop the extra reference count */
 	dec_pending(ci.io, errno_to_blk_status(error));
-	return ret;
 }
 
-static blk_qc_t dm_submit_bio(struct bio *bio)
+static void dm_submit_bio(struct bio *bio)
 {
 	struct mapped_device *md = bio->bi_bdev->bd_disk->private_data;
-	blk_qc_t ret = BLK_QC_T_NONE;
 	int srcu_idx;
 	struct dm_table *map;
 
@@ -1709,10 +1702,9 @@ static blk_qc_t dm_submit_bio(struct bio *bio)
 	if (is_abnormal_io(bio))
 		blk_queue_split(&bio);
 
-	ret = __split_and_process_bio(md, map, bio);
+	__split_and_process_bio(md, map, bio);
 out:
 	dm_put_live_table(md, srcu_idx);
-	return ret;
 }
 
 /*-----------------------------------------------------------------
diff --git a/drivers/md/md.c b/drivers/md/md.c
index d806be8cc210..7a98437a376b 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -465,19 +465,19 @@ static void md_end_io(struct bio *bio)
 		bio->bi_end_io(bio);
 }
 
-static blk_qc_t md_submit_bio(struct bio *bio)
+static void md_submit_bio(struct bio *bio)
 {
 	const int rw = bio_data_dir(bio);
 	struct mddev *mddev = bio->bi_bdev->bd_disk->private_data;
 
 	if (mddev == NULL || mddev->pers == NULL) {
 		bio_io_error(bio);
-		return BLK_QC_T_NONE;
+		return;
 	}
 
 	if (unlikely(test_bit(MD_BROKEN, &mddev->flags)) && (rw == WRITE)) {
 		bio_io_error(bio);
-		return BLK_QC_T_NONE;
+		return;
 	}
 
 	blk_queue_split(&bio);
@@ -486,7 +486,7 @@ static blk_qc_t md_submit_bio(struct bio *bio)
 		if (bio_sectors(bio) != 0)
 			bio->bi_status = BLK_STS_IOERR;
 		bio_endio(bio);
-		return BLK_QC_T_NONE;
+		return;
 	}
 
 	if (bio->bi_end_io != md_end_io) {
@@ -508,8 +508,6 @@ static blk_qc_t md_submit_bio(struct bio *bio)
 	bio->bi_opf &= ~REQ_NOMERGE;
 
 	md_handle_request(mddev, bio);
-
-	return BLK_QC_T_NONE;
 }
 
 /* mddev_suspend makes sure no new requests are submitted
diff --git a/drivers/nvdimm/blk.c b/drivers/nvdimm/blk.c
index 088d3dd6f6fa..b6c6866f9259 100644
--- a/drivers/nvdimm/blk.c
+++ b/drivers/nvdimm/blk.c
@@ -162,7 +162,7 @@ static int nsblk_do_bvec(struct nd_namespace_blk *nsblk,
 	return err;
 }
 
-static blk_qc_t nd_blk_submit_bio(struct bio *bio)
+static void nd_blk_submit_bio(struct bio *bio)
 {
 	struct bio_integrity_payload *bip;
 	struct nd_namespace_blk *nsblk = bio->bi_bdev->bd_disk->private_data;
@@ -173,7 +173,7 @@ static blk_qc_t nd_blk_submit_bio(struct bio *bio)
 	bool do_acct;
 
 	if (!bio_integrity_prep(bio))
-		return BLK_QC_T_NONE;
+		return;
 
 	bip = bio_integrity(bio);
 	rw = bio_data_dir(bio);
@@ -199,7 +199,6 @@ static blk_qc_t nd_blk_submit_bio(struct bio *bio)
 		bio_end_io_acct(bio, start);
 
 	bio_endio(bio);
-	return BLK_QC_T_NONE;
 }
 
 static int nsblk_rw_bytes(struct nd_namespace_common *ndns,
diff --git a/drivers/nvdimm/btt.c b/drivers/nvdimm/btt.c
index 92dec4952297..4295fa809420 100644
--- a/drivers/nvdimm/btt.c
+++ b/drivers/nvdimm/btt.c
@@ -1440,7 +1440,7 @@ static int btt_do_bvec(struct btt *btt, struct bio_integrity_payload *bip,
 	return ret;
 }
 
-static blk_qc_t btt_submit_bio(struct bio *bio)
+static void btt_submit_bio(struct bio *bio)
 {
 	struct bio_integrity_payload *bip = bio_integrity(bio);
 	struct btt *btt = bio->bi_bdev->bd_disk->private_data;
@@ -1451,7 +1451,7 @@ static blk_qc_t btt_submit_bio(struct bio *bio)
 	bool do_acct;
 
 	if (!bio_integrity_prep(bio))
-		return BLK_QC_T_NONE;
+		return;
 
 	do_acct = blk_queue_io_stat(bio->bi_bdev->bd_disk->queue);
 	if (do_acct)
@@ -1483,7 +1483,6 @@ static blk_qc_t btt_submit_bio(struct bio *bio)
 		bio_end_io_acct(bio, start);
 
 	bio_endio(bio);
-	return BLK_QC_T_NONE;
 }
 
 static int btt_rw_page(struct block_device *bdev, sector_t sector,
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 1e0615b8565e..773385e16b4d 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -190,7 +190,7 @@ static blk_status_t pmem_do_write(struct pmem_device *pmem,
 	return rc;
 }
 
-static blk_qc_t pmem_submit_bio(struct bio *bio)
+static void pmem_submit_bio(struct bio *bio)
 {
 	int ret = 0;
 	blk_status_t rc = 0;
@@ -229,7 +229,6 @@ static blk_qc_t pmem_submit_bio(struct bio *bio)
 		bio->bi_status = errno_to_blk_status(ret);
 
 	bio_endio(bio);
-	return BLK_QC_T_NONE;
 }
 
 static int pmem_rw_page(struct block_device *bdev, sector_t sector,
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 1f2ecb2fe9b1..fe0b8da3de7f 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1032,7 +1032,7 @@ static void nvme_execute_rq_polled(struct request_queue *q,
 	blk_execute_rq_nowait(bd_disk, rq, at_head, nvme_end_sync_rq);
 
 	while (!completion_done(&wait)) {
-		blk_poll(q, request_to_qc_t(rq->mq_hctx, rq), 0);
+		bio_poll(rq->bio, 0);
 		cond_resched();
 	}
 }
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index b5fbdb416022..c1b4d627c05b 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -297,12 +297,11 @@ static bool nvme_available_path(struct nvme_ns_head *head)
 	return false;
 }
 
-static blk_qc_t nvme_ns_head_submit_bio(struct bio *bio)
+static void nvme_ns_head_submit_bio(struct bio *bio)
 {
 	struct nvme_ns_head *head = bio->bi_bdev->bd_disk->private_data;
 	struct device *dev = disk_to_dev(head->disk);
 	struct nvme_ns *ns;
-	blk_qc_t ret = BLK_QC_T_NONE;
 	int srcu_idx;
 
 	/*
@@ -319,7 +318,7 @@ static blk_qc_t nvme_ns_head_submit_bio(struct bio *bio)
 		bio->bi_opf |= REQ_NVME_MPATH;
 		trace_block_bio_remap(bio, disk_devt(ns->head->disk),
 				      bio->bi_iter.bi_sector);
-		ret = submit_bio_noacct(bio);
+		submit_bio_noacct(bio);
 	} else if (nvme_available_path(head)) {
 		dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");
 
@@ -334,7 +333,6 @@ static blk_qc_t nvme_ns_head_submit_bio(struct bio *bio)
 	}
 
 	srcu_read_unlock(&head->srcu, srcu_idx);
-	return ret;
 }
 
 static int nvme_ns_head_open(struct block_device *bdev, fmode_t mode)
diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c
index 7faa56399999..72a34d1486d1 100644
--- a/drivers/s390/block/dcssblk.c
+++ b/drivers/s390/block/dcssblk.c
@@ -31,7 +31,7 @@
 
 static int dcssblk_open(struct block_device *bdev, fmode_t mode);
 static void dcssblk_release(struct gendisk *disk, fmode_t mode);
-static blk_qc_t dcssblk_submit_bio(struct bio *bio);
+static void dcssblk_submit_bio(struct bio *bio);
 static long dcssblk_dax_direct_access(struct dax_device *dax_dev, pgoff_t pgoff,
 		long nr_pages, void **kaddr, pfn_t *pfn);
 
@@ -855,7 +855,7 @@ dcssblk_release(struct gendisk *disk, fmode_t mode)
 	up_write(&dcssblk_devices_sem);
 }
 
-static blk_qc_t
+static void
 dcssblk_submit_bio(struct bio *bio)
 {
 	struct dcssblk_dev_info *dev_info;
@@ -909,10 +909,9 @@ dcssblk_submit_bio(struct bio *bio)
 		bytes_done += bvec.bv_len;
 	}
 	bio_endio(bio);
-	return BLK_QC_T_NONE;
+	return;
 fail:
 	bio_io_error(bio);
-	return BLK_QC_T_NONE;
 }
 
 static long
diff --git a/drivers/s390/block/xpram.c b/drivers/s390/block/xpram.c
index 91ef710edfd2..aa49a6022d45 100644
--- a/drivers/s390/block/xpram.c
+++ b/drivers/s390/block/xpram.c
@@ -181,7 +181,7 @@ static unsigned long xpram_highest_page_index(void)
 /*
  * Block device make request function.
  */
-static blk_qc_t xpram_submit_bio(struct bio *bio)
+static void xpram_submit_bio(struct bio *bio)
 {
 	xpram_device_t *xdev = bio->bi_bdev->bd_disk->private_data;
 	struct bio_vec bvec;
@@ -223,10 +223,9 @@ static blk_qc_t xpram_submit_bio(struct bio *bio)
 		}
 	}
 	bio_endio(bio);
-	return BLK_QC_T_NONE;
+	return;
 fail:
 	bio_io_error(bio);
-	return BLK_QC_T_NONE;
 }
 
 static int xpram_getgeo(struct block_device *bdev, struct hd_geometry *geo)
diff --git a/fs/block_dev.c b/fs/block_dev.c
index c9bdaa138909..6be5ddba86a3 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -239,7 +239,6 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter,
 	bool should_dirty = false;
 	struct bio bio;
 	ssize_t ret;
-	blk_qc_t qc;
 
 	if ((pos | iov_iter_alignment(iter)) &
 	    (bdev_logical_block_size(bdev) - 1))
@@ -280,13 +279,12 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter,
 	if (iocb->ki_flags & IOCB_HIPRI)
 		bio_set_polled(&bio, iocb);
 
-	qc = submit_bio(&bio);
+	submit_bio(&bio);
 	for (;;) {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		if (!READ_ONCE(bio.bi_private))
 			break;
-		if (!(iocb->ki_flags & IOCB_HIPRI) ||
-		    !blk_poll(bdev_get_queue(bdev), qc, 0))
+		if (!(iocb->ki_flags & IOCB_HIPRI) || !bio_poll(&bio, 0))
 			blk_io_schedule();
 	}
 	__set_current_state(TASK_RUNNING);
@@ -319,14 +317,6 @@ struct blkdev_dio {
 
 static struct bio_set blkdev_dio_pool;
 
-static int blkdev_iopoll(struct kiocb *kiocb, unsigned int flags)
-{
-	struct block_device *bdev = I_BDEV(kiocb->ki_filp->f_mapping->host);
-	struct request_queue *q = bdev_get_queue(bdev);
-
-	return blk_poll(q, READ_ONCE(kiocb->ki_cookie), flags);
-}
-
 static void blkdev_bio_end_io(struct bio *bio)
 {
 	struct blkdev_dio *dio = bio->bi_private;
@@ -340,6 +330,8 @@ static void blkdev_bio_end_io(struct bio *bio)
 			struct kiocb *iocb = dio->iocb;
 			ssize_t ret;
 
+			WRITE_ONCE(iocb->private, NULL);
+
 			if (likely(!dio->bio.bi_status)) {
 				ret = dio->size;
 				iocb->ki_pos += ret;
@@ -378,7 +370,6 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 	bool do_poll = (iocb->ki_flags & IOCB_HIPRI);
 	bool is_read = (iov_iter_rw(iter) == READ), is_sync;
 	loff_t pos = iocb->ki_pos;
-	blk_qc_t qc = BLK_QC_T_NONE;
 	int ret = 0;
 
 	if ((pos | iov_iter_alignment(iter)) &
@@ -440,9 +431,9 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 		if (!nr_pages) {
 			if (do_poll)
 				bio_set_polled(bio, iocb);
-			qc = submit_bio(bio);
+			submit_bio(bio);
 			if (do_poll)
-				WRITE_ONCE(iocb->ki_cookie, qc);
+				WRITE_ONCE(iocb->private, bio);
 			break;
 		}
 		if (!dio->multi_bio) {
@@ -475,7 +466,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 		if (!READ_ONCE(dio->waiter))
 			break;
 
-		if (!do_poll || !blk_poll(bdev_get_queue(bdev), qc, 0))
+		if (!do_poll || !bio_poll(bio, 0))
 			blk_io_schedule();
 	}
 	__set_current_state(TASK_RUNNING);
@@ -1825,7 +1816,7 @@ const struct file_operations def_blk_fops = {
 	.llseek		= block_llseek,
 	.read_iter	= blkdev_read_iter,
 	.write_iter	= blkdev_write_iter,
-	.iopoll		= blkdev_iopoll,
+	.iopoll		= iocb_bio_iopoll,
 	.mmap		= generic_file_mmap,
 	.fsync		= blkdev_fsync,
 	.unlocked_ioctl	= block_ioctl,
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 33f14573f2ec..450d82998c23 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -8104,7 +8104,7 @@ static struct btrfs_dio_private *btrfs_create_dio_private(struct bio *dio_bio,
 	return dip;
 }
 
-static blk_qc_t btrfs_submit_direct(struct inode *inode, struct iomap *iomap,
+static void btrfs_submit_direct(struct inode *inode, struct iomap *iomap,
 		struct bio *dio_bio, loff_t file_offset)
 {
 	const bool write = (btrfs_op(dio_bio) == BTRFS_MAP_WRITE);
@@ -8133,7 +8133,7 @@ static blk_qc_t btrfs_submit_direct(struct inode *inode, struct iomap *iomap,
 		}
 		dio_bio->bi_status = BLK_STS_RESOURCE;
 		bio_endio(dio_bio);
-		return BLK_QC_T_NONE;
+		return;
 	}
 
 	if (!write) {
@@ -8227,15 +8227,13 @@ static blk_qc_t btrfs_submit_direct(struct inode *inode, struct iomap *iomap,
 
 		free_extent_map(em);
 	} while (submit_len > 0);
-	return BLK_QC_T_NONE;
+	return;
 
 out_err_em:
 	free_extent_map(em);
 out_err:
 	dip->dio_bio->bi_status = status;
 	btrfs_dio_private_put(dip);
-
-	return BLK_QC_T_NONE;
 }
 
 const struct iomap_ops btrfs_dio_iomap_ops = {
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index 816dedcbd541..6146c4203fda 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -913,7 +913,7 @@ const struct file_operations ext4_file_operations = {
 	.llseek		= ext4_llseek,
 	.read_iter	= ext4_file_read_iter,
 	.write_iter	= ext4_file_write_iter,
-	.iopoll		= iomap_dio_iopoll,
+	.iopoll		= iocb_bio_iopoll,
 	.unlocked_ioctl = ext4_ioctl,
 #ifdef CONFIG_COMPAT
 	.compat_ioctl	= ext4_compat_ioctl,
diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
index a0b542d84cd9..d51d0c663bde 100644
--- a/fs/gfs2/file.c
+++ b/fs/gfs2/file.c
@@ -1334,7 +1334,7 @@ const struct file_operations gfs2_file_fops = {
 	.llseek		= gfs2_llseek,
 	.read_iter	= gfs2_file_read_iter,
 	.write_iter	= gfs2_file_write_iter,
-	.iopoll		= iomap_dio_iopoll,
+	.iopoll		= iocb_bio_iopoll,
 	.unlocked_ioctl	= gfs2_ioctl,
 	.compat_ioctl	= gfs2_compat_ioctl,
 	.mmap		= gfs2_mmap,
@@ -1367,7 +1367,7 @@ const struct file_operations gfs2_file_fops_nolock = {
 	.llseek		= gfs2_llseek,
 	.read_iter	= gfs2_file_read_iter,
 	.write_iter	= gfs2_file_write_iter,
-	.iopoll		= iomap_dio_iopoll,
+	.iopoll		= iocb_bio_iopoll,
 	.unlocked_ioctl	= gfs2_ioctl,
 	.compat_ioctl	= gfs2_compat_ioctl,
 	.mmap		= gfs2_mmap,
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 9b6c26da3a2d..76bc7c4b0deb 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -38,8 +38,7 @@ struct iomap_dio {
 		struct {
 			struct iov_iter		*iter;
 			struct task_struct	*waiter;
-			struct request_queue	*last_queue;
-			blk_qc_t		cookie;
+			struct bio		*poll_bio;
 		} submit;
 
 		/* used for aio completion: */
@@ -49,31 +48,21 @@ struct iomap_dio {
 	};
 };
 
-int iomap_dio_iopoll(struct kiocb *kiocb, unsigned int flags)
-{
-	struct request_queue *q = READ_ONCE(kiocb->private);
-
-	if (!q)
-		return 0;
-	return blk_poll(q, READ_ONCE(kiocb->ki_cookie), flags);
-}
-EXPORT_SYMBOL_GPL(iomap_dio_iopoll);
-
 static void iomap_dio_submit_bio(struct iomap_dio *dio, struct iomap *iomap,
 		struct bio *bio, loff_t pos)
 {
 	atomic_inc(&dio->ref);
 
-	if (dio->iocb->ki_flags & IOCB_HIPRI)
+	if (dio->iocb->ki_flags & IOCB_HIPRI) {
 		bio_set_polled(bio, dio->iocb);
+		dio->submit.poll_bio = bio;
+	}
 
-	dio->submit.last_queue = bdev_get_queue(iomap->bdev);
 	if (dio->dops && dio->dops->submit_io)
-		dio->submit.cookie = dio->dops->submit_io(
-				file_inode(dio->iocb->ki_filp),
-				iomap, bio, pos);
+		dio->dops->submit_io(file_inode(dio->iocb->ki_filp), iomap, bio,
+				     pos);
 	else
-		dio->submit.cookie = submit_bio(bio);
+		submit_bio(bio);
 }
 
 ssize_t iomap_dio_complete(struct iomap_dio *dio)
@@ -166,9 +155,11 @@ static void iomap_dio_bio_end_io(struct bio *bio)
 		} else if (dio->flags & IOMAP_DIO_WRITE) {
 			struct inode *inode = file_inode(dio->iocb->ki_filp);
 
+			WRITE_ONCE(dio->iocb->private, NULL);
 			INIT_WORK(&dio->aio.work, iomap_dio_complete_work);
 			queue_work(inode->i_sb->s_dio_done_wq, &dio->aio.work);
 		} else {
+			WRITE_ONCE(dio->iocb->private, NULL);
 			iomap_dio_complete_work(&dio->aio.work);
 		}
 	}
@@ -492,8 +483,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 
 	dio->submit.iter = iter;
 	dio->submit.waiter = current;
-	dio->submit.cookie = BLK_QC_T_NONE;
-	dio->submit.last_queue = NULL;
+	dio->submit.poll_bio = NULL;
 
 	if (iov_iter_rw(iter) == READ) {
 		if (pos >= dio->i_size)
@@ -609,8 +599,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 	if (dio->flags & IOMAP_DIO_WRITE_FUA)
 		dio->flags &= ~IOMAP_DIO_NEED_SYNC;
 
-	WRITE_ONCE(iocb->ki_cookie, dio->submit.cookie);
-	WRITE_ONCE(iocb->private, dio->submit.last_queue);
+	WRITE_ONCE(iocb->private, dio->submit.poll_bio);
 
 	/*
 	 * We are about to drop our additional submission reference, which
@@ -637,10 +626,8 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 			if (!READ_ONCE(dio->submit.waiter))
 				break;
 
-			if (!(iocb->ki_flags & IOCB_HIPRI) ||
-			    !dio->submit.last_queue ||
-			    !blk_poll(dio->submit.last_queue,
-					 dio->submit.cookie, 0))
+			if (!dio->submit.poll_bio ||
+			    !bio_poll(dio->submit.poll_bio, 0))
 				blk_io_schedule();
 		}
 		__set_current_state(TASK_RUNNING);
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 396ef36dcd0a..d33f6e8bd313 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -1429,7 +1429,7 @@ const struct file_operations xfs_file_operations = {
 	.write_iter	= xfs_file_write_iter,
 	.splice_read	= generic_file_splice_read,
 	.splice_write	= iter_file_splice_write,
-	.iopoll		= iomap_dio_iopoll,
+	.iopoll		= iocb_bio_iopoll,
 	.unlocked_ioctl	= xfs_file_ioctl,
 #ifdef CONFIG_COMPAT
 	.compat_ioctl	= xfs_file_compat_ioctl,
diff --git a/fs/zonefs/super.c b/fs/zonefs/super.c
index cd145d318b17..2db3ba568bcf 100644
--- a/fs/zonefs/super.c
+++ b/fs/zonefs/super.c
@@ -1143,7 +1143,7 @@ static const struct file_operations zonefs_file_operations = {
 	.write_iter	= zonefs_file_write_iter,
 	.splice_read	= generic_file_splice_read,
 	.splice_write	= iter_file_splice_write,
-	.iopoll		= iomap_dio_iopoll,
+	.iopoll		= iocb_bio_iopoll,
 };
 
 static struct kmem_cache *zonefs_inode_cachep;
diff --git a/include/linux/bio.h b/include/linux/bio.h
index feaebd2367a0..bc6ad6d71208 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -422,7 +422,7 @@ static inline struct bio *bio_alloc(gfp_t gfp_mask, unsigned short nr_iovecs)
 	return bio_alloc_bioset(gfp_mask, nr_iovecs, &fs_bio_set);
 }
 
-extern blk_qc_t submit_bio(struct bio *);
+void submit_bio(struct bio *bio);
 
 extern void bio_endio(struct bio *);
 
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 02a4aab0aeac..21140132a30d 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -148,9 +148,9 @@ struct blk_mq_hw_ctx {
 	/** @kobj: Kernel object for sysfs. */
 	struct kobject		kobj;
 
-	/** @poll_considered: Count times blk_poll() was called. */
+	/** @poll_considered: Count times blk_mq_poll() was called. */
 	unsigned long		poll_considered;
-	/** @poll_invoked: Count how many requests blk_poll() polled. */
+	/** @poll_invoked: Count how many requests blk_mq_poll() polled. */
 	unsigned long		poll_invoked;
 	/** @poll_success: Count how many polled requests were completed. */
 	unsigned long		poll_success;
@@ -605,16 +605,6 @@ static inline void *blk_mq_rq_to_pdu(struct request *rq)
 	for ((i) = 0; (i) < (hctx)->nr_ctx &&				\
 	     ({ ctx = (hctx)->ctxs[(i)]; 1; }); (i)++)
 
-static inline blk_qc_t request_to_qc_t(struct blk_mq_hw_ctx *hctx,
-		struct request *rq)
-{
-	if (rq->tag != -1)
-		return rq->tag | (hctx->queue_num << BLK_QC_T_SHIFT);
-
-	return rq->internal_tag | (hctx->queue_num << BLK_QC_T_SHIFT) |
-			BLK_QC_T_INTERNAL;
-}
-
 static inline void blk_mq_cleanup_rq(struct request *rq)
 {
 	if (rq->q->mq_ops->cleanup_rq)
@@ -633,7 +623,6 @@ static inline void blk_rq_bio_prep(struct request *rq, struct bio *bio,
 		rq->rq_disk = bio->bi_bdev->bd_disk;
 }
 
-blk_qc_t blk_mq_submit_bio(struct bio *bio);
 void blk_mq_hctx_set_fq_lock_class(struct blk_mq_hw_ctx *hctx,
 		struct lock_class_key *key);
 
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index bb53aaa36caf..6e6c2af48d74 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -212,6 +212,9 @@ static inline void bio_issue_init(struct bio_issue *issue,
 			((u64)size << BIO_ISSUE_SIZE_SHIFT));
 }
 
+typedef unsigned int blk_qc_t;
+#define BLK_QC_T_NONE		-1U
+
 /*
  * main unit of I/O for the block layer and lower layers (ie drivers and
  * stacking drivers)
@@ -231,8 +234,8 @@ struct bio {
 
 	struct bvec_iter	bi_iter;
 
+	blk_qc_t		bi_cookie;
 	bio_end_io_t		*bi_end_io;
-
 	void			*bi_private;
 #ifdef CONFIG_BLK_CGROUP
 	/*
@@ -388,7 +391,7 @@ enum req_flag_bits {
 	/* command specific flags for REQ_OP_WRITE_ZEROES: */
 	__REQ_NOUNMAP,		/* do not free blocks when zeroing */
 
-	__REQ_POLLED,		/* caller polls for completion using blk_poll */
+	__REQ_POLLED,		/* caller polls for completion using bio_poll */
 
 	/* for driver use */
 	__REQ_DRV,
@@ -501,11 +504,6 @@ static inline int op_stat_group(unsigned int op)
 	return op_is_write(op);
 }
 
-typedef unsigned int blk_qc_t;
-#define BLK_QC_T_NONE		-1U
-#define BLK_QC_T_SHIFT		16
-#define BLK_QC_T_INTERNAL	(1U << 31)
-
 struct blk_rq_stat {
 	u64 mean;
 	u64 min;
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 821d721f78e7..561b04117bd4 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -38,6 +38,7 @@ struct sg_io_hdr;
 struct bsg_job;
 struct blkcg_gq;
 struct blk_flush_queue;
+struct kiocb;
 struct pr_ops;
 struct rq_qos;
 struct blk_queue_stats;
@@ -900,7 +901,7 @@ static inline void rq_flush_dcache_pages(struct request *rq)
 
 extern int blk_register_queue(struct gendisk *disk);
 extern void blk_unregister_queue(struct gendisk *disk);
-blk_qc_t submit_bio_noacct(struct bio *bio);
+void submit_bio_noacct(struct bio *bio);
 extern void blk_rq_init(struct request_queue *q, struct request *rq);
 extern void blk_put_request(struct request *);
 extern struct request *blk_get_request(struct request_queue *, unsigned int op,
@@ -950,7 +951,8 @@ blk_status_t errno_to_blk_status(int errno);
 #define BLK_POLL_ONESHOT		(1 << 0)
 /* do not sleep to wait for the expected completion time */
 #define BLK_POLL_NOSLEEP		(1 << 1)
-int blk_poll(struct request_queue *q, blk_qc_t cookie, unsigned int flags);
+int bio_poll(struct bio *bio, unsigned int flags);
+int iocb_bio_iopoll(struct kiocb *kiocb, unsigned int flags);
 
 static inline struct request_queue *bdev_get_queue(struct block_device *bdev)
 {
@@ -1854,7 +1856,7 @@ static inline void blk_ksm_unregister(struct request_queue *q) { }
 
 
 struct block_device_operations {
-	blk_qc_t (*submit_bio) (struct bio *bio);
+	void (*submit_bio)(struct bio *bio);
 	int (*open) (struct block_device *, fmode_t);
 	void (*release) (struct gendisk *, fmode_t);
 	int (*rw_page)(struct block_device *, sector_t, struct page *, unsigned int);
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 483fb557d92f..876ec2b11a7d 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -332,11 +332,7 @@ struct kiocb {
 	int			ki_flags;
 	u16			ki_hint;
 	u16			ki_ioprio; /* See linux/ioprio.h */
-	union {
-		unsigned int		ki_cookie; /* for ->iopoll */
-		struct wait_page_queue	*ki_waitq; /* for async buffered IO */
-	};
-
+	struct wait_page_queue	*ki_waitq; /* for async buffered IO */
 	randomized_struct_fields_end
 };
 
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 56e5949ccb60..ac825a498816 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -251,7 +251,7 @@ int iomap_writepages(struct address_space *mapping,
 struct iomap_dio_ops {
 	int (*end_io)(struct kiocb *iocb, ssize_t size, int error,
 		      unsigned flags);
-	blk_qc_t (*submit_io)(struct inode *inode, struct iomap *iomap,
+	void (*submit_io)(struct inode *inode, struct iomap *iomap,
 			struct bio *bio, loff_t file_offset);
 };
 
@@ -275,7 +275,6 @@ struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 		const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
 		unsigned int dio_flags);
 ssize_t iomap_dio_complete(struct iomap_dio *dio);
-int iomap_dio_iopoll(struct kiocb *kiocb, unsigned int flags);
 
 #ifdef CONFIG_SWAP
 struct file;
diff --git a/mm/page_io.c b/mm/page_io.c
index ed2eded74f3a..a68faab5b310 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -358,8 +358,6 @@ int swap_readpage(struct page *page, bool synchronous)
 	struct bio *bio;
 	int ret = 0;
 	struct swap_info_struct *sis = page_swap_info(page);
-	blk_qc_t qc;
-	struct gendisk *disk;
 	unsigned long pflags;
 
 	VM_BUG_ON_PAGE(!PageSwapCache(page) && !synchronous, page);
@@ -409,8 +407,6 @@ int swap_readpage(struct page *page, bool synchronous)
 	bio->bi_iter.bi_sector = swap_page_sector(page);
 	bio->bi_end_io = end_swap_bio_read;
 	bio_add_page(bio, page, thp_size(page), 0);
-
-	disk = bio->bi_bdev->bd_disk;
 	/*
 	 * Keep this task valid during swap readpage because the oom killer may
 	 * attempt to access it in the page fault retry time check.
@@ -422,13 +418,13 @@ int swap_readpage(struct page *page, bool synchronous)
 	}
 	count_vm_event(PSWPIN);
 	bio_get(bio);
-	qc = submit_bio(bio);
+	submit_bio(bio);
 	while (synchronous) {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		if (!READ_ONCE(bio->bi_private))
 			break;
 
-		if (!blk_poll(disk->queue, qc, 0))
+		if (!bio_poll(bio, 0))
 			blk_io_schedule();
 	}
 	__set_current_state(TASK_RUNNING);
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:30 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:30 +0200
Subject: [PATCH 12/16] block: define 'struct bvec_iter' as packed
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-13-hch@lst.de>

From: Ming Lei <ming.lei at redhat.com>

'struct bvec_iter' is embedded into 'struct bio', define it as packed
so that we can get one extra 4bytes for other uses without expanding
bio.

'struct bvec_iter' is often allocated on stack, so making it packed
doesn't affect performance. Also I have run io_uring on both
nvme/null_blk, and not observe performance effect in this way.

Suggested-by: Christoph Hellwig <hch at lst.de>
Signed-off-by: Ming Lei <ming.lei at redhat.com>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Christoph Hellwig <hch at lst.de>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 include/linux/bvec.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/bvec.h b/include/linux/bvec.h
index ff832e698efb..a0c4f41dfc83 100644
--- a/include/linux/bvec.h
+++ b/include/linux/bvec.h
@@ -43,7 +43,7 @@ struct bvec_iter {
 
 	unsigned int            bi_bvec_done;	/* number of bytes completed in
 						   current bvec */
-};
+} __packed;
 
 struct bvec_iter_all {
 	struct bio_vec	bv;
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:29 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:29 +0200
Subject: [PATCH 11/16] block: use SLAB_TYPESAFE_BY_RCU for the bio slab
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-12-hch@lst.de>

This flags ensures that the pages will not be reused for non-bio
allocations before the end of an RCU grace period.  With that we can
safely use a RCU lookup for bio polling as long as we are fine with
occasionally polling the wrong device.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 block/bio.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/block/bio.c b/block/bio.c
index 44205dfb6b60..de5505d7018e 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -82,7 +82,8 @@ static struct bio_slab *create_bio_slab(unsigned int size)
 
 	snprintf(bslab->name, sizeof(bslab->name), "bio-%d", size);
 	bslab->slab = kmem_cache_create(bslab->name, size,
-			ARCH_KMALLOC_MINALIGN, SLAB_HWCACHE_ALIGN, NULL);
+			ARCH_KMALLOC_MINALIGN,
+			SLAB_HWCACHE_ALIGN | SLAB_TYPESAFE_BY_RCU, NULL);
 	if (!bslab->slab)
 		goto fail_alloc_slab;
 
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:28 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:28 +0200
Subject: [PATCH 10/16] block: rename REQ_HIPRI to REQ_POLLED
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-11-hch@lst.de>

Unlike the RWF_HIPRI userspace ABI which is intentionally kept vague,
the bio flag is specific to the polling implementation, so rename and
document it properly.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 block/blk-core.c          |  2 +-
 block/blk-merge.c         |  2 +-
 block/blk-mq-debugfs.c    |  2 +-
 block/blk-mq.c            |  4 ++--
 block/blk-mq.h            |  4 ++--
 drivers/nvme/host/core.c  |  2 +-
 drivers/scsi/scsi_debug.c | 10 +++++-----
 include/linux/bio.h       |  2 +-
 include/linux/blk_types.h |  4 ++--
 mm/page_io.c              |  2 +-
 10 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c
index 514838ccab2d..858a76858cb0 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -836,7 +836,7 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio)
 	}
 
 	if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
-		bio->bi_opf &= ~REQ_HIPRI;
+		bio->bi_opf &= ~REQ_POLLED;
 
 	switch (bio_op(bio)) {
 	case REQ_OP_DISCARD:
diff --git a/block/blk-merge.c b/block/blk-merge.c
index 4d97fb6dd226..5c9d2a4ece86 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -285,7 +285,7 @@ static struct bio *blk_bio_segment_split(struct request_queue *q,
 	 * iopoll in direct IO routine. Given performance gain of iopoll for
 	 * big IO can be trival, disable iopoll when split needed.
 	 */
-	bio->bi_opf &= ~REQ_HIPRI;
+	bio->bi_opf &= ~REQ_POLLED;
 
 	return bio_split(bio, sectors, GFP_NOIO, bs);
 }
diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c
index 2a75bc7401df..0b821b369ffd 100644
--- a/block/blk-mq-debugfs.c
+++ b/block/blk-mq-debugfs.c
@@ -286,7 +286,7 @@ static const char *const cmd_flag_name[] = {
 	CMD_FLAG_NAME(BACKGROUND),
 	CMD_FLAG_NAME(NOWAIT),
 	CMD_FLAG_NAME(NOUNMAP),
-	CMD_FLAG_NAME(HIPRI),
+	CMD_FLAG_NAME(POLLED),
 };
 #undef CMD_FLAG_NAME
 
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 86e59ffcca44..a0cc1eedc15a 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -667,7 +667,7 @@ bool blk_mq_complete_request_remote(struct request *rq)
 	 * For a polled request, always complete locallly, it's pointless
 	 * to redirect the completion.
 	 */
-	if (rq->cmd_flags & REQ_HIPRI)
+	if (rq->cmd_flags & REQ_POLLED)
 		return false;
 
 	if (blk_mq_complete_need_ipi(rq)) {
@@ -2220,7 +2220,7 @@ blk_qc_t blk_mq_submit_bio(struct bio *bio)
 
 	rq_qos_throttle(q, bio);
 
-	hipri = bio->bi_opf & REQ_HIPRI;
+	hipri = bio->bi_opf & REQ_POLLED;
 
 	data.cmd_flags = bio->bi_opf;
 	rq = __blk_mq_alloc_request(&data);
diff --git a/block/blk-mq.h b/block/blk-mq.h
index 4b1ca7b7bbeb..8444e00e8d25 100644
--- a/block/blk-mq.h
+++ b/block/blk-mq.h
@@ -109,9 +109,9 @@ static inline struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *q,
 	enum hctx_type type = HCTX_TYPE_DEFAULT;
 
 	/*
-	 * The caller ensure that if REQ_HIPRI, poll must be enabled.
+	 * The caller ensure that if REQ_POLLED, poll must be enabled.
 	 */
-	if (flags & REQ_HIPRI)
+	if (flags & REQ_POLLED)
 		type = HCTX_TYPE_POLL;
 	else if ((flags & REQ_OP_MASK) == REQ_OP_READ)
 		type = HCTX_TYPE_READ;
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 115c8d90530a..1f2ecb2fe9b1 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1027,7 +1027,7 @@ static void nvme_execute_rq_polled(struct request_queue *q,
 
 	WARN_ON_ONCE(!test_bit(QUEUE_FLAG_POLL, &q->queue_flags));
 
-	rq->cmd_flags |= REQ_HIPRI;
+	rq->cmd_flags |= REQ_POLLED;
 	rq->end_io_data = &wait;
 	blk_execute_rq_nowait(bd_disk, rq, at_head, nvme_end_sync_rq);
 
diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c
index a5d1633b5bd8..81384d28ff90 100644
--- a/drivers/scsi/scsi_debug.c
+++ b/drivers/scsi/scsi_debug.c
@@ -5367,7 +5367,7 @@ static int schedule_resp(struct scsi_cmnd *cmnd, struct sdebug_dev_info *devip,
 {
 	bool new_sd_dp;
 	bool inject = false;
-	bool hipri = (cmnd->request->cmd_flags & REQ_HIPRI);
+	bool polled = (cmnd->request->cmd_flags & REQ_POLLED);
 	int k, num_in_q, qdepth;
 	unsigned long iflags;
 	u64 ns_from_boot = 0;
@@ -5454,7 +5454,7 @@ static int schedule_resp(struct scsi_cmnd *cmnd, struct sdebug_dev_info *devip,
 	if (sdebug_host_max_queue)
 		sd_dp->hc_idx = get_tag(cmnd);
 
-	if (hipri)
+	if (polled)
 		ns_from_boot = ktime_get_boottime_ns();
 
 	/* one of the resp_*() response functions is called here */
@@ -5514,7 +5514,7 @@ static int schedule_resp(struct scsi_cmnd *cmnd, struct sdebug_dev_info *devip,
 				kt -= d;
 			}
 		}
-		if (hipri) {
+		if (polled) {
 			sd_dp->cmpl_ts = ktime_add(ns_to_ktime(ns_from_boot), kt);
 			spin_lock_irqsave(&sqp->qc_lock, iflags);
 			if (!sd_dp->init_poll) {
@@ -5545,7 +5545,7 @@ static int schedule_resp(struct scsi_cmnd *cmnd, struct sdebug_dev_info *devip,
 		if (unlikely((sdebug_opts & SDEBUG_OPT_CMD_ABORT) &&
 			     atomic_read(&sdeb_inject_pending)))
 			sd_dp->aborted = true;
-		if (hipri) {
+		if (polled) {
 			sd_dp->cmpl_ts = ns_to_ktime(ns_from_boot);
 			spin_lock_irqsave(&sqp->qc_lock, iflags);
 			if (!sd_dp->init_poll) {
@@ -7313,7 +7313,7 @@ static int sdebug_blk_mq_poll(struct Scsi_Host *shost, unsigned int queue_num)
 			if (kt_from_boot < sd_dp->cmpl_ts)
 				continue;
 
-		} else		/* ignoring non REQ_HIPRI requests */
+		} else		/* ignoring non REQ_POLLED requests */
 			continue;
 		devip = (struct sdebug_dev_info *)scp->device->hostdata;
 		if (likely(devip))
diff --git a/include/linux/bio.h b/include/linux/bio.h
index d2b98efb5cc5..feaebd2367a0 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -813,7 +813,7 @@ static inline int bio_integrity_add_page(struct bio *bio, struct page *page,
  */
 static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb)
 {
-	bio->bi_opf |= REQ_HIPRI;
+	bio->bi_opf |= REQ_POLLED;
 	if (!is_sync_kiocb(kiocb))
 		bio->bi_opf |= REQ_NOWAIT;
 }
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index 295addd38390..bb53aaa36caf 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -388,7 +388,7 @@ enum req_flag_bits {
 	/* command specific flags for REQ_OP_WRITE_ZEROES: */
 	__REQ_NOUNMAP,		/* do not free blocks when zeroing */
 
-	__REQ_HIPRI,
+	__REQ_POLLED,		/* caller polls for completion using blk_poll */
 
 	/* for driver use */
 	__REQ_DRV,
@@ -413,7 +413,7 @@ enum req_flag_bits {
 #define REQ_CGROUP_PUNT		(1ULL << __REQ_CGROUP_PUNT)
 
 #define REQ_NOUNMAP		(1ULL << __REQ_NOUNMAP)
-#define REQ_HIPRI		(1ULL << __REQ_HIPRI)
+#define REQ_POLLED		(1ULL << __REQ_POLLED)
 
 #define REQ_DRV			(1ULL << __REQ_DRV)
 #define REQ_SWAP		(1ULL << __REQ_SWAP)
diff --git a/mm/page_io.c b/mm/page_io.c
index 5d5543fcefa4..ed2eded74f3a 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -416,7 +416,7 @@ int swap_readpage(struct page *page, bool synchronous)
 	 * attempt to access it in the page fault retry time check.
 	 */
 	if (synchronous) {
-		bio->bi_opf |= REQ_HIPRI;
+		bio->bi_opf |= REQ_POLLED;
 		get_task_struct(current);
 		bio->bi_private = current;
 	}
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:27 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:27 +0200
Subject: [PATCH 09/16] io_uring: don't sleep when polling for I/O
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-10-hch@lst.de>

There is no point in sleeping for the expected I/O completion timeout
in the io_uring async polling model as we never poll for a specific
I/O.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 block/blk-mq.c         | 3 ++-
 fs/io_uring.c          | 2 +-
 include/linux/blkdev.h | 2 ++
 3 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index c2b2c7abd712..86e59ffcca44 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -4002,7 +4002,8 @@ int blk_poll(struct request_queue *q, blk_qc_t cookie, unsigned int flags)
 	if (current->plug)
 		blk_flush_plug_list(current->plug, false);
 
-	if (q->poll_nsec != BLK_MQ_POLL_CLASSIC) {
+	if (!(flags & BLK_POLL_NOSLEEP) &&
+	    q->poll_nsec != BLK_MQ_POLL_CLASSIC) {
 		if (blk_mq_poll_hybrid(q, cookie))
 			return 1;
 	}
diff --git a/fs/io_uring.c b/fs/io_uring.c
index de8d39e9a154..6ce1e5b60725 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -2290,7 +2290,7 @@ static void io_iopoll_complete(struct io_ring_ctx *ctx, unsigned int *nr_events,
 static int io_do_iopoll(struct io_ring_ctx *ctx, unsigned int *nr_events,
 			long min)
 {
-	unsigned int poll_flags = 0;
+	unsigned int poll_flags = BLK_POLL_NOSLEEP;
 	struct io_kiocb *req, *tmp;
 	LIST_HEAD(done);
 	int ret = 0;
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index c7061f25ee5f..821d721f78e7 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -948,6 +948,8 @@ blk_status_t errno_to_blk_status(int errno);
 
 /* only poll the hardware once, don't continue until a completion was found */
 #define BLK_POLL_ONESHOT		(1 << 0)
+/* do not sleep to wait for the expected completion time */
+#define BLK_POLL_NOSLEEP		(1 << 1)
 int blk_poll(struct request_queue *q, blk_qc_t cookie, unsigned int flags);
 
 static inline struct request_queue *bdev_get_queue(struct block_device *bdev)
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:26 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:26 +0200
Subject: [PATCH 08/16] block: replace the spin argument to blk_iopoll with a
 flags argument
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-9-hch@lst.de>

Switch the boolean spin argument to blk_poll to passing a set of flags
instead.  This will allow to control polling behavior in a more fine
grained way.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 block/blk-mq.c           | 17 +++++++----------
 drivers/nvme/host/core.c |  2 +-
 fs/block_dev.c           |  8 ++++----
 fs/io_uring.c            | 14 +++++++-------
 fs/iomap/direct-io.c     |  6 +++---
 include/linux/blkdev.h   |  4 +++-
 include/linux/fs.h       |  2 +-
 include/linux/iomap.h    |  2 +-
 mm/page_io.c             |  2 +-
 9 files changed, 28 insertions(+), 29 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index cb94cf9bcf99..c2b2c7abd712 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -3951,7 +3951,7 @@ static bool blk_mq_poll_hybrid(struct request_queue *q, blk_qc_t qc)
 }
 
 static int blk_mq_poll_classic(struct request_queue *q, blk_qc_t cookie,
-		bool spin)
+		unsigned int flags)
 {
 	struct blk_mq_hw_ctx *hctx = blk_qc_to_hctx(q, cookie);
 	long state = current->state;
@@ -3974,7 +3974,7 @@ static int blk_mq_poll_classic(struct request_queue *q, blk_qc_t cookie,
 		if (current->state == TASK_RUNNING)
 			return 1;
 
-		if (ret < 0 || !spin)
+		if (ret < 0 || (flags & BLK_POLL_ONESHOT))
 			break;
 		cpu_relax();
 	} while (!need_resched());
@@ -3987,15 +3987,13 @@ static int blk_mq_poll_classic(struct request_queue *q, blk_qc_t cookie,
  * blk_poll - poll for IO completions
  * @q:  the queue
  * @cookie: cookie passed back at IO submission time
- * @spin: whether to spin for completions
+ * @flags: BLK_POLL_* flags that control the behavior
  *
  * Description:
  *    Poll for completions on the passed in queue. Returns number of
- *    completed entries found. If @spin is true, then blk_poll will continue
- *    looping until at least one completion is found, unless the task is
- *    otherwise marked running (or we need to reschedule).
+ *    completed entries found.
  */
-int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin)
+int blk_poll(struct request_queue *q, blk_qc_t cookie, unsigned int flags)
 {
 	if (cookie == BLK_QC_T_NONE ||
 	    !test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
@@ -4004,12 +4002,11 @@ int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin)
 	if (current->plug)
 		blk_flush_plug_list(current->plug, false);
 
-	/* If specified not to spin, we also should not sleep. */
-	if (spin && q->poll_nsec != BLK_MQ_POLL_CLASSIC) {
+	if (q->poll_nsec != BLK_MQ_POLL_CLASSIC) {
 		if (blk_mq_poll_hybrid(q, cookie))
 			return 1;
 	}
-	return blk_mq_poll_classic(q, cookie, spin);
+	return blk_mq_poll_classic(q, cookie, flags);
 }
 EXPORT_SYMBOL_GPL(blk_poll);
 
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 24bcae88587a..115c8d90530a 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1032,7 +1032,7 @@ static void nvme_execute_rq_polled(struct request_queue *q,
 	blk_execute_rq_nowait(bd_disk, rq, at_head, nvme_end_sync_rq);
 
 	while (!completion_done(&wait)) {
-		blk_poll(q, request_to_qc_t(rq->mq_hctx, rq), true);
+		blk_poll(q, request_to_qc_t(rq->mq_hctx, rq), 0);
 		cond_resched();
 	}
 }
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 8600c651b0b0..c9bdaa138909 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -286,7 +286,7 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter,
 		if (!READ_ONCE(bio.bi_private))
 			break;
 		if (!(iocb->ki_flags & IOCB_HIPRI) ||
-		    !blk_poll(bdev_get_queue(bdev), qc, true))
+		    !blk_poll(bdev_get_queue(bdev), qc, 0))
 			blk_io_schedule();
 	}
 	__set_current_state(TASK_RUNNING);
@@ -319,12 +319,12 @@ struct blkdev_dio {
 
 static struct bio_set blkdev_dio_pool;
 
-static int blkdev_iopoll(struct kiocb *kiocb, bool wait)
+static int blkdev_iopoll(struct kiocb *kiocb, unsigned int flags)
 {
 	struct block_device *bdev = I_BDEV(kiocb->ki_filp->f_mapping->host);
 	struct request_queue *q = bdev_get_queue(bdev);
 
-	return blk_poll(q, READ_ONCE(kiocb->ki_cookie), wait);
+	return blk_poll(q, READ_ONCE(kiocb->ki_cookie), flags);
 }
 
 static void blkdev_bio_end_io(struct bio *bio)
@@ -475,7 +475,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 		if (!READ_ONCE(dio->waiter))
 			break;
 
-		if (!do_poll || !blk_poll(bdev_get_queue(bdev), qc, true))
+		if (!do_poll || !blk_poll(bdev_get_queue(bdev), qc, 0))
 			blk_io_schedule();
 	}
 	__set_current_state(TASK_RUNNING);
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 5f82954004f6..de8d39e9a154 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -2290,18 +2290,18 @@ static void io_iopoll_complete(struct io_ring_ctx *ctx, unsigned int *nr_events,
 static int io_do_iopoll(struct io_ring_ctx *ctx, unsigned int *nr_events,
 			long min)
 {
+	unsigned int poll_flags = 0;
 	struct io_kiocb *req, *tmp;
 	LIST_HEAD(done);
-	bool spin;
-	int ret;
+	int ret = 0;
 
 	/*
 	 * Only spin for completions if we don't have multiple devices hanging
 	 * off our complete list, and we're under the requested amount.
 	 */
-	spin = !ctx->poll_multi_file && *nr_events < min;
+	if (ctx->poll_multi_file || *nr_events >= min)
+		poll_flags |= BLK_POLL_ONESHOT;
 
-	ret = 0;
 	list_for_each_entry_safe(req, tmp, &ctx->iopoll_list, inflight_entry) {
 		struct kiocb *kiocb = &req->rw.kiocb;
 
@@ -2317,7 +2317,7 @@ static int io_do_iopoll(struct io_ring_ctx *ctx, unsigned int *nr_events,
 		if (!list_empty(&done))
 			break;
 
-		ret = kiocb->ki_filp->f_op->iopoll(kiocb, spin);
+		ret = kiocb->ki_filp->f_op->iopoll(kiocb, poll_flags);
 		if (ret < 0)
 			break;
 
@@ -2325,8 +2325,8 @@ static int io_do_iopoll(struct io_ring_ctx *ctx, unsigned int *nr_events,
 		if (READ_ONCE(req->iopoll_completed))
 			list_move_tail(&req->inflight_entry, &done);
 
-		if (ret && spin)
-			spin = false;
+		if (ret)
+			poll_flags |= BLK_POLL_ONESHOT;
 		ret = 0;
 	}
 
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index d5637f467109..9b6c26da3a2d 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -49,13 +49,13 @@ struct iomap_dio {
 	};
 };
 
-int iomap_dio_iopoll(struct kiocb *kiocb, bool spin)
+int iomap_dio_iopoll(struct kiocb *kiocb, unsigned int flags)
 {
 	struct request_queue *q = READ_ONCE(kiocb->private);
 
 	if (!q)
 		return 0;
-	return blk_poll(q, READ_ONCE(kiocb->ki_cookie), spin);
+	return blk_poll(q, READ_ONCE(kiocb->ki_cookie), flags);
 }
 EXPORT_SYMBOL_GPL(iomap_dio_iopoll);
 
@@ -640,7 +640,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 			if (!(iocb->ki_flags & IOCB_HIPRI) ||
 			    !dio->submit.last_queue ||
 			    !blk_poll(dio->submit.last_queue,
-					 dio->submit.cookie, true))
+					 dio->submit.cookie, 0))
 				blk_io_schedule();
 		}
 		__set_current_state(TASK_RUNNING);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index d66d0da72529..c7061f25ee5f 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -946,7 +946,9 @@ extern const char *blk_op_str(unsigned int op);
 int blk_status_to_errno(blk_status_t status);
 blk_status_t errno_to_blk_status(int errno);
 
-int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin);
+/* only poll the hardware once, don't continue until a completion was found */
+#define BLK_POLL_ONESHOT		(1 << 0)
+int blk_poll(struct request_queue *q, blk_qc_t cookie, unsigned int flags);
 
 static inline struct request_queue *bdev_get_queue(struct block_device *bdev)
 {
diff --git a/include/linux/fs.h b/include/linux/fs.h
index c3c88fdb9b2a..483fb557d92f 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -2026,7 +2026,7 @@ struct file_operations {
 	ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
 	ssize_t (*read_iter) (struct kiocb *, struct iov_iter *);
 	ssize_t (*write_iter) (struct kiocb *, struct iov_iter *);
-	int (*iopoll)(struct kiocb *kiocb, bool spin);
+	int (*iopoll)(struct kiocb *kiocb, unsigned int flags);
 	int (*iterate) (struct file *, struct dir_context *);
 	int (*iterate_shared) (struct file *, struct dir_context *);
 	__poll_t (*poll) (struct file *, struct poll_table_struct *);
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index c87d0cb0de6d..56e5949ccb60 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -275,7 +275,7 @@ struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 		const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
 		unsigned int dio_flags);
 ssize_t iomap_dio_complete(struct iomap_dio *dio);
-int iomap_dio_iopoll(struct kiocb *kiocb, bool spin);
+int iomap_dio_iopoll(struct kiocb *kiocb, unsigned int flags);
 
 #ifdef CONFIG_SWAP
 struct file;
diff --git a/mm/page_io.c b/mm/page_io.c
index c493ce9ebcf5..5d5543fcefa4 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -428,7 +428,7 @@ int swap_readpage(struct page *page, bool synchronous)
 		if (!READ_ONCE(bio->bi_private))
 			break;
 
-		if (!blk_poll(disk->queue, qc, true))
+		if (!blk_poll(disk->queue, qc, 0))
 			blk_io_schedule();
 	}
 	__set_current_state(TASK_RUNNING);
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:25 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:25 +0200
Subject: [PATCH 07/16] blk-mq: remove blk_qc_t_valid
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-8-hch@lst.de>

Move the trivial check into the only caller.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 block/blk-mq.c            | 2 +-
 include/linux/blk_types.h | 5 -----
 2 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index eb861839ff49..cb94cf9bcf99 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -3997,7 +3997,7 @@ static int blk_mq_poll_classic(struct request_queue *q, blk_qc_t cookie,
  */
 int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin)
 {
-	if (!blk_qc_t_valid(cookie) ||
+	if (cookie == BLK_QC_T_NONE ||
 	    !test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
 		return 0;
 
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index abe00ac81b2e..295addd38390 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -506,11 +506,6 @@ typedef unsigned int blk_qc_t;
 #define BLK_QC_T_SHIFT		16
 #define BLK_QC_T_INTERNAL	(1U << 31)
 
-static inline bool blk_qc_t_valid(blk_qc_t cookie)
-{
-	return cookie != BLK_QC_T_NONE;
-}
-
 struct blk_rq_stat {
 	u64 mean;
 	u64 min;
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:24 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:24 +0200
Subject: [PATCH 06/16] blk-mq: remove blk_qc_t_to_tag and blk_qc_t_is_internal
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-7-hch@lst.de>

Merge both functions into their only caller to keep the blk-mq tag to
blk_qc_t mapping as private as possible in blk-mq.c.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 block/blk-mq.c            |  8 +++++---
 include/linux/blk_types.h | 10 ----------
 2 files changed, 5 insertions(+), 13 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index f4bb6e1db36a..eb861839ff49 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -72,9 +72,11 @@ static inline struct blk_mq_hw_ctx *blk_qc_to_hctx(struct request_queue *q,
 static inline struct request *blk_qc_to_rq(struct blk_mq_hw_ctx *hctx,
 		blk_qc_t qc)
 {
-	if (blk_qc_t_is_internal(qc))
-		return blk_mq_tag_to_rq(hctx->sched_tags, blk_qc_t_to_tag(qc));
-	return blk_mq_tag_to_rq(hctx->tags, blk_qc_t_to_tag(qc));
+	unsigned int tag = qc & ((1U << BLK_QC_T_SHIFT) - 1);
+
+	if (qc & BLK_QC_T_INTERNAL)
+		return blk_mq_tag_to_rq(hctx->sched_tags, tag);
+	return blk_mq_tag_to_rq(hctx->tags, tag);
 }
 
 /*
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index e2c4a7d5de05..abe00ac81b2e 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -511,16 +511,6 @@ static inline bool blk_qc_t_valid(blk_qc_t cookie)
 	return cookie != BLK_QC_T_NONE;
 }
 
-static inline unsigned int blk_qc_t_to_tag(blk_qc_t cookie)
-{
-	return cookie & ((1u << BLK_QC_T_SHIFT) - 1);
-}
-
-static inline bool blk_qc_t_is_internal(blk_qc_t cookie)
-{
-	return (cookie & BLK_QC_T_INTERNAL) != 0;
-}
-
 struct blk_rq_stat {
 	u64 mean;
 	u64 min;
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:23 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:23 +0200
Subject: [PATCH 05/16] blk-mq: factor out a "classic" poll helper
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-6-hch@lst.de>

Factor the code to do the classic full metal polling out of blk_poll into
a separate blk_mq_poll_classic helper.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 block/blk-mq.c | 120 +++++++++++++++++++++++--------------------------
 1 file changed, 56 insertions(+), 64 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 9b3fe1062061..f4bb6e1db36a 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -69,6 +69,14 @@ static inline struct blk_mq_hw_ctx *blk_qc_to_hctx(struct request_queue *q,
 	return q->queue_hw_ctx[(qc & ~BLK_QC_T_INTERNAL) >> BLK_QC_T_SHIFT];
 }
 
+static inline struct request *blk_qc_to_rq(struct blk_mq_hw_ctx *hctx,
+		blk_qc_t qc)
+{
+	if (blk_qc_t_is_internal(qc))
+		return blk_mq_tag_to_rq(hctx->sched_tags, blk_qc_t_to_tag(qc));
+	return blk_mq_tag_to_rq(hctx->tags, blk_qc_t_to_tag(qc));
+}
+
 /*
  * Check if any of the ctx, dispatch list or elevator
  * have pending work in this hardware queue.
@@ -3874,15 +3882,20 @@ static unsigned long blk_mq_poll_nsecs(struct request_queue *q,
 	return ret;
 }
 
-static bool blk_mq_poll_hybrid_sleep(struct request_queue *q,
-				     struct request *rq)
+static bool blk_mq_poll_hybrid(struct request_queue *q, blk_qc_t qc)
 {
+	struct blk_mq_hw_ctx *hctx = blk_qc_to_hctx(q, qc);
+	struct request *rq = blk_qc_to_rq(hctx, qc);
 	struct hrtimer_sleeper hs;
 	enum hrtimer_mode mode;
 	unsigned int nsecs;
 	ktime_t kt;
 
-	if (rq->rq_flags & RQF_MQ_POLL_SLEPT)
+	/*
+	 * If a request has completed on queue that uses an I/O scheduler, we
+	 * won't get back a request from blk_qc_to_rq.
+	 */
+	if (!rq || (rq->rq_flags & RQF_MQ_POLL_SLEPT))
 		return false;
 
 	/*
@@ -3924,32 +3937,48 @@ static bool blk_mq_poll_hybrid_sleep(struct request_queue *q,
 
 	__set_current_state(TASK_RUNNING);
 	destroy_hrtimer_on_stack(&hs.timer);
+
+	/*
+	 * If we sleep, have the caller restart the poll loop to reset the
+	 * state.  Like for the other success return cases, the caller is
+	 * responsible for checking if the IO completed.  If the IO isn't
+	 * complete, we'll get called again and will go straight to the busy
+	 * poll loop.
+	 */
 	return true;
 }
 
-static bool blk_mq_poll_hybrid(struct request_queue *q,
-			       struct blk_mq_hw_ctx *hctx, blk_qc_t cookie)
+static int blk_mq_poll_classic(struct request_queue *q, blk_qc_t cookie,
+		bool spin)
 {
-	struct request *rq;
+	struct blk_mq_hw_ctx *hctx = blk_qc_to_hctx(q, cookie);
+	long state = current->state;
+	int ret;
 
-	if (q->poll_nsec == BLK_MQ_POLL_CLASSIC)
-		return false;
+	hctx->poll_considered++;
 
-	if (!blk_qc_t_is_internal(cookie))
-		rq = blk_mq_tag_to_rq(hctx->tags, blk_qc_t_to_tag(cookie));
-	else {
-		rq = blk_mq_tag_to_rq(hctx->sched_tags, blk_qc_t_to_tag(cookie));
-		/*
-		 * With scheduling, if the request has completed, we'll
-		 * get a NULL return here, as we clear the sched tag when
-		 * that happens. The request still remains valid, like always,
-		 * so we should be safe with just the NULL check.
-		 */
-		if (!rq)
-			return false;
-	}
+	do {
+		hctx->poll_invoked++;
 
-	return blk_mq_poll_hybrid_sleep(q, rq);
+		ret = q->mq_ops->poll(hctx);
+		if (ret > 0) {
+			hctx->poll_success++;
+			__set_current_state(TASK_RUNNING);
+			return ret;
+		}
+
+		if (signal_pending_state(state, current))
+			__set_current_state(TASK_RUNNING);
+		if (current->state == TASK_RUNNING)
+			return 1;
+
+		if (ret < 0 || !spin)
+			break;
+		cpu_relax();
+	} while (!need_resched());
+
+	__set_current_state(TASK_RUNNING);
+	return 0;
 }
 
 /**
@@ -3966,9 +3995,6 @@ static bool blk_mq_poll_hybrid(struct request_queue *q,
  */
 int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin)
 {
-	struct blk_mq_hw_ctx *hctx;
-	long state;
-
 	if (!blk_qc_t_valid(cookie) ||
 	    !test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
 		return 0;
@@ -3976,46 +4002,12 @@ int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin)
 	if (current->plug)
 		blk_flush_plug_list(current->plug, false);
 
-	hctx = blk_qc_to_hctx(q, cookie);
-
-	/*
-	 * If we sleep, have the caller restart the poll loop to reset
-	 * the state. Like for the other success return cases, the
-	 * caller is responsible for checking if the IO completed. If
-	 * the IO isn't complete, we'll get called again and will go
-	 * straight to the busy poll loop. If specified not to spin,
-	 * we also should not sleep.
-	 */
-	if (spin && blk_mq_poll_hybrid(q, hctx, cookie))
-		return 1;
-
-	hctx->poll_considered++;
-
-	state = current->state;
-	do {
-		int ret;
-
-		hctx->poll_invoked++;
-
-		ret = q->mq_ops->poll(hctx);
-		if (ret > 0) {
-			hctx->poll_success++;
-			__set_current_state(TASK_RUNNING);
-			return ret;
-		}
-
-		if (signal_pending_state(state, current))
-			__set_current_state(TASK_RUNNING);
-
-		if (current->state == TASK_RUNNING)
+	/* If specified not to spin, we also should not sleep. */
+	if (spin && q->poll_nsec != BLK_MQ_POLL_CLASSIC) {
+		if (blk_mq_poll_hybrid(q, cookie))
 			return 1;
-		if (ret < 0 || !spin)
-			break;
-		cpu_relax();
-	} while (!need_resched());
-
-	__set_current_state(TASK_RUNNING);
-	return 0;
+	}
+	return blk_mq_poll_classic(q, cookie, spin);
 }
 EXPORT_SYMBOL_GPL(blk_poll);
 
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:22 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:22 +0200
Subject: [PATCH 04/16] blk-mq: factor out a blk_qc_to_hctx helper
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-5-hch@lst.de>

Add a helper to get the hctx from a request_queue and cookie, and fold
the blk_qc_t_to_queue_num helper into it as no other callers are left.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 block/blk-mq.c            | 8 +++++++-
 include/linux/blk_types.h | 5 -----
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 3115ea2d0990..9b3fe1062061 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -63,6 +63,12 @@ static int blk_mq_poll_stats_bkt(const struct request *rq)
 	return bucket;
 }
 
+static inline struct blk_mq_hw_ctx *blk_qc_to_hctx(struct request_queue *q,
+		blk_qc_t qc)
+{
+	return q->queue_hw_ctx[(qc & ~BLK_QC_T_INTERNAL) >> BLK_QC_T_SHIFT];
+}
+
 /*
  * Check if any of the ctx, dispatch list or elevator
  * have pending work in this hardware queue.
@@ -3970,7 +3976,7 @@ int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin)
 	if (current->plug)
 		blk_flush_plug_list(current->plug, false);
 
-	hctx = q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)];
+	hctx = blk_qc_to_hctx(q, cookie);
 
 	/*
 	 * If we sleep, have the caller restart the poll loop to reset
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index fd3860d18d7e..e2c4a7d5de05 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -511,11 +511,6 @@ static inline bool blk_qc_t_valid(blk_qc_t cookie)
 	return cookie != BLK_QC_T_NONE;
 }
 
-static inline unsigned int blk_qc_t_to_queue_num(blk_qc_t cookie)
-{
-	return (cookie & ~BLK_QC_T_INTERNAL) >> BLK_QC_T_SHIFT;
-}
-
 static inline unsigned int blk_qc_t_to_tag(blk_qc_t cookie)
 {
 	return cookie & ((1u << BLK_QC_T_SHIFT) - 1);
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:21 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:21 +0200
Subject: [PATCH 03/16] iomap: don't try to poll multi-bio I/Os in
 __iomap_dio_rw
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-4-hch@lst.de>

If an iocb is split into multiple bios we can't poll for both.  So don't
bother to even try to poll in that case.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 fs/iomap/direct-io.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 9398b8c31323..d5637f467109 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -282,6 +282,13 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 	if (!iov_iter_count(dio->submit.iter))
 		goto out;
 
+	/*
+	 * We can only poll for single bio I/Os.
+	 */
+	if (need_zeroout ||
+	    ((dio->flags & IOMAP_DIO_WRITE) && pos >= i_size_read(inode)))
+		dio->iocb->ki_flags &= ~IOCB_HIPRI;
+
 	if (need_zeroout) {
 		/* zero out from the start of the block to the write offset */
 		pad = pos & (fs_block_size - 1);
@@ -339,6 +346,11 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
 
 		nr_pages = bio_iov_vecs_to_alloc(dio->submit.iter,
 						 BIO_MAX_VECS);
+		/*
+		 * We can only poll for single bio I/Os.
+		 */
+		if (nr_pages)
+			dio->iocb->ki_flags &= ~IOCB_HIPRI;
 		iomap_dio_submit_bio(dio, iomap, bio, pos);
 		pos += n;
 	} while (nr_pages);
@@ -579,6 +591,11 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 			iov_iter_revert(iter, pos - dio->i_size);
 			break;
 		}
+
+		/*
+		 * We can only poll for single bio I/Os.
+		 */
+		iocb->ki_flags &= ~IOCB_HIPRI;
 	} while ((count = iov_iter_count(iter)) > 0);
 	blk_finish_plug(&plug);
 
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:20 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:20 +0200
Subject: [PATCH 02/16] block: don't try to poll multi-bio I/Os in
 __blkdev_direct_IO
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-3-hch@lst.de>

If an iocb is split into multiple bios we can't poll for both.  So don't
bother to even try to poll in that case.

Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 fs/block_dev.c | 21 +++++++--------------
 1 file changed, 7 insertions(+), 14 deletions(-)

diff --git a/fs/block_dev.c b/fs/block_dev.c
index ac9b3c158a77..8600c651b0b0 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -375,7 +375,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 	struct blk_plug plug;
 	struct blkdev_dio *dio;
 	struct bio *bio;
-	bool is_poll = (iocb->ki_flags & IOCB_HIPRI) != 0;
+	bool do_poll = (iocb->ki_flags & IOCB_HIPRI);
 	bool is_read = (iov_iter_rw(iter) == READ), is_sync;
 	loff_t pos = iocb->ki_pos;
 	blk_qc_t qc = BLK_QC_T_NONE;
@@ -404,7 +404,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 	 * Don't plug for HIPRI/polled IO, as those should go straight
 	 * to issue
 	 */
-	if (!is_poll)
+	if (!(iocb->ki_flags & IOCB_HIPRI))
 		blk_start_plug(&plug);
 
 	for (;;) {
@@ -438,20 +438,13 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 
 		nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS);
 		if (!nr_pages) {
-			bool polled = false;
-
-			if (iocb->ki_flags & IOCB_HIPRI) {
+			if (do_poll)
 				bio_set_polled(bio, iocb);
-				polled = true;
-			}
-
 			qc = submit_bio(bio);
-
-			if (polled)
+			if (do_poll)
 				WRITE_ONCE(iocb->ki_cookie, qc);
 			break;
 		}
-
 		if (!dio->multi_bio) {
 			/*
 			 * AIO needs an extra reference to ensure the dio
@@ -462,6 +455,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 				bio_get(bio);
 			dio->multi_bio = true;
 			atomic_set(&dio->ref, 2);
+			do_poll = false;
 		} else {
 			atomic_inc(&dio->ref);
 		}
@@ -470,7 +464,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 		bio = bio_alloc(GFP_KERNEL, nr_pages);
 	}
 
-	if (!is_poll)
+	if (!(iocb->ki_flags & IOCB_HIPRI))
 		blk_finish_plug(&plug);
 
 	if (!is_sync)
@@ -481,8 +475,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 		if (!READ_ONCE(dio->waiter))
 			break;
 
-		if (!(iocb->ki_flags & IOCB_HIPRI) ||
-		    !blk_poll(bdev_get_queue(bdev), qc, true))
+		if (!do_poll || !blk_poll(bdev_get_queue(bdev), qc, true))
 			blk_io_schedule();
 	}
 	__set_current_state(TASK_RUNNING);
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:19 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:19 +0200
Subject: [PATCH 01/16] direct-io: remove blk_poll support
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615131034.752623-2-hch@lst.de>

The polling support in the legacy direct-io support is a little crufty.
It already doesn't support the asynchronous polling needed for io_uring
polling, and is hard to adopt to upcoming changes in the polling
interfaces.  Given that all the major file systems already use the iomap
direct I/O code, just drop the polling support.

Signed-off-by: Christoph Hellwig <hch at lst.de>
Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>
---
 fs/direct-io.c | 14 ++++----------
 1 file changed, 4 insertions(+), 10 deletions(-)

diff --git a/fs/direct-io.c b/fs/direct-io.c
index b2e86e739d7a..453dcff0e7f5 100644
--- a/fs/direct-io.c
+++ b/fs/direct-io.c
@@ -119,7 +119,6 @@ struct dio {
 	int flags;			/* doesn't change */
 	int op;
 	int op_flags;
-	blk_qc_t bio_cookie;
 	struct gendisk *bio_disk;
 	struct inode *inode;
 	loff_t i_size;			/* i_size when submitted */
@@ -438,11 +437,10 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio)
 
 	dio->bio_disk = bio->bi_bdev->bd_disk;
 
-	if (sdio->submit_io) {
+	if (sdio->submit_io)
 		sdio->submit_io(bio, dio->inode, sdio->logical_offset_in_bio);
-		dio->bio_cookie = BLK_QC_T_NONE;
-	} else
-		dio->bio_cookie = submit_bio(bio);
+	else
+		submit_bio(bio);
 
 	sdio->bio = NULL;
 	sdio->boundary = 0;
@@ -481,9 +479,7 @@ static struct bio *dio_await_one(struct dio *dio)
 		__set_current_state(TASK_UNINTERRUPTIBLE);
 		dio->waiter = current;
 		spin_unlock_irqrestore(&dio->bio_lock, flags);
-		if (!(dio->iocb->ki_flags & IOCB_HIPRI) ||
-		    !blk_poll(dio->bio_disk->queue, dio->bio_cookie, true))
-			blk_io_schedule();
+		blk_io_schedule();
 		/* wake up sets us TASK_RUNNING */
 		spin_lock_irqsave(&dio->bio_lock, flags);
 		dio->waiter = NULL;
@@ -1214,8 +1210,6 @@ do_blockdev_direct_IO(struct kiocb *iocb, struct inode *inode,
 	} else {
 		dio->op = REQ_OP_READ;
 	}
-	if (iocb->ki_flags & IOCB_HIPRI)
-		dio->op_flags |= REQ_HIPRI;
 
 	/*
 	 * For AIO O_(D)SYNC writes we need to defer completions to a workqueue
-- 
2.30.2



From hch at lst.de  Tue Jun 15 06:10:18 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 15:10:18 +0200
Subject: switch block layer polling to a bio based model v4
Message-ID: <20210615131034.752623-1-hch@lst.de>

Hi all,

This series clean up the block polling code a bit and changes the interface
to poll for a specific bio instead of a request_queue and cookie pair.

Polling for the bio itself leads to a few advantages:

  - the cookie construction can made entirely private in blk-mq.c
  - the caller does not need to remember the request_queue and cookie
    separately and thus sidesteps their lifetime issues
  - keeping the device and the cookie inside the bio allows to trivially
    support polling BIOs remapping by stacking drivers
  - a lot of code to propagate the cookie back up the submission path can
    removed entirely

The one major caveat is that this requires RCU freeing polled BIOs to make
sure the bio that contains the polling information is still alive when
io_uring tries to poll it through the iocb. For synchronous polling all the
callers have a bio reference anyway, so this is not an issue.

Git tree:

   git://git.infradead.org/users/hch/block.git bio-poll

Gitweb:

    http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/bio-poll

Chances since v3:
 - rebased to the latests for-5.14/block tree
 - fix the refcount logic in __blkdev_direct_IO
 - split up a patch to make it easier to review
 - grab a queue reference in bio_poll
 - better document the RCU assumptions in bio_poll

Chances since v2:
 - remove support for writing to the poll attribute
 - better document the block_device life time assumptions in bio_poll
 - only set QUEUE_FLAG_POLL on nvme-multipath queues where it makes sense

Chances since v1:
 - use SLAB_TYPESAFE_BY_RCU to only free the pages backing the bio
   slabs bio RCU
 - split the spin argument to bio_poll to avoid sleeping under RCU from
   io_uring
 - add support for polling nvme multipath devices

Diffstat:
 arch/m68k/emu/nfblock.c             |    3 
 arch/xtensa/platforms/iss/simdisk.c |    3 
 block/bio.c                         |    4 
 block/blk-core.c                    |  129 ++++++++++++++++++++------
 block/blk-merge.c                   |    2 
 block/blk-mq-debugfs.c              |    2 
 block/blk-mq.c                      |  173 ++++++++++++++----------------------
 block/blk-mq.h                      |    6 -
 block/blk-sysfs.c                   |   23 ----
 drivers/block/brd.c                 |   12 +-
 drivers/block/drbd/drbd_int.h       |    2 
 drivers/block/drbd/drbd_req.c       |    3 
 drivers/block/n64cart.c             |   12 +-
 drivers/block/null_blk/main.c       |    3 
 drivers/block/pktcdvd.c             |    7 -
 drivers/block/ps3vram.c             |    6 -
 drivers/block/rsxx/dev.c            |    7 -
 drivers/block/zram/zram_drv.c       |   10 --
 drivers/lightnvm/pblk-init.c        |    6 -
 drivers/md/bcache/request.c         |   13 +-
 drivers/md/bcache/request.h         |    4 
 drivers/md/dm.c                     |   28 ++---
 drivers/md/md.c                     |   10 --
 drivers/nvdimm/blk.c                |    5 -
 drivers/nvdimm/btt.c                |    5 -
 drivers/nvdimm/pmem.c               |    3 
 drivers/nvme/host/core.c            |    4 
 drivers/nvme/host/multipath.c       |   17 ++-
 drivers/s390/block/dcssblk.c        |    7 -
 drivers/s390/block/xpram.c          |    5 -
 drivers/scsi/scsi_debug.c           |   10 +-
 fs/block_dev.c                      |   44 ++-------
 fs/btrfs/inode.c                    |    8 -
 fs/direct-io.c                      |   14 --
 fs/ext4/file.c                      |    2 
 fs/gfs2/file.c                      |    4 
 fs/io_uring.c                       |   14 +-
 fs/iomap/direct-io.c                |   56 ++++++-----
 fs/xfs/xfs_file.c                   |    2 
 fs/zonefs/super.c                   |    2 
 include/linux/bio.h                 |    4 
 include/linux/blk-mq.h              |   15 ---
 include/linux/blk_types.h           |   34 +------
 include/linux/blkdev.h              |   12 +-
 include/linux/bvec.h                |    2 
 include/linux/fs.h                  |    8 -
 include/linux/iomap.h               |    3 
 mm/page_io.c                        |   10 --
 48 files changed, 350 insertions(+), 408 deletions(-)


From cachen at purestorage.com  Mon Jun 14 23:17:10 2021
From: cachen at purestorage.com (Casey Chen)
Date: Mon, 14 Jun 2021 23:17:10 -0700
Subject: nvme-pci: Fix multiple races in nvme_setup_io_queues()
Message-ID: <CALCePG0up2GK_ihFnLxHuUS4Zn0kfSAC2x1U4Vm_6BQw3Deo5w@mail.gmail.com>

Hi,

We found multiple races in nvme_setup_io_queues() by power-cycling a
PCIe NVMe drive for hours. nvme_setup_io_queues() could overlap with
nvme_dev_disable() if we power off a drive quickly after powering it
on. Multiple races exist because of shutdown_lock missing and improper
use of NVMEQ_ENABLED bit. All problematic places and their fix are
explained in the commit message. This is also reproducible on Intel
Optane drive. Fix has been tested for a week.

All tests and fix are based on tag nvme-5.14-2021-06-08 of repo
http://git.infradead.org/nvme.git

1. Testing Method
while :; power off drive X; sleep $((RANDOM%3)).$((RANDOM%10)); power
on drive X; sleep $((RANDOM%3)).$((RANDOM%10)); done

2. Crash Call Trace
[11668.533392] pcieport 0000:87:08.0: pciehp: pending interrupts
0x0008 from Slot Status
[11668.533431] pcieport 0000:87:08.0: pciehp: Slot(402): Card present
...
[11668.681298] nvme nvme12: pci function 0000:8c:00.0
[11668.681354] nvme 0000:8c:00.0: enabling device (0100 -> 0102)
[11669.046119] pcieport 0000:87:08.0: pciehp: pending interrupts
0x0108 from Slot Status
[11669.046142] pcieport 0000:87:08.0: pciehp: Slot(402): Link Down
[11669.046146] pcieport 0000:87:08.0: pciehp: Slot(402): Card not present
[11669.046149] pcieport 0000:87:08.0: pciehp:
pciehp_unconfigure_device: domain:bus:dev = 0000:8c:00
[11669.077428] ------------[ cut here ]------------
[11669.077431] kernel BUG at drivers/pci/msi.c:348!
[11669.077555] invalid opcode: 0000 [#1] SMP KASAN
[11669.077658] CPU: 31 PID: 716 Comm: irq/127-pciehp Not tainted 5.13.0-rc3+
[11669.077869] Hardware name: <MASKED OFF>
[11669.078022] RIP: 0010:free_msi_irqs+0x28a/0x2d0
[11669.078129] Code: f7 83 c5 01 e8 87 44 c7 ff 39 6b 14 0f 86 d6 fd
ff ff 4c 89 ef e8 76 44 c7 ff 8b 43 10 8d 7c 05 00 e8 8a 0d a7 ff 84
c0 74 d4 <0f> 0b 48 8d 7b 60 e8 5b 45 c7 ff 48 8b 7b 60 e8 52 46 9b ff
48 89
[11669.078453] RSP: 0018:ffff88813917fa38 EFLAGS: 00010202
[11669.078564] RAX: 0000000000000001 RBX: ffff88e042875800 RCX: ffffffff81160812
[11669.078703] RDX: dffffc0000000000 RSI: ffff88e54fc86740 RDI: ffff892fe1627ca0
[11669.078843] RBP: 0000000000000000 R08: ffffed278641a4fd R09: ffffed278641a4fd
[11669.078982] R10: ffff893c320d27e5 R11: ffffed278641a4fc R12: ffff893c320d22e8
[11669.079121] R13: ffff88e042875810 R14: ffff88e042875814 R15: ffff893c320d2000
[11669.079260] FS:  0000000000000000(0000) GS:ffff893ebe640000(0000)
knlGS:0000000000000000
[11669.079415] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11669.079531] CR2: 00007f1fc13fa000 CR3: 00000063a2de2002 CR4: 00000000005706e0
[11669.079670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[11669.085209] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[11669.091054] PKRU: 55555554
[11669.093982] Call Trace:
[11669.096850]  pci_free_irq_vectors+0xe/0x20
[11669.099695]  nvme_dev_disable+0x140/0x760 [nvme]
[11669.102503]  ? _raw_spin_lock_irqsave+0x9c/0x100
[11669.105271]  ? trace_hardirqs_on+0x2c/0xe0
[11669.107994]  nvme_remove+0x191/0x1e0 [nvme]
[11669.110689]  pci_device_remove+0x6b/0x110
[11669.113316]  device_release_driver_internal+0x14f/0x280
[11669.115939]  pci_stop_bus_device+0xcb/0x100
[11669.118515]  pci_stop_and_remove_bus_device+0xe/0x20
[11669.121079]  pciehp_unconfigure_device+0xfa/0x200
[11669.123597]  ? pciehp_configure_device+0x1c0/0x1c0
[11669.126049]  ? trace_hardirqs_on+0x2c/0xe0
[11669.128444]  pciehp_disable_slot+0xc4/0x1a0
[11669.130771]  ? pciehp_runtime_suspend+0x40/0x40
[11669.133054]  ? __mutex_lock_slowpath+0x10/0x10
[11669.135289]  ? trace_hardirqs_on+0x2c/0xe0
[11669.137462]  pciehp_handle_presence_or_link_change+0x15c/0x4f0
[11669.139632]  ? down_read+0x11f/0x1a0
[11669.141731]  ? pciehp_handle_disable_request+0x80/0x80
[11669.143817]  ? rwsem_down_read_slowpath+0x600/0x600
[11669.145851]  ? __radix_tree_lookup+0xb2/0x130
[11669.147842]  pciehp_ist+0x19d/0x1a0
[11669.149790]  ? pciehp_set_indicators+0xe0/0xe0
[11669.151704]  ? irq_finalize_oneshot.part.46+0x1d0/0x1d0
[11669.153588]  irq_thread_fn+0x3f/0xa0
[11669.155407]  irq_thread+0x195/0x290
[11669.157147]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
[11669.158883]  ? _raw_read_lock_irq+0x50/0x50
[11669.160611]  ? _raw_read_lock_irq+0x50/0x50
[11669.162320]  ? irq_forced_thread_fn+0xf0/0xf0
[11669.164032]  ? trace_hardirqs_on+0x2c/0xe0
[11669.165731]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
[11669.167461]  kthread+0x1c8/0x1f0
[11669.169173]  ? kthread_parkme+0x40/0x40
[11669.170883]  ret_from_fork+0x22/0x30
[11669.172592] Modules linked in: iptable_nat nf_nat ip6table_raw
xt_CT iptable_raw ip6t_REJECT nf_reject_ipv6 ip6table_mangle
ipt_REJECT nf_reject_ipv4 xt_owner xt_conntrack iptable_mangle xt_ipvs
ip_vs nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype
br_netfilter xt_mark xt_tcpudp ip6table_filter ip6_tables
iptable_filter nvme ip6_gre ip6_tunnel tunnel6 gre macvlan bridge stp
llc nvme_rdma nvme_fabrics nvme_core vfio_pci vfio_virqfd
vfio_iommu_type1 vfio qede qed crc8 scsi_dh_alua intel_rapl_msr
intel_rapl_common skx_edac x86_pkg_temp_thermal coretemp kvm_intel kvm
irqbypass ntb_hw_intel ntb intel_pch_thermal hed sunrpc msr bonding
ip_tables x_tables autofs4 rdma_ucm rdma_cm mlx4_ib mlx4_core iw_cm
ib_umad ib_ipoib ib_cm mlx5_ib ib_uverbs ib_core ghash_clmulni_intel
aesni_intel crypto_simd cryptd mlx5_core mlxfw pci_hyperv_intf mpt3sas
raid_class scsi_transport_sas i2c_i801 i2c_smbus pure_gpio
pinctrl_lewisburg pinctrl_intel hid_generic usbhid hid
[11669.172735]  [last unloaded: qla2xxx]
[11669.201990] Dumping ftrace buffer:
[11669.204376]    (ftrace buffer empty)
[11669.206766] ---[ end trace 07b68fb4b0c1a54c ]---
[11669.652093] RIP: 0010:free_msi_irqs+0x28a/0x2d0
[11669.654461] Code: f7 83 c5 01 e8 87 44 c7 ff 39 6b 14 0f 86 d6 fd
ff ff 4c 89 ef e8 76 44 c7 ff 8b 43 10 8d 7c 05 00 e8 8a 0d a7 ff 84
c0 74 d4 <0f> 0b 48 8d 7b 60 e8 5b 45 c7 ff 48 8b 7b 60 e8 52 46 9b ff
48 89
[11669.661764] RSP: 0018:ffff88813917fa38 EFLAGS: 00010202
[11669.664271] RAX: 0000000000000001 RBX: ffff88e042875800 RCX: ffffffff81160812
[11669.669382] RDX: dffffc0000000000 RSI: ffff88e54fc86740 RDI: ffff892fe1627ca0
[11669.674692] RBP: 0000000000000000 R08: ffffed278641a4fd R09: ffffed278641a4fd
[11669.680179] R10: ffff893c320d27e5 R11: ffffed278641a4fc R12: ffff893c320d22e8
[11669.685880] R13: ffff88e042875810 R14: ffff88e042875814 R15: ffff893c320d2000
[11669.691587] FS:  0000000000000000(0000) GS:ffff893ebe640000(0000)
knlGS:0000000000000000
[11669.697296] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11669.700174] CR2: 00007f1fc13fa000 CR3: 00000063a2de2002 CR4: 00000000005706e0

3. Proposed Fix
>From b609626a69d584ce36bad4787ea6b4c6e6dba453 Mon Sep 17 00:00:00 2001
From: Casey Chen <cachen at purestorage.com>
Date: Mon, 31 Aug 2020 18:14:23 -0600
Subject: [PATCH] nvme-pci: Fix multiple races in nvme_setup_io_queues()

Below two paths could overlap each other if we power off a drive quickly
after power it on. There are multiple races in nvme_setup_io_queues()
because of shutdown_lock missing and improper use of NVMEQ_ENABLED bit.

nvme_reset_work()
      nvme_remove()
  nvme_setup_io_queues()
   nvme_dev_disable()
  ...
                       ...
A1  clear NVMEQ_ENABLED bit for admin queue                  lock
    retry:
                 B1  nvme_suspend_io_queues()
A2    pci_free_irq() admin queue
B2  nvme_suspend_queue() admin queue
A3    pci_free_irq_vectors()
           nvme_pci_disable()
A4    nvme_setup_irqs();
    B3     pci_free_irq_vectors()
      ...
                          unlock
A5    queue_request_irq() for admin queue
      set NVMEQ_ENABLED bit
      ...
      nvme_create_io_queues()
A6      result = queue_request_irq();
        set NVMEQ_ENABLED bit
      ...
      fail to allocate enough IO queues:
A7      nvme_suspend_io_queues()
        goto retry

1). If B3 runs in between A1 and A2, it will crash if irqaction haven't
  been freed by A2. B2 is supposed to free admin queue IRQ but it simply
  can't fulfill the job as A1 has cleared NVMEQ_ENABLED bit.

Fix: combine A1 A2 so IRQ get freed as soon as NVMEQ_ENABLED bit get cleared.

2). After solved #1, A2 could race with B3 if A2 is freeing IRQ while B3
  is checking irqaction. A3 also could race with B2 if B2 is freeing
  IRQ while A3 is checking irqaction.

Fix: A2 and A3 take lock for mutual exclusion.

3) A3 could race with B3 since they could run free_msi_irqs() in parallel.

Fix: A3 takes lock for mutual exclusion.

4). A4 could fail to allocate all needed IRQ vectors if A3 and A4 are
  interrupted by B3.

Fix: A4 takes lock for mutual exclusion.

5). If A5/A6 happened after B2/B1, B3 will crash since irqaction is not NULL.
  They are just allocated by A5/A6.

Fix: Lock queue_request_irq() and setting of NVMEQ_ENABLED bit.

6). A7 could get chance to pci_free_irq() for certain IO queue while B3
  is checking irqaction.

Fix: A7 takes lock.

7). nvme_dev->online_queues need to be protected by shutdown_lock. Since it
  is not atomic, both paths could modify it using its own copy.

Co-developed-by: Yuanyuan Zhong <yzhong at purestorage.com>
Signed-off-by: Casey Chen <cachen at purestorage.com>
---
 drivers/nvme/host/pci.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..81e53aaaa77c 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1590,8 +1590,9 @@ static int nvme_create_queue(struct nvme_queue
*nvmeq, int qid, bool polled)
  goto release_cq;

  nvmeq->cq_vector = vector;
- nvme_init_queue(nvmeq, qid);

+ mutex_lock(&dev->shutdown_lock);
+ nvme_init_queue(nvmeq, qid);
  if (!polled) {
  result = queue_request_irq(nvmeq);
  if (result < 0)
@@ -1599,10 +1600,12 @@ static int nvme_create_queue(struct nvme_queue
*nvmeq, int qid, bool polled)
  }

  set_bit(NVMEQ_ENABLED, &nvmeq->flags);
+ mutex_unlock(&dev->shutdown_lock);
  return result;

 release_sq:
  dev->online_queues--;
+ mutex_unlock(&dev->shutdown_lock);
  adapter_delete_sq(dev, qid);
 release_cq:
  adapter_delete_cq(dev, qid);
@@ -2176,7 +2179,16 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
  if (nr_io_queues == 0)
  return 0;

- clear_bit(NVMEQ_ENABLED, &adminq->flags);
+ /*
+ * Free IRQ resources as soon as NVMEQ_ENABLED bit transitions
+ * from set to unset. If there is a window to it is truely freed,
+ * pci_free_irq_vectors() jumping into this window will crash.
+ * And take lock to avoid racing with pci_free_irq_vectors() in
+ * nvme_dev_disable() path.
+ */
+ mutex_lock(&dev->shutdown_lock);
+ if (test_and_clear_bit(NVMEQ_ENABLED, &adminq->flags))
+ pci_free_irq(pdev, 0, adminq);

  if (dev->cmb_use_sqes) {
  result = nvme_cmb_qdepth(dev, nr_io_queues,
@@ -2192,14 +2204,17 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
  result = nvme_remap_bar(dev, size);
  if (!result)
  break;
- if (!--nr_io_queues)
+ if (!--nr_io_queues) {
+ mutex_unlock(&dev->shutdown_lock);
  return -ENOMEM;
+ }
  } while (1);
  adminq->q_db = dev->dbs;

  retry:
  /* Deregister the admin queue's interrupt */
- pci_free_irq(pdev, 0, adminq);
+ if (test_and_clear_bit(NVMEQ_ENABLED, &adminq->flags))
+ pci_free_irq(pdev, 0, adminq);

  /*
  * If we enable msix early due to not intx, disable it again before
@@ -2208,8 +2223,10 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
  pci_free_irq_vectors(pdev);

  result = nvme_setup_irqs(dev, nr_io_queues);
- if (result <= 0)
+ if (result <= 0) {
+ mutex_unlock(&dev->shutdown_lock);
  return -EIO;
+ }

  dev->num_vecs = result;
  result = max(result - 1, 1);
@@ -2222,9 +2239,12 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
  * number of interrupts.
  */
  result = queue_request_irq(adminq);
- if (result)
+ if (result) {
+ mutex_unlock(&dev->shutdown_lock);
  return result;
+ }
  set_bit(NVMEQ_ENABLED, &adminq->flags);
+ mutex_unlock(&dev->shutdown_lock);

  result = nvme_create_io_queues(dev);
  if (result || dev->online_queues < 2)
@@ -2233,6 +2253,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
  if (dev->online_queues - 1 < dev->max_qid) {
  nr_io_queues = dev->online_queues - 1;
  nvme_disable_io_queues(dev);
+ mutex_lock(&dev->shutdown_lock);
  nvme_suspend_io_queues(dev);
  goto retry;
  }
--
2.17.1


From hch at lst.de  Mon Jun 14 22:55:24 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 07:55:24 +0200
Subject: [PATCH v8 2/2] ACPI: Add quirks for AMD Renoir/Lucienne CPUs
 to force the D3 hint
In-Reply-To: <831df669-b38e-9f4b-dd67-b613aaab6c28@amd.com>
References: <20210609184018.26017-1-mario.limonciello@amd.com>
 <20210609184018.26017-2-mario.limonciello@amd.com>
 <831df669-b38e-9f4b-dd67-b613aaab6c28@amd.com>
Message-ID: <20210615055524.GA21218@lst.de>

On Mon, Jun 14, 2021 at 09:02:36PM -0500, Limonciello, Mario wrote:
> Christoph, Keith ping on this series.  Any other thoughts?
> If it's good can you take this in for nvme-5.14 since Rafael mentioned
> the conflict from the other StorageD3Enable commit in nvme-5.14?

I'll queue it up for nvme-5.14.  I was planning to do a run today.


From chaitanya.kulkarni at wdc.com  Mon Jun 14 19:45:52 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 14 Jun 2021 19:45:52 -0700
Subject: [PATCH 2/4] nvme-fabrics: remove memset in nvmf_reg_write32()
In-Reply-To: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
References: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210615024554.42585-3-chaitanya.kulkarni@wdc.com>

Declare and initialize structure variable to the zero values so that we
can get rid of the zeroout memset call.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/fabrics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 506bb976251b..ccec9acedef5 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -231,10 +231,9 @@ EXPORT_SYMBOL_GPL(nvmf_reg_read64);
  */
 int nvmf_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val)
 {
-	struct nvme_command cmd;
+	struct nvme_command cmd = { };
 	int ret;
 
-	memset(&cmd, 0, sizeof(cmd));
 	cmd.prop_set.opcode = nvme_fabrics_command;
 	cmd.prop_set.fctype = nvme_fabrics_type_property_set;
 	cmd.prop_set.attrib = 0;
-- 
2.22.1



From cachen at purestorage.com  Mon Jun 14 23:26:55 2021
From: cachen at purestorage.com (Casey Chen)
Date: Mon, 14 Jun 2021 23:26:55 -0700
Subject: nvme-pci: Fix multiple races in nvme_setup_io_queues()
Message-ID: <CALCePG2ukbm=mjZiKij5_3T4sTc4H5wRuityrr2Ljv0HvfJqtQ@mail.gmail.com>

(Please ignore the previous email, the call paths comparison shown in
commit message should look better if you copy then paste in a code
editor)

Hi,

We found multiple races in nvme_setup_io_queues() by power-cycling a
PCIe NVMe drive for hours. nvme_setup_io_queues() could overlap with
nvme_dev_disable() if we power off a drive quickly after powering it
on. Multiple races exist because of shutdown_lock missing and improper
use of NVMEQ_ENABLED bit. All problematic places and their fix are
explained in the commit message. This is also reproducible on Intel
Optane drive. Fix has been tested for a week.

All tests and fix are based on tag nvme-5.14-2021-06-08 of repo
http://git.infradead.org/nvme.git

1. Testing Method
while :; power off drive X; sleep $((RANDOM%3)).$((RANDOM%10)); power
on drive X; sleep $((RANDOM%3)).$((RANDOM%10)); done

2. Crash Call Trace
[11668.533392] pcieport 0000:87:08.0: pciehp: pending interrupts
0x0008 from Slot Status
[11668.533431] pcieport 0000:87:08.0: pciehp: Slot(402): Card present
...
[11668.681298] nvme nvme12: pci function 0000:8c:00.0
[11668.681354] nvme 0000:8c:00.0: enabling device (0100 -> 0102)
[11669.046119] pcieport 0000:87:08.0: pciehp: pending interrupts
0x0108 from Slot Status
[11669.046142] pcieport 0000:87:08.0: pciehp: Slot(402): Link Down
[11669.046146] pcieport 0000:87:08.0: pciehp: Slot(402): Card not present
[11669.046149] pcieport 0000:87:08.0: pciehp:
pciehp_unconfigure_device: domain:bus:dev = 0000:8c:00
[11669.077428] ------------[ cut here ]------------
[11669.077431] kernel BUG at drivers/pci/msi.c:348!
[11669.077555] invalid opcode: 0000 [#1] SMP KASAN
[11669.077658] CPU: 31 PID: 716 Comm: irq/127-pciehp Not tainted 5.13.0-rc3+
[11669.077869] Hardware name: <MASKED OFF>
[11669.078022] RIP: 0010:free_msi_irqs+0x28a/0x2d0
[11669.078129] Code: f7 83 c5 01 e8 87 44 c7 ff 39 6b 14 0f 86 d6 fd
ff ff 4c 89 ef e8 76 44 c7 ff 8b 43 10 8d 7c 05 00 e8 8a 0d a7 ff 84
c0 74 d4 <0f> 0b 48 8d 7b 60 e8 5b 45 c7 ff 48 8b 7b 60 e8 52 46 9b ff
48 89
[11669.078453] RSP: 0018:ffff88813917fa38 EFLAGS: 00010202
[11669.078564] RAX: 0000000000000001 RBX: ffff88e042875800 RCX: ffffffff81160812
[11669.078703] RDX: dffffc0000000000 RSI: ffff88e54fc86740 RDI: ffff892fe1627ca0
[11669.078843] RBP: 0000000000000000 R08: ffffed278641a4fd R09: ffffed278641a4fd
[11669.078982] R10: ffff893c320d27e5 R11: ffffed278641a4fc R12: ffff893c320d22e8
[11669.079121] R13: ffff88e042875810 R14: ffff88e042875814 R15: ffff893c320d2000
[11669.079260] FS:  0000000000000000(0000) GS:ffff893ebe640000(0000)
knlGS:0000000000000000
[11669.079415] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11669.079531] CR2: 00007f1fc13fa000 CR3: 00000063a2de2002 CR4: 00000000005706e0
[11669.079670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[11669.085209] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[11669.091054] PKRU: 55555554
[11669.093982] Call Trace:
[11669.096850]  pci_free_irq_vectors+0xe/0x20
[11669.099695]  nvme_dev_disable+0x140/0x760 [nvme]
[11669.102503]  ? _raw_spin_lock_irqsave+0x9c/0x100
[11669.105271]  ? trace_hardirqs_on+0x2c/0xe0
[11669.107994]  nvme_remove+0x191/0x1e0 [nvme]
[11669.110689]  pci_device_remove+0x6b/0x110
[11669.113316]  device_release_driver_internal+0x14f/0x280
[11669.115939]  pci_stop_bus_device+0xcb/0x100
[11669.118515]  pci_stop_and_remove_bus_device+0xe/0x20
[11669.121079]  pciehp_unconfigure_device+0xfa/0x200
[11669.123597]  ? pciehp_configure_device+0x1c0/0x1c0
[11669.126049]  ? trace_hardirqs_on+0x2c/0xe0
[11669.128444]  pciehp_disable_slot+0xc4/0x1a0
[11669.130771]  ? pciehp_runtime_suspend+0x40/0x40
[11669.133054]  ? __mutex_lock_slowpath+0x10/0x10
[11669.135289]  ? trace_hardirqs_on+0x2c/0xe0
[11669.137462]  pciehp_handle_presence_or_link_change+0x15c/0x4f0
[11669.139632]  ? down_read+0x11f/0x1a0
[11669.141731]  ? pciehp_handle_disable_request+0x80/0x80
[11669.143817]  ? rwsem_down_read_slowpath+0x600/0x600
[11669.145851]  ? __radix_tree_lookup+0xb2/0x130
[11669.147842]  pciehp_ist+0x19d/0x1a0
[11669.149790]  ? pciehp_set_indicators+0xe0/0xe0
[11669.151704]  ? irq_finalize_oneshot.part.46+0x1d0/0x1d0
[11669.153588]  irq_thread_fn+0x3f/0xa0
[11669.155407]  irq_thread+0x195/0x290
[11669.157147]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
[11669.158883]  ? _raw_read_lock_irq+0x50/0x50
[11669.160611]  ? _raw_read_lock_irq+0x50/0x50
[11669.162320]  ? irq_forced_thread_fn+0xf0/0xf0
[11669.164032]  ? trace_hardirqs_on+0x2c/0xe0
[11669.165731]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
[11669.167461]  kthread+0x1c8/0x1f0
[11669.169173]  ? kthread_parkme+0x40/0x40
[11669.170883]  ret_from_fork+0x22/0x30
[11669.172592] Modules linked in: iptable_nat nf_nat ip6table_raw
xt_CT iptable_raw ip6t_REJECT nf_reject_ipv6 ip6table_mangle
ipt_REJECT nf_reject_ipv4 xt_owner xt_conntrack iptable_mangle xt_ipvs
ip_vs nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype
br_netfilter xt_mark xt_tcpudp ip6table_filter ip6_tables
iptable_filter nvme ip6_gre ip6_tunnel tunnel6 gre macvlan bridge stp
llc nvme_rdma nvme_fabrics nvme_core vfio_pci vfio_virqfd
vfio_iommu_type1 vfio qede qed crc8 scsi_dh_alua intel_rapl_msr
intel_rapl_common skx_edac x86_pkg_temp_thermal coretemp kvm_intel kvm
irqbypass ntb_hw_intel ntb intel_pch_thermal hed sunrpc msr bonding
ip_tables x_tables autofs4 rdma_ucm rdma_cm mlx4_ib mlx4_core iw_cm
ib_umad ib_ipoib ib_cm mlx5_ib ib_uverbs ib_core ghash_clmulni_intel
aesni_intel crypto_simd cryptd mlx5_core mlxfw pci_hyperv_intf mpt3sas
raid_class scsi_transport_sas i2c_i801 i2c_smbus pure_gpio
pinctrl_lewisburg pinctrl_intel hid_generic usbhid hid
[11669.172735]  [last unloaded: qla2xxx]
[11669.201990] Dumping ftrace buffer:
[11669.204376]    (ftrace buffer empty)
[11669.206766] ---[ end trace 07b68fb4b0c1a54c ]---
[11669.652093] RIP: 0010:free_msi_irqs+0x28a/0x2d0
[11669.654461] Code: f7 83 c5 01 e8 87 44 c7 ff 39 6b 14 0f 86 d6 fd
ff ff 4c 89 ef e8 76 44 c7 ff 8b 43 10 8d 7c 05 00 e8 8a 0d a7 ff 84
c0 74 d4 <0f> 0b 48 8d 7b 60 e8 5b 45 c7 ff 48 8b 7b 60 e8 52 46 9b ff
48 89
[11669.661764] RSP: 0018:ffff88813917fa38 EFLAGS: 00010202
[11669.664271] RAX: 0000000000000001 RBX: ffff88e042875800 RCX: ffffffff81160812
[11669.669382] RDX: dffffc0000000000 RSI: ffff88e54fc86740 RDI: ffff892fe1627ca0
[11669.674692] RBP: 0000000000000000 R08: ffffed278641a4fd R09: ffffed278641a4fd
[11669.680179] R10: ffff893c320d27e5 R11: ffffed278641a4fc R12: ffff893c320d22e8
[11669.685880] R13: ffff88e042875810 R14: ffff88e042875814 R15: ffff893c320d2000
[11669.691587] FS:  0000000000000000(0000) GS:ffff893ebe640000(0000)
knlGS:0000000000000000
[11669.697296] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11669.700174] CR2: 00007f1fc13fa000 CR3: 00000063a2de2002 CR4: 00000000005706e0

3. Proposed Fix
>From b609626a69d584ce36bad4787ea6b4c6e6dba453 Mon Sep 17 00:00:00 2001
From: Casey Chen <cachen at purestorage.com>
Date: Mon, 31 Aug 2020 18:14:23 -0600
Subject: [PATCH] nvme-pci: Fix multiple races in nvme_setup_io_queues()

Below two paths could overlap each other if we power off a drive quickly
after power it on. There are multiple races in nvme_setup_io_queues()
because of shutdown_lock missing and improper use of NVMEQ_ENABLED bit.

nvme_reset_work()                                nvme_remove()
  nvme_setup_io_queues()                           nvme_dev_disable()
  ...                                              ...
A1  clear NVMEQ_ENABLED bit for admin queue          lock
    retry:                                       B1  nvme_suspend_io_queues()
A2    pci_free_irq() admin queue                 B2
nvme_suspend_queue() admin queue
A3    pci_free_irq_vectors()                         nvme_pci_disable()
A4    nvme_setup_irqs();                         B3    pci_free_irq_vectors()
      ...                                            unlock
A5    queue_request_irq() for admin queue
      set NVMEQ_ENABLED bit
      ...
      nvme_create_io_queues()
A6      result = queue_request_irq();
        set NVMEQ_ENABLED bit
      ...
      fail to allocate enough IO queues:
A7      nvme_suspend_io_queues()
        goto retry

1). If B3 runs in between A1 and A2, it will crash if irqaction haven't
  been freed by A2. B2 is supposed to free admin queue IRQ but it simply
  can't fulfill the job as A1 has cleared NVMEQ_ENABLED bit.

Fix: combine A1 A2 so IRQ get freed as soon as NVMEQ_ENABLED bit get cleared.

2). After solved #1, A2 could race with B3 if A2 is freeing IRQ while B3
  is checking irqaction. A3 also could race with B2 if B2 is freeing
  IRQ while A3 is checking irqaction.

Fix: A2 and A3 take lock for mutual exclusion.

3) A3 could race with B3 since they could run free_msi_irqs() in parallel.

Fix: A3 takes lock for mutual exclusion.

4). A4 could fail to allocate all needed IRQ vectors if A3 and A4 are
  interrupted by B3.

Fix: A4 takes lock for mutual exclusion.

5). If A5/A6 happened after B2/B1, B3 will crash since irqaction is not NULL.
  They are just allocated by A5/A6.

Fix: Lock queue_request_irq() and setting of NVMEQ_ENABLED bit.

6). A7 could get chance to pci_free_irq() for certain IO queue while B3
  is checking irqaction.

Fix: A7 takes lock.

7). nvme_dev->online_queues need to be protected by shutdown_lock. Since it
  is not atomic, both paths could modify it using its own copy.

Co-developed-by: Yuanyuan Zhong <yzhong at purestorage.com>
Signed-off-by: Casey Chen <cachen at purestorage.com>
---
 drivers/nvme/host/pci.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..81e53aaaa77c 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1590,8 +1590,9 @@ static int nvme_create_queue(struct nvme_queue
*nvmeq, int qid, bool polled)
  goto release_cq;

  nvmeq->cq_vector = vector;
- nvme_init_queue(nvmeq, qid);

+ mutex_lock(&dev->shutdown_lock);
+ nvme_init_queue(nvmeq, qid);
  if (!polled) {
  result = queue_request_irq(nvmeq);
  if (result < 0)
@@ -1599,10 +1600,12 @@ static int nvme_create_queue(struct nvme_queue
*nvmeq, int qid, bool polled)
  }

  set_bit(NVMEQ_ENABLED, &nvmeq->flags);
+ mutex_unlock(&dev->shutdown_lock);
  return result;

 release_sq:
  dev->online_queues--;
+ mutex_unlock(&dev->shutdown_lock);
  adapter_delete_sq(dev, qid);
 release_cq:
  adapter_delete_cq(dev, qid);
@@ -2176,7 +2179,16 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
  if (nr_io_queues == 0)
  return 0;

- clear_bit(NVMEQ_ENABLED, &adminq->flags);
+ /*
+ * Free IRQ resources as soon as NVMEQ_ENABLED bit transitions
+ * from set to unset. If there is a window to it is truely freed,
+ * pci_free_irq_vectors() jumping into this window will crash.
+ * And take lock to avoid racing with pci_free_irq_vectors() in
+ * nvme_dev_disable() path.
+ */
+ mutex_lock(&dev->shutdown_lock);
+ if (test_and_clear_bit(NVMEQ_ENABLED, &adminq->flags))
+ pci_free_irq(pdev, 0, adminq);

  if (dev->cmb_use_sqes) {
  result = nvme_cmb_qdepth(dev, nr_io_queues,
@@ -2192,14 +2204,17 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
  result = nvme_remap_bar(dev, size);
  if (!result)
  break;
- if (!--nr_io_queues)
+ if (!--nr_io_queues) {
+ mutex_unlock(&dev->shutdown_lock);
  return -ENOMEM;
+ }
  } while (1);
  adminq->q_db = dev->dbs;

  retry:
  /* Deregister the admin queue's interrupt */
- pci_free_irq(pdev, 0, adminq);
+ if (test_and_clear_bit(NVMEQ_ENABLED, &adminq->flags))
+ pci_free_irq(pdev, 0, adminq);

  /*
  * If we enable msix early due to not intx, disable it again before
@@ -2208,8 +2223,10 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
  pci_free_irq_vectors(pdev);

  result = nvme_setup_irqs(dev, nr_io_queues);
- if (result <= 0)
+ if (result <= 0) {
+ mutex_unlock(&dev->shutdown_lock);
  return -EIO;
+ }

  dev->num_vecs = result;
  result = max(result - 1, 1);
@@ -2222,9 +2239,12 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
  * number of interrupts.
  */
  result = queue_request_irq(adminq);
- if (result)
+ if (result) {
+ mutex_unlock(&dev->shutdown_lock);
  return result;
+ }
  set_bit(NVMEQ_ENABLED, &adminq->flags);
+ mutex_unlock(&dev->shutdown_lock);

  result = nvme_create_io_queues(dev);
  if (result || dev->online_queues < 2)
@@ -2233,6 +2253,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
  if (dev->online_queues - 1 < dev->max_qid) {
  nr_io_queues = dev->online_queues - 1;
  nvme_disable_io_queues(dev);
+ mutex_lock(&dev->shutdown_lock);
  nvme_suspend_io_queues(dev);
  goto retry;
  }
--
2.17.1


From chaitanya.kulkarni at wdc.com  Mon Jun 14 19:45:50 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 14 Jun 2021 19:45:50 -0700
Subject: [PATCH 0/4] nvme-fabrics: remove struct zeroout memset calls
Message-ID: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>

Hi,

While reviewing the code I did removed the calls to zero out structures
using memset() for the lio target, which are reviewed and accepted in
the linux kernel tree. (see [1])

This patch series is based on the same cleanup pattern for NVMe host.
This will allow us to have uniform pattern in the future code review to
avoid similar calls which seems unnecessary.

If this is accepted I'll gradually convert the rest of the host code
mainly core and pci.

-ck

Chaitanya Kulkarni (4):
  nvme-fabrics: remove memset in nvmf_reg_read64()
  nvme-fabrics: remove memset in nvmf_reg_write32()
  nvme-fabrics: remove memset in connect admin q
  nvme-fabrics: remove memset in connect io q

 drivers/nvme/host/fabrics.c | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

[1] https://lore.kernel.org/linux-scsi/20210228055645.22253-1-chaitanya.kulkarni at wdc.com/
https://lore.kernel.org/linux-scsi/20210228055645.22253-13-chaitanya.kulkarni at wdc.com/
https://lore.kernel.org/linux-scsi/20210228055645.22253-14-chaitanya.kulkarni at wdc.com/
https://lore.kernel.org/linux-scsi/20210228055645.22253-15-chaitanya.kulkarni at wdc.com/
https://lore.kernel.org/linux-scsi/20210228055645.22253-16-chaitanya.kulkarni at wdc.com/
https://lore.kernel.org/linux-scsi/20210228055645.22253-17-chaitanya.kulkarni at wdc.com/
https://lore.kernel.org/linux-scsi/20210228055645.22253-18-chaitanya.kulkarni at wdc.com/
https://lore.kernel.org/linux-scsi/20210228055645.22253-19-chaitanya.kulkarni at wdc.com/

-- 
2.22.1



From kbusch at kernel.org  Tue Jun 15 07:37:06 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 15 Jun 2021 07:37:06 -0700
Subject: switch block layer polling to a bio based model v4
In-Reply-To: <20210615131034.752623-1-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
Message-ID: <20210615143706.GB646237@dhcp-10-100-145-180.wdc.com>

On Tue, Jun 15, 2021 at 03:10:18PM +0200, Christoph Hellwig wrote:
> Chances since v3:
>  - rebased to the latests for-5.14/block tree
>  - fix the refcount logic in __blkdev_direct_IO
>  - split up a patch to make it easier to review
>  - grab a queue reference in bio_poll
>  - better document the RCU assumptions in bio_poll

It still doesn't look like a failover will work when polling through
pvsync2. We previously discussed that here:

  https://marc.info/?l=linux-block&m=162100971816071&w=2


From chaitanya.kulkarni at wdc.com  Mon Jun 14 19:45:53 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 14 Jun 2021 19:45:53 -0700
Subject: [PATCH 3/4] nvme-fabrics: remove memset in connect admin q
In-Reply-To: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
References: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210615024554.42585-4-chaitanya.kulkarni@wdc.com>

Declare and initialize structure variable to the zero values so that we
can get rid of the zeroout memset call.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/fabrics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index ccec9acedef5..86a1a05de71b 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -354,12 +354,11 @@ static void nvmf_log_connect_error(struct nvme_ctrl *ctrl,
  */
 int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 {
-	struct nvme_command cmd;
+	struct nvme_command cmd = { };
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
 
-	memset(&cmd, 0, sizeof(cmd));
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
 	cmd.connect.qid = 0;
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun 14 19:45:51 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 14 Jun 2021 19:45:51 -0700
Subject: [PATCH 1/4] nvme-fabrics: remove memset in nvmf_reg_read64()
In-Reply-To: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
References: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210615024554.42585-2-chaitanya.kulkarni@wdc.com>

Declare and initialize structure variable to the zero values so that we
can get rid of the zeroout memset call.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/fabrics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 690e749eee83..506bb976251b 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -186,11 +186,10 @@ EXPORT_SYMBOL_GPL(nvmf_reg_read32);
  */
 int nvmf_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val)
 {
-	struct nvme_command cmd;
+	struct nvme_command cmd = { };
 	union nvme_result res;
 	int ret;
 
-	memset(&cmd, 0, sizeof(cmd));
 	cmd.prop_get.opcode = nvme_fabrics_command;
 	cmd.prop_get.fctype = nvme_fabrics_type_property_get;
 	cmd.prop_get.attrib = 1;
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Mon Jun 14 19:45:54 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 14 Jun 2021 19:45:54 -0700
Subject: [PATCH 4/4] nvme-fabrics: remove memset in connect io q
In-Reply-To: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
References: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210615024554.42585-5-chaitanya.kulkarni@wdc.com>

Declare and initialize structure variable to the zero values so that we
can get rid of the zeroout memset call.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/fabrics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 86a1a05de71b..b13b42bf2074 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -421,12 +421,11 @@ EXPORT_SYMBOL_GPL(nvmf_connect_admin_queue);
  */
 int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid, bool poll)
 {
-	struct nvme_command cmd;
+	struct nvme_command cmd = { };
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
 
-	memset(&cmd, 0, sizeof(cmd));
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
 	cmd.connect.qid = cpu_to_le16(qid);
-- 
2.22.1



From mario.limonciello at amd.com  Mon Jun 14 19:02:36 2021
From: mario.limonciello at amd.com (Limonciello, Mario)
Date: Mon, 14 Jun 2021 21:02:36 -0500
Subject: [PATCH v8 2/2] ACPI: Add quirks for AMD Renoir/Lucienne CPUs to
 force the D3 hint
In-Reply-To: <20210609184018.26017-2-mario.limonciello@amd.com>
References: <20210609184018.26017-1-mario.limonciello@amd.com>
 <20210609184018.26017-2-mario.limonciello@amd.com>
Message-ID: <831df669-b38e-9f4b-dd67-b613aaab6c28@amd.com>

On 6/9/2021 13:40, Mario Limonciello wrote:
> AMD systems from Renoir and Lucienne require that the NVME controller
> is put into D3 over a Modern Standby / suspend-to-idle
> cycle.  This is "typically" accomplished using the `StorageD3Enable`
> property in the _DSD, but this property was introduced after many
> of these systems launched and most OEM systems don't have it in
> their BIOS.
> 
> On AMD Renoir without these drives going into D3 over suspend-to-idle
> the resume will fail with the NVME controller being reset and a trace
> like this in the kernel logs:
> ```
> [   83.556118] nvme nvme0: I/O 161 QID 2 timeout, aborting
> [   83.556178] nvme nvme0: I/O 162 QID 2 timeout, aborting
> [   83.556187] nvme nvme0: I/O 163 QID 2 timeout, aborting
> [   83.556196] nvme nvme0: I/O 164 QID 2 timeout, aborting
> [   95.332114] nvme nvme0: I/O 25 QID 0 timeout, reset controller
> [   95.332843] nvme nvme0: Abort status: 0x371
> [   95.332852] nvme nvme0: Abort status: 0x371
> [   95.332856] nvme nvme0: Abort status: 0x371
> [   95.332859] nvme nvme0: Abort status: 0x371
> [   95.332909] PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -16
> [   95.332936] nvme 0000:03:00.0: PM: failed to resume async: error -16
> ```
> 
> The Microsoft documentation for StorageD3Enable mentioned that Windows has
> a hardcoded allowlist for D3 support, which was used for these platforms.
> Introduce quirks to hardcode them for Linux as well.
> 
> As this property is now "standardized", OEM systems using AMD Cezanne and
> newer APU's have adopted this property, and quirks like this should not be
> necessary.
> 
> CC: Shyam-sundar S-k <Shyam-sundar.S-k at amd.com>
> CC: Alexander Deucher <Alexander.Deucher at amd.com>
> CC: Prike Liang <prike.liang at amd.com>
> Link: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-intro
> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> Acked-by: Rafael J. Wysocki <rafael.j.wysocki at intel.com>
> Tested-by: Julian Sikorski <belegdol at gmail.com>
> ---
>   drivers/acpi/device_pm.c |  3 +++
>   drivers/acpi/internal.h  |  9 +++++++++
>   drivers/acpi/x86/utils.c | 25 +++++++++++++++++++++++++
>   3 files changed, 37 insertions(+)
>   
> Changes from v4->v5:
>   * Add this patch back in as it's been made apparent that the
>     system needs to be hardcoded for these.
>     Changes:
>     - Drop Cezanne - it's now covered by StorageD3Enable
>     - Rebase ontop of acpi_storage_d3 outside of NVME
> Changes from v5->v6:
>   * Move the quirk check into drivers/acpi/x86/ as suggested by
>     Rafael.
> Changes from v6->v7:
>   * Move header location
>   * Optimization of force function
> Changes from v7->v8:
>   * Add tags
> 
> diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
> index c8600978b283..c40fc135ff0c 100644
> --- a/drivers/acpi/device_pm.c
> +++ b/drivers/acpi/device_pm.c
> @@ -1360,6 +1360,9 @@ bool acpi_storage_d3(struct device *dev)
>   	struct acpi_device *adev = ACPI_COMPANION(dev);
>   	u8 val;
>   
> +	if (force_storage_d3())
> +		return true;
> +
>   	if (!adev)
>   		return false;
>   	if (fwnode_property_read_u8(acpi_fwnode_handle(adev), "StorageD3Enable",
> diff --git a/drivers/acpi/internal.h b/drivers/acpi/internal.h
> index f973bbe90e5e..e29ec463bb07 100644
> --- a/drivers/acpi/internal.h
> +++ b/drivers/acpi/internal.h
> @@ -236,6 +236,15 @@ static inline int suspend_nvs_save(void) { return 0; }
>   static inline void suspend_nvs_restore(void) {}
>   #endif
>   
> +#ifdef CONFIG_X86
> +bool force_storage_d3(void);
> +#else
> +static inline bool force_storage_d3(void)
> +{
> +	return false;
> +}
> +#endif
> +
>   /*--------------------------------------------------------------------------
>   				Device properties
>     -------------------------------------------------------------------------- */
> diff --git a/drivers/acpi/x86/utils.c b/drivers/acpi/x86/utils.c
> index bdc1ba00aee9..5298bb4d81fe 100644
> --- a/drivers/acpi/x86/utils.c
> +++ b/drivers/acpi/x86/utils.c
> @@ -135,3 +135,28 @@ bool acpi_device_always_present(struct acpi_device *adev)
>   
>   	return ret;
>   }
> +
> +/*
> + * AMD systems from Renoir and Lucienne *require* that the NVME controller
> + * is put into D3 over a Modern Standby / suspend-to-idle cycle.
> + *
> + * This is "typically" accomplished using the `StorageD3Enable`
> + * property in the _DSD that is checked via the `acpi_storage_d3` function
> + * but this property was introduced after many of these systems launched
> + * and most OEM systems don't have it in their BIOS.
> + *
> + * The Microsoft documentation for StorageD3Enable mentioned that Windows has
> + * a hardcoded allowlist for D3 support, which was used for these platforms.
> + *
> + * This allows quirking on Linux in a similar fashion.
> + */
> +const struct x86_cpu_id storage_d3_cpu_ids[] = {
> +	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),	/* Renoir */
> +	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL),	/* Lucienne */
> +	{}
> +};
> +
> +bool force_storage_d3(void)
> +{
> +	return x86_match_cpu(storage_d3_cpu_ids);
> +}
> 

Christoph, Keith ping on this series.  Any other thoughts?
If it's good can you take this in for nvme-5.14 since Rafael mentioned
the conflict from the other StorageD3Enable commit in nvme-5.14?



From hch at lst.de  Tue Jun 15 09:06:19 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 18:06:19 +0200
Subject: switch block layer polling to a bio based model v4
In-Reply-To: <20210615143706.GB646237@dhcp-10-100-145-180.wdc.com>
References: <20210615131034.752623-1-hch@lst.de>
 <20210615143706.GB646237@dhcp-10-100-145-180.wdc.com>
Message-ID: <20210615160619.GA32435@lst.de>

On Tue, Jun 15, 2021 at 07:37:06AM -0700, Keith Busch wrote:
> On Tue, Jun 15, 2021 at 03:10:18PM +0200, Christoph Hellwig wrote:
> > Chances since v3:
> >  - rebased to the latests for-5.14/block tree
> >  - fix the refcount logic in __blkdev_direct_IO
> >  - split up a patch to make it easier to review
> >  - grab a queue reference in bio_poll
> >  - better document the RCU assumptions in bio_poll
> 
> It still doesn't look like a failover will work when polling through
> pvsync2. We previously discussed that here:
> 
>   https://marc.info/?l=linux-block&m=162100971816071&w=2

Yes, sorry for losing this.  I'll resend with that added after a short
delay to see if anything else needs attention.


From hch at lst.de  Tue Jun 15 09:18:15 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 18:18:15 +0200
Subject: [PATCH v8 1/2] ACPI: Check StorageD3Enable _DSD property in
 ACPI code
In-Reply-To: <20210609184018.26017-1-mario.limonciello@amd.com>
References: <20210609184018.26017-1-mario.limonciello@amd.com>
Message-ID: <20210615161815.GA986@lst.de>

Thanks, I've applied the series to nvme-5.14.


From hch at lst.de  Tue Jun 15 09:18:25 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 18:18:25 +0200
Subject: [PATCH 0/4] nvme: add ctrl sgl supported helper
In-Reply-To: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
References: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210615161825.GB986@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Tue Jun 15 09:18:39 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 18:18:39 +0200
Subject: [PATCH 0/6] nvmet: small cleanup
In-Reply-To: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
References: <20210614015851.53925-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210615161839.GC986@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Tue Jun 15 09:18:51 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 18:18:51 +0200
Subject: [PATCH trivial] nvme: Grammar s/each namespaces/each namespace/
In-Reply-To: <d195ebc405033c1de6ca01937a906afc97f285f7.1623680137.git.geert+renesas@glider.be>
References: <d195ebc405033c1de6ca01937a906afc97f285f7.1623680137.git.geert+renesas@glider.be>
Message-ID: <20210615161851.GD986@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Tue Jun 15 09:19:01 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 18:19:01 +0200
Subject: [PATCH 0/4] nvme-fabrics: remove struct zeroout memset calls
In-Reply-To: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
References: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210615161901.GE986@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Tue Jun 15 09:25:07 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 18:25:07 +0200
Subject: [PATCH V15 0/5] nvmet: add ZBD backened support
In-Reply-To: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
References: <20210610013252.53874-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210615162507.GA1089@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Tue Jun 15 09:27:01 2021
From: hch at lst.de (hch at lst.de)
Date: Tue, 15 Jun 2021 18:27:01 +0200
Subject: [PATCH V6] nvme-core: use xarray for ctrl ns tracking
In-Reply-To: <BYAPR04MB49657620DF9C7EA9158DB60686369@BYAPR04MB4965.namprd04.prod.outlook.com>
References: <20210608024247.35286-1-chaitanya.kulkarni@wdc.com>
 <dbab5916-a6de-5cd0-4bb5-b06d0349df0b@grimberg.me>
 <BYAPR04MB49657620DF9C7EA9158DB60686369@BYAPR04MB4965.namprd04.prod.outlook.com>
Message-ID: <20210615162701.GB1089@lst.de>

On Wed, Jun 09, 2021 at 08:48:10PM +0000, Chaitanya Kulkarni wrote:
> 
> This is to void the another call to the synchronize_rcu(),
> 
> I'll add a second call to synchronize_rcu() and keep the original
> position.

Or document why this is changed in the commit log, that is what it is
for.


From hch at infradead.org  Tue Jun 15 09:39:54 2021
From: hch at infradead.org (Christoph Hellwig)
Date: Tue, 15 Jun 2021 17:39:54 +0100
Subject: nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <CALCePG0up2GK_ihFnLxHuUS4Zn0kfSAC2x1U4Vm_6BQw3Deo5w@mail.gmail.com>
References: <CALCePG0up2GK_ihFnLxHuUS4Zn0kfSAC2x1U4Vm_6BQw3Deo5w@mail.gmail.com>
Message-ID: <YMjX2qpSQCjVkToL@infradead.org>

The patch is so wrapped that it not only doesn't apply, but that I also
have a hard time reading it.  Can you resend it using git-send-email?


From sagi at grimberg.me  Tue Jun 15 11:02:10 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 15 Jun 2021 11:02:10 -0700
Subject: [PATCH] nvmet-tcp: Add a multex lock to start_queue failure case
In-Reply-To: <20210610084112.30400-1-amit.engel@dell.com>
References: <20210610084112.30400-1-amit.engel@dell.com>
Message-ID: <c79420e6-8110-4367-8361-cf1244f60318@grimberg.me>


> From: Amit Engel <amit.engel at dell.com>
> 
> this is in order to serialize nvme_tcp_stop_queue

Thinking about this, I don't think this is needed, can you describe
what is the scenario that requires this protection?


From kbusch at kernel.org  Tue Jun 15 11:42:26 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 15 Jun 2021 11:42:26 -0700
Subject: [PATCH 0/4] nvme-fabrics: remove struct zeroout memset calls
In-Reply-To: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
References: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210615184226.GC646237@dhcp-10-100-145-180.wdc.com>

On Mon, Jun 14, 2021 at 07:45:50PM -0700, Chaitanya Kulkarni wrote:
> Hi,
> 
> While reviewing the code I did removed the calls to zero out structures
> using memset() for the lio target, which are reviewed and accepted in
> the linux kernel tree. (see [1])
> 
> This patch series is based on the same cleanup pattern for NVMe host.
> This will allow us to have uniform pattern in the future code review to
> avoid similar calls which seems unnecessary.
> 
> If this is accepted I'll gradually convert the rest of the host code
> mainly core and pci.

This looks good, and aligns with the local style. For future cleanups, I
think it would be fine to do it a single patch instead of individual
patches per function.

Reviewed-by: Keith Busch <kbusch at kernel.org>


From lkp at intel.com  Tue Jun 15 12:45:11 2021
From: lkp at intel.com (kernel test robot)
Date: Wed, 16 Jun 2021 03:45:11 +0800
Subject: [linux-nvme:nvme-5.14 32/32] drivers/nvme/target/zns.c:257:6:
 warning: no previous prototype for 'nvmet_bdev_zone_zmgmt_recv_work'
Message-ID: <202106160306.n1I35Kah-lkp@intel.com>

tree:   git://git.infradead.org/nvme.git nvme-5.14
head:   9653e01d9c3c46930da3f007a39a2aaf230bd8ff
commit: 9653e01d9c3c46930da3f007a39a2aaf230bd8ff [32/32] nvmet: add ZBD over ZNS backend support
config: m68k-allmodconfig (attached as .config)
compiler: m68k-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        git remote add linux-nvme git://git.infradead.org/nvme.git
        git fetch --no-tags linux-nvme nvme-5.14
        git checkout 9653e01d9c3c46930da3f007a39a2aaf230bd8ff
        # save the attached .config to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=m68k 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp at intel.com>

All warnings (new ones prefixed by >>):

>> drivers/nvme/target/zns.c:257:6: warning: no previous prototype for 'nvmet_bdev_zone_zmgmt_recv_work' [-Wmissing-prototypes]
     257 | void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
         |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


vim +/nvmet_bdev_zone_zmgmt_recv_work +257 drivers/nvme/target/zns.c

   256	
 > 257	void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
   258	{
   259		struct nvmet_req *req = container_of(w, struct nvmet_req, z.zmgmt_work);
   260		sector_t start_sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
   261		unsigned long req_slba_nr_zones = nvmet_req_nr_zones_from_slba(req);
   262		u32 out_bufsize = (le32_to_cpu(req->cmd->zmr.numd) + 1) << 2;
   263		__le64 nr_zones;
   264		u16 status;
   265		int ret;
   266		struct nvmet_report_zone_data rz_data = {
   267			.out_nr_zones = get_nr_zones_from_buf(req, out_bufsize),
   268			/* leave the place for report zone header */
   269			.out_buf_offset = sizeof(struct nvme_zone_report),
   270			.zrasf = req->cmd->zmr.zrasf,
   271			.nr_zones = 0,
   272			.req = req,
   273		};
   274	
   275		status = nvmet_bdev_validate_zone_mgmt_recv(req);
   276		if (status)
   277			goto out;
   278	
   279		if (!req_slba_nr_zones) {
   280			status = NVME_SC_SUCCESS;
   281			goto out;
   282		}
   283	
   284		ret = blkdev_report_zones(req->ns->bdev, start_sect, req_slba_nr_zones,
   285					 nvmet_bdev_report_zone_cb, &rz_data);
   286		if (ret < 0) {
   287			status = NVME_SC_INTERNAL;
   288			goto out;
   289		}
   290	
   291		/*
   292		 * When partial bit is set nr_zones must indicate the number of zone
   293		 * descriptors actually transferred.
   294		 */
   295		if (req->cmd->zmr.pr)
   296			rz_data.nr_zones = min(rz_data.nr_zones, rz_data.out_nr_zones);
   297	
   298		nr_zones = cpu_to_le64(rz_data.nr_zones);
   299		status = nvmet_copy_to_sgl(req, 0, &nr_zones, sizeof(nr_zones));
   300	
   301	out:
   302		nvmet_req_complete(req, status);
   303	}
   304	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 60393 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20210616/5fb16d66/attachment-0001.gz>

From helgaas at kernel.org  Tue Jun 15 12:57:07 2021
From: helgaas at kernel.org (Bjorn Helgaas)
Date: Tue, 15 Jun 2021 14:57:07 -0500
Subject: [patch v6 3/7] genirq/affinity: Add new callback for
 (re)calculating interrupt sets
In-Reply-To: <20190216172228.512444498@linutronix.de>
Message-ID: <20210615195707.GA2909907@bjorn-Precision-5520>

On Sat, Feb 16, 2019 at 06:13:09PM +0100, Thomas Gleixner wrote:
> From: Ming Lei <ming.lei at redhat.com>
> 
> The interrupt affinity spreading mechanism supports to spread out
> affinities for one or more interrupt sets. A interrupt set contains one or
> more interrupts. Each set is mapped to a specific functionality of a
> device, e.g. general I/O queues and read I/O queus of multiqueue block
> devices.
> 
> The number of interrupts per set is defined by the driver. It depends on
> the total number of available interrupts for the device, which is
> determined by the PCI capabilites and the availability of underlying CPU
> resources, and the number of queues which the device provides and the
> driver wants to instantiate.
> 
> The driver passes initial configuration for the interrupt allocation via a
> pointer to struct irq_affinity.
> 
> Right now the allocation mechanism is complex as it requires to have a loop
> in the driver to determine the maximum number of interrupts which are
> provided by the PCI capabilities and the underlying CPU resources.  This
> loop would have to be replicated in every driver which wants to utilize
> this mechanism. That's unwanted code duplication and error prone.
> 
> In order to move this into generic facilities it is required to have a
> mechanism, which allows the recalculation of the interrupt sets and their
> size, in the core code. As the core code does not have any knowledge about the
> underlying device, a driver specific callback is required in struct
> irq_affinity, which can be invoked by the core code. The callback gets the
> number of available interupts as an argument, so the driver can calculate the
> corresponding number and size of interrupt sets.
> 
> At the moment the struct irq_affinity pointer which is handed in from the
> driver and passed through to several core functions is marked 'const', but for
> the callback to be able to modify the data in the struct it's required to
> remove the 'const' qualifier.
> 
> Add the optional callback to struct irq_affinity, which allows drivers to
> recalculate the number and size of interrupt sets and remove the 'const'
> qualifier.
> 
> For simple invocations, which do not supply a callback, a default callback
> is installed, which just sets nr_sets to 1 and transfers the number of
> spreadable vectors to the set_size array at index 0.
> 
> This is for now guarded by a check for nr_sets != 0 to keep the NVME driver
> working until it is converted to the callback mechanism.
> 
> To make sure that the driver configuration is correct under all circumstances
> the callback is invoked even when there are no interrupts for queues left,
> i.e. the pre/post requirements already exhaust the numner of available
> interrupts.
> 
> At the PCI layer irq_create_affinity_masks() has to be invoked even for the
> case where the legacy interrupt is used. That ensures that the callback is
> invoked and the device driver can adjust to that situation.
> 
> [ tglx: Fixed the simple case (no sets required). Moved the sanity check
>   	for nr_sets after the invocation of the callback so it catches
>   	broken drivers. Fixed the kernel doc comments for struct
>   	irq_affinity and de-'This patch'-ed the changelog ]
> 
> Signed-off-by: Ming Lei <ming.lei at redhat.com>
> Signed-off-by: Thomas Gleixner <tglx at linutronix.de>

> @@ -1196,6 +1196,13 @@ int pci_alloc_irq_vectors_affinity(struc
>  	/* use legacy irq if allowed */
>  	if (flags & PCI_IRQ_LEGACY) {
>  		if (min_vecs == 1 && dev->irq) {
> +			/*
> +			 * Invoke the affinity spreading logic to ensure that
> +			 * the device driver can adjust queue configuration
> +			 * for the single interrupt case.
> +			 */
> +			if (affd)
> +				irq_create_affinity_masks(1, affd);

This looks like a leak because irq_create_affinity_masks() returns a
pointer to kcalloc()ed space, but we throw away the pointer.

Or is there something very subtle going on here, like this special
case doesn't allocate anything?  I do see the "Nothing to assign?"
case that returns NULL with no alloc, but it's not completely trivial
to verify that we take that case here.

>  			pci_intx(dev, 1);
>  			return 1;
>  		}


From hch at lst.de  Tue Jun 15 13:04:43 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 15 Jun 2021 22:04:43 +0200
Subject: [patch v6 3/7] genirq/affinity: Add new callback for
 (re)calculating interrupt sets
In-Reply-To: <20210615195707.GA2909907@bjorn-Precision-5520>
References: <20190216172228.512444498@linutronix.de>
 <20210615195707.GA2909907@bjorn-Precision-5520>
Message-ID: <20210615200443.GA6557@lst.de>

On Tue, Jun 15, 2021 at 02:57:07PM -0500, Bjorn Helgaas wrote:
> On Sat, Feb 16, 2019 at 06:13:09PM +0100, Thomas Gleixner wrote:
> > From: Ming Lei <ming.lei at redhat.com>
> > 
> > The interrupt affinity spreading mechanism supports to spread out
> > affinities for one or more interrupt sets. A interrupt set contains one or
> > more interrupts. Each set is mapped to a specific functionality of a
> > device, e.g. general I/O queues and read I/O queus of multiqueue block
> > devices.
> > 
> > The number of interrupts per set is defined by the driver. It depends on
> > the total number of available interrupts for the device, which is
> > determined by the PCI capabilites and the availability of underlying CPU
> > resources, and the number of queues which the device provides and the
> > driver wants to instantiate.
> > 
> > The driver passes initial configuration for the interrupt allocation via a
> > pointer to struct irq_affinity.
> > 
> > Right now the allocation mechanism is complex as it requires to have a loop
> > in the driver to determine the maximum number of interrupts which are
> > provided by the PCI capabilities and the underlying CPU resources.  This
> > loop would have to be replicated in every driver which wants to utilize
> > this mechanism. That's unwanted code duplication and error prone.
> > 
> > In order to move this into generic facilities it is required to have a
> > mechanism, which allows the recalculation of the interrupt sets and their
> > size, in the core code. As the core code does not have any knowledge about the
> > underlying device, a driver specific callback is required in struct
> > irq_affinity, which can be invoked by the core code. The callback gets the
> > number of available interupts as an argument, so the driver can calculate the
> > corresponding number and size of interrupt sets.
> > 
> > At the moment the struct irq_affinity pointer which is handed in from the
> > driver and passed through to several core functions is marked 'const', but for
> > the callback to be able to modify the data in the struct it's required to
> > remove the 'const' qualifier.
> > 
> > Add the optional callback to struct irq_affinity, which allows drivers to
> > recalculate the number and size of interrupt sets and remove the 'const'
> > qualifier.
> > 
> > For simple invocations, which do not supply a callback, a default callback
> > is installed, which just sets nr_sets to 1 and transfers the number of
> > spreadable vectors to the set_size array at index 0.
> > 
> > This is for now guarded by a check for nr_sets != 0 to keep the NVME driver
> > working until it is converted to the callback mechanism.
> > 
> > To make sure that the driver configuration is correct under all circumstances
> > the callback is invoked even when there are no interrupts for queues left,
> > i.e. the pre/post requirements already exhaust the numner of available
> > interrupts.
> > 
> > At the PCI layer irq_create_affinity_masks() has to be invoked even for the
> > case where the legacy interrupt is used. That ensures that the callback is
> > invoked and the device driver can adjust to that situation.
> > 
> > [ tglx: Fixed the simple case (no sets required). Moved the sanity check
> >   	for nr_sets after the invocation of the callback so it catches
> >   	broken drivers. Fixed the kernel doc comments for struct
> >   	irq_affinity and de-'This patch'-ed the changelog ]
> > 
> > Signed-off-by: Ming Lei <ming.lei at redhat.com>
> > Signed-off-by: Thomas Gleixner <tglx at linutronix.de>
> 
> > @@ -1196,6 +1196,13 @@ int pci_alloc_irq_vectors_affinity(struc
> >  	/* use legacy irq if allowed */
> >  	if (flags & PCI_IRQ_LEGACY) {
> >  		if (min_vecs == 1 && dev->irq) {
> > +			/*
> > +			 * Invoke the affinity spreading logic to ensure that
> > +			 * the device driver can adjust queue configuration
> > +			 * for the single interrupt case.
> > +			 */
> > +			if (affd)
> > +				irq_create_affinity_masks(1, affd);
> 
> This looks like a leak because irq_create_affinity_masks() returns a
> pointer to kcalloc()ed space, but we throw away the pointer.
> 
> Or is there something very subtle going on here, like this special
> case doesn't allocate anything?  I do see the "Nothing to assign?"
> case that returns NULL with no alloc, but it's not completely trivial
> to verify that we take that case here.
> 
> >  			pci_intx(dev, 1);
> >  			return 1;
> >  		}
---end quoted text---


From Chaitanya.Kulkarni at wdc.com  Tue Jun 15 13:04:45 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue, 15 Jun 2021 20:04:45 +0000
Subject: [linux-nvme:nvme-5.14 32/32] drivers/nvme/target/zns.c:257:6:
 warning: no previous prototype for 'nvmet_bdev_zone_zmgmt_recv_work'
References: <202106160306.n1I35Kah-lkp@intel.com>
Message-ID: <BYAPR04MB49655BF0B42EEEC3DC86FD4186309@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/15/21 12:46, kernel test robot wrote:
> tree:   git://git.infradead.org/nvme.git nvme-5.14
> head:   9653e01d9c3c46930da3f007a39a2aaf230bd8ff
> commit: 9653e01d9c3c46930da3f007a39a2aaf230bd8ff [32/32] nvmet: add ZBD over ZNS backend support
> config: m68k-allmodconfig (attached as .config)
> compiler: m68k-linux-gcc (GCC) 9.3.0
> reproduce (this is a W=1 build):
>         wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>         chmod +x ~/bin/make.cross
>         git remote add linux-nvme git://git.infradead.org/nvme.git
>         git fetch --no-tags linux-nvme nvme-5.14
>         git checkout 9653e01d9c3c46930da3f007a39a2aaf230bd8ff
>         # save the attached .config to linux build tree
>         COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=m68k 
>
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp at intel.com>
>
> All warnings (new ones prefixed by >>):
>
>>> drivers/nvme/target/zns.c:257:6: warning: no previous prototype for 'nvmet_bdev_zone_zmgmt_recv_work' [-Wmissing-prototypes]
>      257 | void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
>          |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
> vim +/nvmet_bdev_zone_zmgmt_recv_work +257 drivers/nvme/target/zns.c
>
>    256	
>  > 257	void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
>    258	{
>    259		struct nvmet_req *req = container_of(w, struct nvmet_req, z.zmgmt_work);
>    260		sector_t start_sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
>    261		unsigned long req_slba_nr_zones = nvmet_req_nr_zones_from_slba(req);
>    262		u32 out_bufsize = (le32_to_cpu(req->cmd->zmr.numd) + 1) << 2;
>    263		__le64 nr_zones;
>    264		u16 status;
>    265		int ret;
>    266		struct nvmet_report_zone_data rz_data = {
>    267			.out_nr_zones = get_nr_zones_from_buf(req, out_bufsize),
>    268			/* leave the place for report zone header */
>    269			.out_buf_offset = sizeof(struct nvme_zone_report),
>    270			.zrasf = req->cmd->zmr.zrasf,
>    271			.nr_zones = 0,
>    272			.req = req,
>    273		};
>    274	
>    275		status = nvmet_bdev_validate_zone_mgmt_recv(req);
>    276		if (status)
>    277			goto out;
>    278	
>    279		if (!req_slba_nr_zones) {
>    280			status = NVME_SC_SUCCESS;
>    281			goto out;
>    282		}
>    283	
>    284		ret = blkdev_report_zones(req->ns->bdev, start_sect, req_slba_nr_zones,
>    285					 nvmet_bdev_report_zone_cb, &rz_data);
>    286		if (ret < 0) {
>    287			status = NVME_SC_INTERNAL;
>    288			goto out;
>    289		}
>    290	
>    291		/*
>    292		 * When partial bit is set nr_zones must indicate the number of zone
>    293		 * descriptors actually transferred.
>    294		 */
>    295		if (req->cmd->zmr.pr)
>    296			rz_data.nr_zones = min(rz_data.nr_zones, rz_data.out_nr_zones);
>    297	
>    298		nr_zones = cpu_to_le64(rz_data.nr_zones);
>    299		status = nvmet_copy_to_sgl(req, 0, &nr_zones, sizeof(nr_zones));
>    300	
>    301	out:
>    302		nvmet_req_complete(req, status);
>    303	}
>    304	
>
> ---
> 0-DAY CI Kernel Test Service, Intel Corporation
> https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org

I don't understand why does this need a prototype ?

The function nvmet_bdev_zone_zmgmt_recv_work() is defined  (line 257)
before it was used (line 307).

Will try and reproduce this on latest 5.14.






From Chaitanya.Kulkarni at wdc.com  Tue Jun 15 13:07:04 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue, 15 Jun 2021 20:07:04 +0000
Subject: [PATCH 0/4] nvme-fabrics: remove struct zeroout memset calls
References: <20210615024554.42585-1-chaitanya.kulkarni@wdc.com>
 <20210615184226.GC646237@dhcp-10-100-145-180.wdc.com>
Message-ID: <BYAPR04MB4965C9ECA762535BCFB5532286309@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/15/21 11:42, Keith Busch wrote:
>> If this is accepted I'll gradually convert the rest of the host code
>> mainly core and pci.
> This looks good, and aligns with the local style. For future cleanups, I
> think it would be fine to do it a single patch instead of individual
> patches per function.
>
> Reviewed-by: Keith Busch <kbusch at kernel.org>
>

Sound good will send soon.




From kbusch at kernel.org  Tue Jun 15 13:53:18 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 15 Jun 2021 13:53:18 -0700
Subject: nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <CALCePG2ukbm=mjZiKij5_3T4sTc4H5wRuityrr2Ljv0HvfJqtQ@mail.gmail.com>
References: <CALCePG2ukbm=mjZiKij5_3T4sTc4H5wRuityrr2Ljv0HvfJqtQ@mail.gmail.com>
Message-ID: <20210615205318.GD646237@dhcp-10-100-145-180.wdc.com>

On Mon, Jun 14, 2021 at 11:26:55PM -0700, Casey Chen wrote:
> (Please ignore the previous email, the call paths comparison shown in
> commit message should look better if you copy then paste in a code
> editor)

Your email client also mangles your patch, rendering it unable to apply.
If you're able to set up 'git send-email' from your development machine,
that will always format correctly for mailing list patch consumption.

I am aware of the race condition you've described. I never bothered with
it because of the unsual circumstances to hit it, but since you have
identified a test case, I agree it's time we address it.

> ---
>  drivers/nvme/host/pci.c | 33 +++++++++++++++++++++++++++------
>  1 file changed, 27 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 3aa7245a505f..81e53aaaa77c 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -1590,8 +1590,9 @@ static int nvme_create_queue(struct nvme_queue
> *nvmeq, int qid, bool polled)
>   goto release_cq;
> 
>   nvmeq->cq_vector = vector;
> - nvme_init_queue(nvmeq, qid);
> 
> + mutex_lock(&dev->shutdown_lock);

There doesn't seem to be a reason to wait for the lock here. A
mutex_try_lock() should be fine, and simply abandon queue initialization
if we locking fails.


From lkp at intel.com  Tue Jun 15 14:39:43 2021
From: lkp at intel.com (kernel test robot)
Date: Wed, 16 Jun 2021 05:39:43 +0800
Subject: [linux-nvme:nvme-5.14 32/32] drivers/nvme/target/zns.c:257:6:
 warning: no previous prototype for function
 'nvmet_bdev_zone_zmgmt_recv_work'
Message-ID: <202106160534.PvpGDOHM-lkp@intel.com>

tree:   git://git.infradead.org/nvme.git nvme-5.14
head:   9653e01d9c3c46930da3f007a39a2aaf230bd8ff
commit: 9653e01d9c3c46930da3f007a39a2aaf230bd8ff [32/32] nvmet: add ZBD over ZNS backend support
config: x86_64-randconfig-a014-20210615 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 64720f57bea6a6bf033feef4a5751ab9c0c3b401)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # install x86_64 cross compiling tool for clang build
        # apt-get install binutils-x86-64-linux-gnu
        git remote add linux-nvme git://git.infradead.org/nvme.git
        git fetch --no-tags linux-nvme nvme-5.14
        git checkout 9653e01d9c3c46930da3f007a39a2aaf230bd8ff
        # save the attached .config to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp at intel.com>

All warnings (new ones prefixed by >>):

>> drivers/nvme/target/zns.c:257:6: warning: no previous prototype for function 'nvmet_bdev_zone_zmgmt_recv_work' [-Wmissing-prototypes]
   void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
        ^
   drivers/nvme/target/zns.c:257:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
   void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
   ^
   static 
   1 warning generated.


vim +/nvmet_bdev_zone_zmgmt_recv_work +257 drivers/nvme/target/zns.c

   256	
 > 257	void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
   258	{
   259		struct nvmet_req *req = container_of(w, struct nvmet_req, z.zmgmt_work);
   260		sector_t start_sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
   261		unsigned long req_slba_nr_zones = nvmet_req_nr_zones_from_slba(req);
   262		u32 out_bufsize = (le32_to_cpu(req->cmd->zmr.numd) + 1) << 2;
   263		__le64 nr_zones;
   264		u16 status;
   265		int ret;
   266		struct nvmet_report_zone_data rz_data = {
   267			.out_nr_zones = get_nr_zones_from_buf(req, out_bufsize),
   268			/* leave the place for report zone header */
   269			.out_buf_offset = sizeof(struct nvme_zone_report),
   270			.zrasf = req->cmd->zmr.zrasf,
   271			.nr_zones = 0,
   272			.req = req,
   273		};
   274	
   275		status = nvmet_bdev_validate_zone_mgmt_recv(req);
   276		if (status)
   277			goto out;
   278	
   279		if (!req_slba_nr_zones) {
   280			status = NVME_SC_SUCCESS;
   281			goto out;
   282		}
   283	
   284		ret = blkdev_report_zones(req->ns->bdev, start_sect, req_slba_nr_zones,
   285					 nvmet_bdev_report_zone_cb, &rz_data);
   286		if (ret < 0) {
   287			status = NVME_SC_INTERNAL;
   288			goto out;
   289		}
   290	
   291		/*
   292		 * When partial bit is set nr_zones must indicate the number of zone
   293		 * descriptors actually transferred.
   294		 */
   295		if (req->cmd->zmr.pr)
   296			rz_data.nr_zones = min(rz_data.nr_zones, rz_data.out_nr_zones);
   297	
   298		nr_zones = cpu_to_le64(rz_data.nr_zones);
   299		status = nvmet_copy_to_sgl(req, 0, &nr_zones, sizeof(nr_zones));
   300	
   301	out:
   302		nvmet_req_complete(req, status);
   303	}
   304	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 36588 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20210616/1ef6d285/attachment-0001.gz>

From Chaitanya.Kulkarni at wdc.com  Tue Jun 15 14:41:48 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue, 15 Jun 2021 21:41:48 +0000
Subject: [linux-nvme:nvme-5.14 32/32] drivers/nvme/target/zns.c:257:6:
 warning: no previous prototype for function 'nvmet_bdev_zone_zmgmt_recv_work'
In-Reply-To: <202106160534.PvpGDOHM-lkp@intel.com>
References: <202106160534.PvpGDOHM-lkp@intel.com>
Message-ID: <472070C8-85CE-4C02-A96C-4D163A7D9296@wdc.com>


> On Jun 15, 2021, at 2:40 PM, kernel test robot <lkp at intel.com> wrote:
> 
>  void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
>        ^
>   drivers/nvme/target/zns.c:257:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
>   void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
>   ^
>   static 
>   1 warning generated.
> 
> 
> vim +/nvmet_bdev_zone_zmgmt_recv_work +257 drivers/nvme/target/zns.c

Yep got it now, will send out the fix soon. 



From cachen at purestorage.com  Tue Jun 15 14:57:40 2021
From: cachen at purestorage.com (Casey Chen)
Date: Tue, 15 Jun 2021 14:57:40 -0700
Subject: Review Request: nvme-pci: Fix multiple races in nvme_setup_io_queues()
Message-ID: <20210615215741.26844-1-cachen@purestorage.com>

All tests and fix are based on tag nvme-5.14-2021-06-08 of repo
http://git.infradead.org/nvme.git

Testing method
while :; power off one drive;
    sleep $((RANDOM%3)).$((RANDOM%10));
    power on the same drive;
    sleep $((RANDOM%3)).$((RANDOM%10));
done

Sample crash call trace
[11668.533431] pcieport 0000:87:08.0: pciehp: Slot(402): Card present
...
[11668.681298] nvme nvme12: pci function 0000:8c:00.0
[11668.681354] nvme 0000:8c:00.0: enabling device (0100 -> 0102)
[11669.046142] pcieport 0000:87:08.0: pciehp: Slot(402): Link Down
[11669.046146] pcieport 0000:87:08.0: pciehp: Slot(402): Card not present
[11669.077428] ------------[ cut here ]------------
[11669.077431] kernel BUG at drivers/pci/msi.c:348!
[11669.077555] invalid opcode: 0000 [#1] SMP KASAN
[11669.077658] CPU: 31 PID: 716 Comm: irq/127-pciehp Not tainted 5.13.0-rc3+
[11669.077869] Hardware name: <MASKED OFF>
[11669.078022] RIP: 0010:free_msi_irqs+0x28a/0x2d0
...
[11669.093982] Call Trace:
[11669.096850]  pci_free_irq_vectors+0xe/0x20
[11669.099695]  nvme_dev_disable+0x140/0x760 [nvme]
[11669.102503]  ? _raw_spin_lock_irqsave+0x9c/0x100
[11669.105271]  ? trace_hardirqs_on+0x2c/0xe0
[11669.107994]  nvme_remove+0x191/0x1e0 [nvme]
[11669.110689]  pci_device_remove+0x6b/0x110
[11669.113316]  device_release_driver_internal+0x14f/0x280
[11669.115939]  pci_stop_bus_device+0xcb/0x100
[11669.118515]  pci_stop_and_remove_bus_device+0xe/0x20
[11669.121079]  pciehp_unconfigure_device+0xfa/0x200
[11669.123597]  ? pciehp_configure_device+0x1c0/0x1c0
[11669.126049]  ? trace_hardirqs_on+0x2c/0xe0
[11669.128444]  pciehp_disable_slot+0xc4/0x1a0
[11669.130771]  ? pciehp_runtime_suspend+0x40/0x40
[11669.133054]  ? __mutex_lock_slowpath+0x10/0x10
[11669.135289]  ? trace_hardirqs_on+0x2c/0xe0
[11669.137462]  pciehp_handle_presence_or_link_change+0x15c/0x4f0
[11669.139632]  ? down_read+0x11f/0x1a0
[11669.141731]  ? pciehp_handle_disable_request+0x80/0x80
[11669.143817]  ? rwsem_down_read_slowpath+0x600/0x600
[11669.145851]  ? __radix_tree_lookup+0xb2/0x130
[11669.147842]  pciehp_ist+0x19d/0x1a0
[11669.149790]  ? pciehp_set_indicators+0xe0/0xe0
[11669.151704]  ? irq_finalize_oneshot.part.46+0x1d0/0x1d0
[11669.153588]  irq_thread_fn+0x3f/0xa0
[11669.155407]  irq_thread+0x195/0x290
[11669.157147]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
[11669.158883]  ? _raw_read_lock_irq+0x50/0x50
[11669.160611]  ? _raw_read_lock_irq+0x50/0x50
[11669.162320]  ? irq_forced_thread_fn+0xf0/0xf0
[11669.164032]  ? trace_hardirqs_on+0x2c/0xe0
[11669.165731]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
[11669.167461]  kthread+0x1c8/0x1f0
[11669.169173]  ? kthread_parkme+0x40/0x40
[11669.170883]  ret_from_fork+0x22/0x30

From: Casey Chen <cachen at purestorage.com>
Reply-To: 
Subject: Review Request: nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: 




From cachen at purestorage.com  Tue Jun 15 14:57:41 2021
From: cachen at purestorage.com (Casey Chen)
Date: Tue, 15 Jun 2021 14:57:41 -0700
Subject: [PATCH] nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <20210615215741.26844-1-cachen@purestorage.com>
References: <20210615215741.26844-1-cachen@purestorage.com>
Message-ID: <20210615215741.26844-2-cachen@purestorage.com>

Below two paths could overlap each other if we power off a drive quickly
after power it on. There are multiple races in nvme_setup_io_queues()
because of shutdown_lock missing and improper use of NVMEQ_ENABLED bit.

nvme_reset_work()                                nvme_remove()
  nvme_setup_io_queues()                           nvme_dev_disable()
  ...                                              ...
A1  clear NVMEQ_ENABLED bit for admin queue          lock
    retry:                                       B1  nvme_suspend_io_queues()
A2    pci_free_irq() admin queue                 B2  nvme_suspend_queue() admin queue
A3    pci_free_irq_vectors()                         nvme_pci_disable()
A4    nvme_setup_irqs();                         B3    pci_free_irq_vectors()
      ...                                            unlock
A5    queue_request_irq() for admin queue
      set NVMEQ_ENABLED bit
      ...
      nvme_create_io_queues()
A6      result = queue_request_irq();
        set NVMEQ_ENABLED bit
      ...
      fail to allocate enough IO queues:
A7      nvme_suspend_io_queues()
        goto retry

1). If B3 runs in between A1 and A2, it will crash if irqaction haven't
  been freed by A2. B2 is supposed to free admin queue IRQ but it simply
  can't fulfill the job as A1 has cleared NVMEQ_ENABLED bit.

Fix: combine A1 A2 so IRQ get freed as soon as NVMEQ_ENABLED bit get cleared.

2). After solved #1, A2 could race with B3 if A2 is freeing IRQ while B3
  is checking irqaction. A3 also could race with B2 if B2 is freeing
  IRQ while A3 is checking irqaction.

Fix: A2 and A3 take lock for mutual exclusion.

3) A3 could race with B3 since they could run free_msi_irqs() in parallel.

Fix: A3 takes lock for mutual exclusion.

4). A4 could fail to allocate all needed IRQ vectors if A3 and A4 are
  interrupted by B3.

Fix: A4 takes lock for mutual exclusion.

5). If A5/A6 happened after B2/B1, B3 will crash since irqaction is not NULL.
  They are just allocated by A5/A6.

Fix: Lock queue_request_irq() and setting of NVMEQ_ENABLED bit.

6). A7 could get chance to pci_free_irq() for certain IO queue while B3
  is checking irqaction.

Fix: A7 takes lock.

7). nvme_dev->online_queues need to be protected by shutdown_lock. Since it
  is not atomic, both paths could modify it using its own copy.

Co-developed-by: Yuanyuan Zhong <yzhong at purestorage.com>
Signed-off-by: Casey Chen <cachen at purestorage.com>
---
 drivers/nvme/host/pci.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..81e53aaaa77c 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1590,8 +1590,9 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
 		goto release_cq;
 
 	nvmeq->cq_vector = vector;
-	nvme_init_queue(nvmeq, qid);
 
+	mutex_lock(&dev->shutdown_lock);
+	nvme_init_queue(nvmeq, qid);
 	if (!polled) {
 		result = queue_request_irq(nvmeq);
 		if (result < 0)
@@ -1599,10 +1600,12 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
 	}
 
 	set_bit(NVMEQ_ENABLED, &nvmeq->flags);
+	mutex_unlock(&dev->shutdown_lock);
 	return result;
 
 release_sq:
 	dev->online_queues--;
+	mutex_unlock(&dev->shutdown_lock);
 	adapter_delete_sq(dev, qid);
 release_cq:
 	adapter_delete_cq(dev, qid);
@@ -2176,7 +2179,16 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	if (nr_io_queues == 0)
 		return 0;
 
-	clear_bit(NVMEQ_ENABLED, &adminq->flags);
+	/*
+	 * Free IRQ resources as soon as NVMEQ_ENABLED bit transitions
+	 * from set to unset. If there is a window to it is truely freed,
+	 * pci_free_irq_vectors() jumping into this window will crash.
+	 * And take lock to avoid racing with pci_free_irq_vectors() in
+	 * nvme_dev_disable() path.
+	 */
+	mutex_lock(&dev->shutdown_lock);
+	if (test_and_clear_bit(NVMEQ_ENABLED, &adminq->flags))
+		pci_free_irq(pdev, 0, adminq);
 
 	if (dev->cmb_use_sqes) {
 		result = nvme_cmb_qdepth(dev, nr_io_queues,
@@ -2192,14 +2204,17 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 		result = nvme_remap_bar(dev, size);
 		if (!result)
 			break;
-		if (!--nr_io_queues)
+		if (!--nr_io_queues) {
+			mutex_unlock(&dev->shutdown_lock);
 			return -ENOMEM;
+		}
 	} while (1);
 	adminq->q_db = dev->dbs;
 
  retry:
 	/* Deregister the admin queue's interrupt */
-	pci_free_irq(pdev, 0, adminq);
+	if (test_and_clear_bit(NVMEQ_ENABLED, &adminq->flags))
+		pci_free_irq(pdev, 0, adminq);
 
 	/*
 	 * If we enable msix early due to not intx, disable it again before
@@ -2208,8 +2223,10 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	pci_free_irq_vectors(pdev);
 
 	result = nvme_setup_irqs(dev, nr_io_queues);
-	if (result <= 0)
+	if (result <= 0) {
+		mutex_unlock(&dev->shutdown_lock);
 		return -EIO;
+	}
 
 	dev->num_vecs = result;
 	result = max(result - 1, 1);
@@ -2222,9 +2239,12 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	 * number of interrupts.
 	 */
 	result = queue_request_irq(adminq);
-	if (result)
+	if (result) {
+		mutex_unlock(&dev->shutdown_lock);
 		return result;
+	}
 	set_bit(NVMEQ_ENABLED, &adminq->flags);
+	mutex_unlock(&dev->shutdown_lock);
 
 	result = nvme_create_io_queues(dev);
 	if (result || dev->online_queues < 2)
@@ -2233,6 +2253,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	if (dev->online_queues - 1 < dev->max_qid) {
 		nr_io_queues = dev->online_queues - 1;
 		nvme_disable_io_queues(dev);
+		mutex_lock(&dev->shutdown_lock);
 		nvme_suspend_io_queues(dev);
 		goto retry;
 	}
-- 
2.17.1



From cachen at purestorage.com  Tue Jun 15 15:33:48 2021
From: cachen at purestorage.com (Casey Chen)
Date: Tue, 15 Jun 2021 15:33:48 -0700
Subject: nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <YMjX2qpSQCjVkToL@infradead.org>
References: <CALCePG0up2GK_ihFnLxHuUS4Zn0kfSAC2x1U4Vm_6BQw3Deo5w@mail.gmail.com>
 <YMjX2qpSQCjVkToL@infradead.org>
Message-ID: <CALCePG3BAbCJo2EDhv_w7kjJpcYUrO4=WAwb4tA-EtUKhBJhgg@mail.gmail.com>

Sorry for the previous bad emails I have sent.

Please discuss and comment on the latest email I sent using git
send-email at 2:58 PM PDT.
There were two emails: one is a summary with subject 'Review Request:
nvme-pci: Fix multiple races in nvme_setup_io_queues()'
The other is patch with subject '[PATCH] nvme-pci: Fix multiple races
in nvme_setup_io_queues()'

I am waiting for more comments and then resolve them together. Thanks!


On Tue, Jun 15, 2021 at 9:40 AM Christoph Hellwig <hch at infradead.org> wrote:
>
> The patch is so wrapped that it not only doesn't apply, but that I also
> have a hard time reading it.  Can you resend it using git-send-email?


From chaitanya.kulkarni at wdc.com  Tue Jun 15 17:18:43 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Tue, 15 Jun 2021 17:18:43 -0700
Subject: [PATCH] nvmet: mark zmgmt recv work function static
Message-ID: <20210616001844.58969-1-chaitanya.kulkarni@wdc.com>

The function nvmet_bdev_zone_zmgmt_recv_work() is only used in
target/zns.c. Mark this function static to avoid compiler warning :-

>> drivers/nvme/target/zns.c:257:6: warning: no previous prototype for
'nvmet_bdev_zone_zmgmt_recv_work' [-Wmissing-prototypes]
     257 | void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
         |

Reported-by: kernel test robot <lkp at intel.com>
Fixes: 9653e01d9c3c ("nvmet: add ZBD over ZNS backend support")
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/zns.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/target/zns.c b/drivers/nvme/target/zns.c
index b8ff75293e53..c4be4012c4d4 100644
--- a/drivers/nvme/target/zns.c
+++ b/drivers/nvme/target/zns.c
@@ -254,7 +254,7 @@ static unsigned long get_nr_zones_from_buf(struct nvmet_req *req, u32 bufsize)
 		sizeof(struct nvme_zone_descriptor);
 }
 
-void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
+static void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
 {
 	struct nvmet_req *req = container_of(w, struct nvmet_req, z.zmgmt_work);
 	sector_t start_sect = nvmet_lba_to_sect(req->ns, req->cmd->zmr.slba);
-- 
2.22.1



From ming.lei at redhat.com  Tue Jun 15 17:40:14 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Wed, 16 Jun 2021 08:40:14 +0800
Subject: [patch v6 3/7] genirq/affinity: Add new callback for
 (re)calculating interrupt sets
In-Reply-To: <20210615195707.GA2909907@bjorn-Precision-5520>
References: <20190216172228.512444498@linutronix.de>
 <20210615195707.GA2909907@bjorn-Precision-5520>
Message-ID: <YMlIbt3EPyRJHNWf@T590>

On Tue, Jun 15, 2021 at 02:57:07PM -0500, Bjorn Helgaas wrote:
> On Sat, Feb 16, 2019 at 06:13:09PM +0100, Thomas Gleixner wrote:
> > From: Ming Lei <ming.lei at redhat.com>
> > 
> > The interrupt affinity spreading mechanism supports to spread out
> > affinities for one or more interrupt sets. A interrupt set contains one or
> > more interrupts. Each set is mapped to a specific functionality of a
> > device, e.g. general I/O queues and read I/O queus of multiqueue block
> > devices.
> > 
> > The number of interrupts per set is defined by the driver. It depends on
> > the total number of available interrupts for the device, which is
> > determined by the PCI capabilites and the availability of underlying CPU
> > resources, and the number of queues which the device provides and the
> > driver wants to instantiate.
> > 
> > The driver passes initial configuration for the interrupt allocation via a
> > pointer to struct irq_affinity.
> > 
> > Right now the allocation mechanism is complex as it requires to have a loop
> > in the driver to determine the maximum number of interrupts which are
> > provided by the PCI capabilities and the underlying CPU resources.  This
> > loop would have to be replicated in every driver which wants to utilize
> > this mechanism. That's unwanted code duplication and error prone.
> > 
> > In order to move this into generic facilities it is required to have a
> > mechanism, which allows the recalculation of the interrupt sets and their
> > size, in the core code. As the core code does not have any knowledge about the
> > underlying device, a driver specific callback is required in struct
> > irq_affinity, which can be invoked by the core code. The callback gets the
> > number of available interupts as an argument, so the driver can calculate the
> > corresponding number and size of interrupt sets.
> > 
> > At the moment the struct irq_affinity pointer which is handed in from the
> > driver and passed through to several core functions is marked 'const', but for
> > the callback to be able to modify the data in the struct it's required to
> > remove the 'const' qualifier.
> > 
> > Add the optional callback to struct irq_affinity, which allows drivers to
> > recalculate the number and size of interrupt sets and remove the 'const'
> > qualifier.
> > 
> > For simple invocations, which do not supply a callback, a default callback
> > is installed, which just sets nr_sets to 1 and transfers the number of
> > spreadable vectors to the set_size array at index 0.
> > 
> > This is for now guarded by a check for nr_sets != 0 to keep the NVME driver
> > working until it is converted to the callback mechanism.
> > 
> > To make sure that the driver configuration is correct under all circumstances
> > the callback is invoked even when there are no interrupts for queues left,
> > i.e. the pre/post requirements already exhaust the numner of available
> > interrupts.
> > 
> > At the PCI layer irq_create_affinity_masks() has to be invoked even for the
> > case where the legacy interrupt is used. That ensures that the callback is
> > invoked and the device driver can adjust to that situation.
> > 
> > [ tglx: Fixed the simple case (no sets required). Moved the sanity check
> >   	for nr_sets after the invocation of the callback so it catches
> >   	broken drivers. Fixed the kernel doc comments for struct
> >   	irq_affinity and de-'This patch'-ed the changelog ]
> > 
> > Signed-off-by: Ming Lei <ming.lei at redhat.com>
> > Signed-off-by: Thomas Gleixner <tglx at linutronix.de>
> 
> > @@ -1196,6 +1196,13 @@ int pci_alloc_irq_vectors_affinity(struc
> >  	/* use legacy irq if allowed */
> >  	if (flags & PCI_IRQ_LEGACY) {
> >  		if (min_vecs == 1 && dev->irq) {
> > +			/*
> > +			 * Invoke the affinity spreading logic to ensure that
> > +			 * the device driver can adjust queue configuration
> > +			 * for the single interrupt case.
> > +			 */
> > +			if (affd)
> > +				irq_create_affinity_masks(1, affd);
> 
> This looks like a leak because irq_create_affinity_masks() returns a
> pointer to kcalloc()ed space, but we throw away the pointer.
> 
> Or is there something very subtle going on here, like this special
> case doesn't allocate anything?  I do see the "Nothing to assign?"
> case that returns NULL with no alloc, but it's not completely trivial
> to verify that we take that case here.

The purpose is to provide chance to call ->calc_sets() for single
interrupt, maybe it can be improved by the following change:

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 217dc9f0231f..025c647279f5 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -1223,8 +1223,7 @@ int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs,
 			 * the device driver can adjust queue configuration
 			 * for the single interrupt case.
 			 */
-			if (affd)
-				irq_create_affinity_masks(1, affd);
+			irq_affinity_calc_sets_legacy(affd);
 			pci_intx(dev, 1);
 			return 1;
 		}
diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index 4777850a6dc7..f21f93ce460b 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -368,6 +368,7 @@ irq_create_affinity_masks(unsigned int nvec, struct irq_affinity *affd);
 
 unsigned int irq_calc_affinity_vectors(unsigned int minvec, unsigned int maxvec,
 				       const struct irq_affinity *affd);
+void irq_affinity_calc_sets_legacy(struct irq_affinity *affd);
 
 #else /* CONFIG_SMP */
 
@@ -419,6 +420,10 @@ irq_calc_affinity_vectors(unsigned int minvec, unsigned int maxvec,
 	return maxvec;
 }
 
+static inline void irq_affinity_calc_sets_legacy(struct irq_affinity *affd)
+{
+}
+
 #endif /* CONFIG_SMP */
 
 /*
diff --git a/kernel/irq/affinity.c b/kernel/irq/affinity.c
index 4d89ad4fae3b..d01f7dfa5712 100644
--- a/kernel/irq/affinity.c
+++ b/kernel/irq/affinity.c
@@ -405,6 +405,30 @@ static void default_calc_sets(struct irq_affinity *affd, unsigned int affvecs)
 	affd->set_size[0] = affvecs;
 }
 
+static void irq_affinity_calc_sets(unsigned int affvecs,
+		struct irq_affinity *affd)
+{
+	/*
+	 * Simple invocations do not provide a calc_sets() callback. Install
+	 * the generic one.
+	 */
+	if (!affd->calc_sets)
+		affd->calc_sets = default_calc_sets;
+
+	/* Recalculate the sets */
+	affd->calc_sets(affd, affvecs);
+
+	WARN_ON_ONCE(affd->nr_sets > IRQ_AFFINITY_MAX_SETS);
+}
+
+/* Provide a chance to call ->calc_sets for legacy */
+void irq_affinity_calc_sets_legacy(struct irq_affinity *affd)
+{
+	if (!affd)
+		return;
+	irq_affinity_calc_sets(0, affd);
+}
+
 /**
  * irq_create_affinity_masks - Create affinity masks for multiqueue spreading
  * @nvecs:	The total number of vectors
@@ -429,17 +453,8 @@ irq_create_affinity_masks(unsigned int nvecs, struct irq_affinity *affd)
 	else
 		affvecs = 0;
 
-	/*
-	 * Simple invocations do not provide a calc_sets() callback. Install
-	 * the generic one.
-	 */
-	if (!affd->calc_sets)
-		affd->calc_sets = default_calc_sets;
-
-	/* Recalculate the sets */
-	affd->calc_sets(affd, affvecs);
-
-	if (WARN_ON_ONCE(affd->nr_sets > IRQ_AFFINITY_MAX_SETS))
+	irq_affinity_calc_sets(affvecs, affd);
+	if (affd->nr_sets > IRQ_AFFINITY_MAX_SETS)
 		return NULL;
 
 	/* Nothing to assign? */


Thanks, 
Ming



From ming.lei at redhat.com  Tue Jun 15 19:30:23 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Wed, 16 Jun 2021 10:30:23 +0800
Subject: [PATCH 13/16] block: switch polling to be bio based
In-Reply-To: <20210615131034.752623-14-hch@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
 <20210615131034.752623-14-hch@lst.de>
Message-ID: <YMliP6sFVuPhMbOB@T590>

On Tue, Jun 15, 2021 at 03:10:31PM +0200, Christoph Hellwig wrote:
> Replace the blk_poll interface that requires the caller to keep a queue
> and cookie from the submissions with polling based on the bio.
> 
> Polling for the bio itself leads to a few advantages:
> 
>  - the cookie construction can made entirely private in blk-mq.c
>  - the caller does not need to remember the request_queue and cookie
>    separately and thus sidesteps their lifetime issues
>  - keeping the device and the cookie inside the bio allows to trivially
>    support polling BIOs remapping by stacking drivers
>  - a lot of code to propagate the cookie back up the submission path can
>    be removed entirely.
> 

...

> +/**
> + * bio_poll - poll for BIO completions
> + * @bio: bio to poll for
> + * @flags: BLK_POLL_* flags that control the behavior
> + *
> + * Poll for completions on queue associated with the bio. Returns number of
> + * completed entries found.
> + *
> + * Note: the caller must either be the context that submitted @bio, or
> + * be in a RCU critical section to prevent freeing of @bio.
> + */
> +int bio_poll(struct bio *bio, unsigned int flags)
> +{
> +	struct request_queue *q = bio->bi_bdev->bd_disk->queue;
> +	blk_qc_t cookie = READ_ONCE(bio->bi_cookie);
> +	int ret;
> +
> +	if (cookie == BLK_QC_T_NONE ||
> +	    !test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
> +		return 0;
> +
> +	if (current->plug)
> +		blk_flush_plug_list(current->plug, false);
> +
> +	if (blk_queue_enter(q, BLK_MQ_REQ_NOWAIT))
> +		return 0;
> +	if (WARN_ON_ONCE(!queue_is_mq(q)))
> +		ret = 0;	/* not yet implemented, should not happen */
> +	else
> +		ret = blk_mq_poll(q, cookie, flags);
> +	blk_queue_exit(q);
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(bio_poll);
> +
> +/*
> + * Helper to implement file_operations.iopoll.  Requires the bio to be stored
> + * in iocb->private, and cleared before freeing the bio.
> + */
> +int iocb_bio_iopoll(struct kiocb *kiocb, unsigned int flags)
> +{
> +	struct bio *bio;
> +	int ret = 0;
> +
> +	/*
> +	 * Note: the bio cache only uses SLAB_TYPESAFE_BY_RCU, so bio can
> +	 * point to a freshly allocated bio at this point.  If that happens
> +	 * we have a few cases to consider:
> +	 *
> +	 *  1) the bio is beeing initialized and bi_bdev is NULL.  We can just
> +	 *     simply nothing in this case
> +	 *  2) the bio points to a not poll enabled device.  bio_poll will catch
> +	 *     this and return 0
> +	 *  3) the bio points to a poll capable device, including but not
> +	 *     limited to the one that the original bio pointed to.  In this
> +	 *     case we will call into the actual poll method and poll for I/O,
> +	 *     even if we don't need to, but it won't cause harm either.
> +	 *
> +	 * For cases 2) and 3) above the RCU grace period ensures that bi_bdev
> +	 * is still allocated. Because partitions hold a reference to the whole
> +	 * device bdev and thus disk, the disk is also still valid.  Grabbing
> +	 * a reference to the queue in bio_poll() ensures the hctxs and requests
> +	 * are still valid as well.
> +	 */

Not sure disk is valid, we only hold the disk when opening a bdev, but
the bdev can be closed during polling. Also disk always holds one
reference on request queue, so if disk is valid, no need to grab queue's
refcnt in bio_poll().


Thanks,
Ming



From lkp at intel.com  Tue Jun 15 20:22:19 2021
From: lkp at intel.com (kernel test robot)
Date: Wed, 16 Jun 2021 11:22:19 +0800
Subject: [RFC PATCH linux-nvme] ACPI: storage_d3_cpu_ids[] can be static
In-Reply-To: <202106161145.HdBOaJkF-lkp@intel.com>
References: <202106161145.HdBOaJkF-lkp@intel.com>
Message-ID: <20210616032219.GA70875@d00c18f5cb94>

drivers/acpi/x86/utils.c:153:25: warning: symbol 'storage_d3_cpu_ids' was not declared. Should it be static?

Fixes: 7c20093d34bc ("ACPI: Add quirks for AMD Renoir/Lucienne CPUs to force the D3 hint")
Reported-by: kernel test robot <lkp at intel.com>
Signed-off-by: kernel test robot <lkp at intel.com>
---
 utils.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/acpi/x86/utils.c b/drivers/acpi/x86/utils.c
index 5298bb4d81fea..f22f23933063b 100644
--- a/drivers/acpi/x86/utils.c
+++ b/drivers/acpi/x86/utils.c
@@ -150,7 +150,7 @@ bool acpi_device_always_present(struct acpi_device *adev)
  *
  * This allows quirking on Linux in a similar fashion.
  */
-const struct x86_cpu_id storage_d3_cpu_ids[] = {
+static const struct x86_cpu_id storage_d3_cpu_ids[] = {
 	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),	/* Renoir */
 	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL),	/* Lucienne */
 	{}


From lkp at intel.com  Tue Jun 15 20:22:16 2021
From: lkp at intel.com (kernel test robot)
Date: Wed, 16 Jun 2021 11:22:16 +0800
Subject: [linux-nvme:nvme-5.14 2/32] drivers/acpi/x86/utils.c:153:25: sparse:
 sparse: symbol 'storage_d3_cpu_ids' was not declared. Should it be
 static?
Message-ID: <202106161145.HdBOaJkF-lkp@intel.com>

tree:   git://git.infradead.org/nvme.git nvme-5.14
head:   9653e01d9c3c46930da3f007a39a2aaf230bd8ff
commit: 7c20093d34bcaf6bc1e2fbe44cb665eaa07f8964 [2/32] ACPI: Add quirks for AMD Renoir/Lucienne CPUs to force the D3 hint
config: i386-randconfig-s001-20210615 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce:
        # apt-get install sparse
        # sparse version: v0.6.3-341-g8af24329-dirty
        git remote add linux-nvme git://git.infradead.org/nvme.git
        git fetch --no-tags linux-nvme nvme-5.14
        git checkout 7c20093d34bcaf6bc1e2fbe44cb665eaa07f8964
        # save the attached .config to linux build tree
        make W=1 C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' W=1 ARCH=i386 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp at intel.com>


sparse warnings: (new ones prefixed by >>)
>> drivers/acpi/x86/utils.c:153:25: sparse: sparse: symbol 'storage_d3_cpu_ids' was not declared. Should it be static?

Please review and possibly fold the followup patch.

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 42304 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20210616/22b229e0/attachment-0001.gz>

From wesley.sheng at amd.com  Tue Jun 15 22:25:08 2021
From: wesley.sheng at amd.com (Wesley Sheng)
Date: Wed, 16 Jun 2021 13:25:08 +0800
Subject: [PATCH] nvme.h: add missing nvme_lba_range_type endianness annotations
Message-ID: <20210616052508.78970-1-wesley.sheng@amd.com>

Signed-off-by: Wesley Sheng <wesley.sheng at amd.com>
---
 include/linux/nvme.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index edcbd60b88b9..378140e16225 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -636,8 +636,8 @@ struct nvme_lba_range_type {
 	__u8			type;
 	__u8			attributes;
 	__u8			rsvd2[14];
-	__u64			slba;
-	__u64			nlb;
+	__le64			slba;
+	__le64			nlb;
 	__u8			guid[16];
 	__u8			rsvd48[16];
 };
-- 
2.25.1



From Vinay.Rao at dell.com  Wed Jun 16 03:38:26 2021
From: Vinay.Rao at dell.com (Rao, Vinay)
Date: Wed, 16 Jun 2021 10:38:26 +0000
Subject: [PATCH 1/1] nvme-fabrics: Generate uevent on namespace AENs
In-Reply-To: <20210503065235.GB32627@lst.de>
References: <BL0PR13MB429142FD8B9FCC57C2FE451D9C409@BL0PR13MB4291.namprd13.prod.outlook.com>
 <20210429063056.GA2482@lst.de>
 <SJ0PR19MB454499BF2899EF772B1EA11CF25F9@SJ0PR19MB4544.namprd19.prod.outlook.com>
 <BYAPR04MB4965BD2AE65469C96CBF1354865F9@BYAPR04MB4965.namprd04.prod.outlook.com>
 <BYAPR19MB29351987D7ABBA3F28791D978F5E9@BYAPR19MB2935.namprd19.prod.outlook.com>
 <20210430162104.GA907317@dhcp-10-100-145-180.wdc.com>
 <20210503065235.GB32627@lst.de>
Message-ID: <BYAPR19MB29356113F0A06BA4925EEF838F0F9@BYAPR19MB2935.namprd19.prod.outlook.com>



-----Original Message-----
From: Christoph Hellwig <hch at lst.de> 
Sent: Monday, May 3, 2021 12:23 PM
To: Keith Busch
Cc: Rao, Vinay; Chaitanya Kulkarni; Belanger, Martin; Hayes, Stuart; Tarikere, Madhu; Christoph Hellwig; Martin Belanger; linux-nvme at lists.infradead.org; axboe at fb.com; sagi at grimberg.me
Subject: Re: [PATCH 1/1] nvme-fabrics: Generate uevent on namespace AENs


[EXTERNAL EMAIL] 

On Fri, Apr 30, 2021 at 09:21:04AM -0700, Keith Busch wrote:
> On Fri, Apr 30, 2021 at 05:14:55AM +0000, Rao, Vinay wrote:
> > + Madhu
> > 
> > Hi Chaitanya,
> > 
> > Some of the AEN's like ANA state change and Name Space attribute change events are of interest outside the NVME driver. 
> > 
> > It would be good to consider these events to be propagated up to user space. I am not yet convinced on how propagating this above to user space is problematic.
> 
> The namespace attribute change event pairs with the Changed Namespace 
> List log, and reading that log will change the result for a subsequent 
> reader. User space racing with the kernel on a log access when there 
> are read side effects creates non-deterministic behavior, and that is 
> problematic.

The only way I could think of making this work is by:

forcing the RAE bit on for all log pages that the kernel cares about, and only delivering the uevent on controllers that actually do support the RAE bit (IIRC it got added in 1.3, but I'd have to check).

>> Just wanted to check if there is any plan to implement the code change of forcing the RAE bit for all log pages that the kernel cares about is taken up?  Can you please let us know what the plan is to address the problem. 


From josef at toxicpanda.com  Wed Jun 16 08:02:02 2021
From: josef at toxicpanda.com (Josef Bacik)
Date: Wed, 16 Jun 2021 11:02:02 -0400
Subject: [REMINDER] LSF/MM/BPF: 2021: Call for Proposals
Message-ID: <60ca126b.1c69fb81.90af4.93a7@mx.google.com>

Thisi is just a reminder that we're still going ahead with an in person
LSF/MM/BPF this year.  The Linux Foundation is working out the safety
guidelines and will update the landing page when that is nailed down.
In the meantime there is plenty of time to register your interest in
attending.  Thanks,

Josef

---- Original CFP ----

The annual Linux Storage, Filesystem, Memory Management, and BPF
(LSF/MM/BPF) Summit for 2021 will be held from December 6 to December 8
at The Margaritaville Resort Palm Springs in Palm Springs, California.
LSF/MM/BPF is an invitation-only technical workshop to map out
improvements to the Linux storage, filesystem, BPF, and memory
management subsystems that will make their way into the mainline kernel
within the coming years.

COVID is at the front of our minds as we attempt to put together the
best and safest conference we can arrange.  The logistics of how to hold
an in person event will change and evolve as we get closer to the actual
date, but rest assured we will do everything recommended by public
health officials.

LSF/MM/BPF 2021 will be a three day, stand-alone conference with four
subsystem-specific tracks, cross-track discussions, as well as BoF and
hacking sessions.

On behalf of the committee I am issuing a call for agenda proposals
that are suitable for cross-track discussion as well as technical
subjects for the breakout sessions.

If advance notice is required for visa applications then please point
that out in your proposal or request to attend, and submit the topic as
soon as possible.

This years instructions are similar to our 2020 attempt.  We're asking
that you please let us know you want to be invited by June 15th, 2021.
We realize that travel is an ever changing target, but it helps us get
an idea of possible attendance numbers.  Clearly things can and will, so
consider the request to attend deadline more about planning and less
about concrete plans.

1) Fill out the following Google form to request attendance and
suggest any topics

	https://forms.gle/Dms7xYPXLrriFkcXA

In previous years we have accidentally missed people's attendance
requests because they either didn't cc lsf-pc@ or we simply missed them
in the flurry of emails we get.?? Our community is large and our
volunteers are busy, filling this out will help us make sure we don't
miss anybody.

2) Proposals for agenda topics should still be sent to the following
lists to allow for discussion among your peers.?? This will help us
figure out which topics are important for the agenda.

?? ?? ?? ????lsf-pc at lists.linux-foundation.org

and CC the mailing lists that are relevant for the topic in question:

?? ?? ?? ?? FS:?? ?? ??linux-fsdevel at vger.kernel.org
?? ?? ?? ?? MM:?? ?? ??linux-mm at kvack.org
?? ?? ?? ?? Block:????linux-block at vger.kernel.org
?? ?? ?? ?? ATA:?? ????linux-ide at vger.kernel.org
?? ?? ?? ?? SCSI:?? ??linux-scsi at vger.kernel.org
?? ?? ?? ?? NVMe:?? ??linux-nvme at lists.infradead.org
?? ?? ?? ?? BPF:?? ????bpf at vger.kernel.org

Please tag your proposal with [LSF/MM/BPF TOPIC] to make it easier to
track. In addition, please make sure to start a new thread for each
topic rather than following up to an existing one. Agenda topics and
attendees will be selected by the program committee, but the final
agenda will be formed by consensus of the attendees on the day.

We will try to cap attendance at around 25-30 per track to facilitate
discussions although the final numbers will depend on the room sizes
at the venue.

For discussion leaders, slides and visualizations are encouraged to
outline the subject matter and focus the discussions. Please refrain
from lengthy presentations and talks; the sessions are supposed to be
interactive, inclusive discussions.

There will be no recording or audio bridge. However, we expect that
written minutes will be published as we did in previous years:

2019:??https://lwn.net/Articles/lsfmm2019/

2018:??https://lwn.net/Articles/lsfmm2018/

2017:??https://lwn.net/Articles/lsfmm2017/

2016:??https://lwn.net/Articles/lsfmm2016/

2015:??https://lwn.net/Articles/lsfmm2015/

2014:??http://lwn.net/Articles/LSFMM2014/

3) If you have feedback on last year's meeting that we can use to
improve this year's, please also send that to:

?? ?? ?? ????lsf-pc at lists.linux-foundation.org

Thank you on behalf of the program committee:

?? ?? ?? ?? Josef Bacik (Filesystems)
?? ?? ?? ?? Amir Goldstein (Filesystems)
?? ?? ?? ?? Martin K. Petersen (Storage)
?? ?? ?? ?? Omar Sandoval (Storage)
?? ?? ?? ?? Michal Hocko (MM)
?? ?? ?? ?? Dan Williams (MM)
?? ?? ?? ?? Alexei Starovoitov (BPF)
?? ?? ?? ?? Daniel Borkmann (BPF)


From mario.limonciello at amd.com  Wed Jun 16 08:59:13 2021
From: mario.limonciello at amd.com (Limonciello, Mario)
Date: Wed, 16 Jun 2021 10:59:13 -0500
Subject: [RFC PATCH linux-nvme] ACPI: storage_d3_cpu_ids[] can be static
In-Reply-To: <20210616032219.GA70875@d00c18f5cb94>
References: <202106161145.HdBOaJkF-lkp@intel.com>
 <20210616032219.GA70875@d00c18f5cb94>
Message-ID: <854b9bdd-7bc2-1210-4523-9def184b3e04@amd.com>

On 6/15/2021 22:22, kernel test robot wrote:
> drivers/acpi/x86/utils.c:153:25: warning: symbol 'storage_d3_cpu_ids' was not declared. Should it be static?
> 
> Fixes: 7c20093d34bc ("ACPI: Add quirks for AMD Renoir/Lucienne CPUs to force the D3 hint")
> Reported-by: kernel test robot <lkp at intel.com>
> Signed-off-by: kernel test robot <lkp at intel.com>
> ---
>   utils.c |    2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/acpi/x86/utils.c b/drivers/acpi/x86/utils.c
> index 5298bb4d81fea..f22f23933063b 100644
> --- a/drivers/acpi/x86/utils.c
> +++ b/drivers/acpi/x86/utils.c
> @@ -150,7 +150,7 @@ bool acpi_device_always_present(struct acpi_device *adev)
>    *
>    * This allows quirking on Linux in a similar fashion.
>    */
> -const struct x86_cpu_id storage_d3_cpu_ids[] = {
> +static const struct x86_cpu_id storage_d3_cpu_ids[] = {
>   	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL),	/* Renoir */
>   	X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL),	/* Lucienne */
>   	{}
> 

Acked-By: Mario Limonciello <mario.limonciello at amd.com>


From kbusch at kernel.org  Wed Jun 16 09:20:56 2021
From: kbusch at kernel.org (Keith Busch)
Date: Wed, 16 Jun 2021 09:20:56 -0700
Subject: [PATCH] nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <20210615215741.26844-2-cachen@purestorage.com>
References: <20210615215741.26844-1-cachen@purestorage.com>
 <20210615215741.26844-2-cachen@purestorage.com>
Message-ID: <20210616162056.GA906592@dhcp-10-100-145-180.wdc.com>

On Tue, Jun 15, 2021 at 02:57:41PM -0700, Casey Chen wrote:
> @@ -1590,8 +1590,9 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
>  		goto release_cq;
>  
>  	nvmeq->cq_vector = vector;
> -	nvme_init_queue(nvmeq, qid);
>  
> +	mutex_lock(&dev->shutdown_lock);

I really think you want to use mutex_trylock() here and return early if
locking fails since that must mean the device is being disabled.
Waiting to acquire the lock in this path doesn't do any good on a
disabled device.


From sagi at grimberg.me  Wed Jun 16 09:28:53 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 16 Jun 2021 09:28:53 -0700
Subject: [PATCH v2 0/4] nvme: protect against possible request reference
 after completion
In-Reply-To: <20210519174340.1144806-1-sagi@grimberg.me>
References: <20210519174340.1144806-1-sagi@grimberg.me>
Message-ID: <777a99bc-b7e3-ce39-cd98-ab1196aebb2e@grimberg.me>


> Nothing in nvme protects against referencing a request after it was completed.
> For example, in case a buggy controller sends a completion twice for the same
> request, the host can access and modify a request that was already completed.
> 
> At best, this will cause a panic, but on the worst case, this can cause a silent
> data corruption if the request was already reused and executed by the time
> we reference it.
> 
> The nvme command_id is an opaque that we simply placed the request tag thus far.
> To protect against a access after completion, we introduce a generation counter
> to the upper 4-bits of the command_id that will increment every invocation and
> be validated upon the reception of a completion. This will limit the maximum
> queue depth to be effectively 4095, but we hardly ever use such long queues
> (in fabrics the maximum is already 1024).

Keith,

Did you get a chance to look at the performance impact of this patch 
set? I think we are all in agreement that this is a useful safeguard if
there is no major performance impact.


From lkp at intel.com  Wed Jun 16 09:32:31 2021
From: lkp at intel.com (kernel test robot)
Date: Thu, 17 Jun 2021 00:32:31 +0800
Subject: [linux-nvme:nvme-5.14] BUILD SUCCESS
 e645b5a5baf2e28329941eb5691ce01e251285b0
Message-ID: <60ca279f.DFxTC69EsCPPvjjC%lkp@intel.com>

tree/branch: git://git.infradead.org/nvme.git nvme-5.14
branch HEAD: e645b5a5baf2e28329941eb5691ce01e251285b0  nvmet: add ZBD over ZNS backend support

elapsed time: 748m

configs tested: 124
configs skipped: 2

The following configs have been built successfully.
More configs may be tested in the coming days.

gcc tested configs:
arm                                 defconfig
arm64                            allyesconfig
arm64                               defconfig
arm                              allyesconfig
arm                              allmodconfig
powerpc                      chrp32_defconfig
powerpc                  mpc866_ads_defconfig
mips                     cu1000-neo_defconfig
ia64                        generic_defconfig
arm                       versatile_defconfig
arm                         lpc32xx_defconfig
powerpc                    ge_imp3a_defconfig
powerpc                   bluestone_defconfig
sh                          r7785rp_defconfig
sh                        sh7785lcr_defconfig
arm                            lart_defconfig
sh                          sdk7786_defconfig
xtensa                    smp_lx200_defconfig
arm                          collie_defconfig
powerpc                 mpc85xx_cds_defconfig
mips                           gcw0_defconfig
mips                           ip32_defconfig
ia64                         bigsur_defconfig
arm                  colibri_pxa270_defconfig
sh                           sh2007_defconfig
arm                          ixp4xx_defconfig
m68k                       m5275evb_defconfig
arm                        keystone_defconfig
arm                         palmz72_defconfig
powerpc                mpc7448_hpc2_defconfig
m68k                          atari_defconfig
powerpc                        fsp2_defconfig
powerpc                     sbc8548_defconfig
s390                             alldefconfig
riscv             nommu_k210_sdcard_defconfig
arm                            zeus_defconfig
m68k                          sun3x_defconfig
arm                          exynos_defconfig
sh                           se7343_defconfig
arm                              alldefconfig
s390                          debug_defconfig
powerpc                 mpc832x_mds_defconfig
arm                        mvebu_v5_defconfig
sparc                       sparc64_defconfig
riscv                          rv32_defconfig
x86_64                            allnoconfig
ia64                             allmodconfig
ia64                                defconfig
ia64                             allyesconfig
m68k                             allmodconfig
m68k                                defconfig
m68k                             allyesconfig
nios2                               defconfig
arc                              allyesconfig
nds32                             allnoconfig
nds32                               defconfig
nios2                            allyesconfig
csky                                defconfig
alpha                               defconfig
alpha                            allyesconfig
xtensa                           allyesconfig
h8300                            allyesconfig
arc                                 defconfig
sh                               allmodconfig
parisc                              defconfig
s390                             allyesconfig
s390                             allmodconfig
parisc                           allyesconfig
s390                                defconfig
i386                             allyesconfig
sparc                            allyesconfig
sparc                               defconfig
i386                                defconfig
mips                             allyesconfig
mips                             allmodconfig
powerpc                          allyesconfig
powerpc                          allmodconfig
powerpc                           allnoconfig
x86_64               randconfig-a001-20210615
x86_64               randconfig-a004-20210615
x86_64               randconfig-a002-20210615
x86_64               randconfig-a003-20210615
x86_64               randconfig-a006-20210615
x86_64               randconfig-a005-20210615
i386                 randconfig-a002-20210615
i386                 randconfig-a006-20210615
i386                 randconfig-a004-20210615
i386                 randconfig-a001-20210615
i386                 randconfig-a005-20210615
i386                 randconfig-a003-20210615
i386                 randconfig-a015-20210615
i386                 randconfig-a013-20210615
i386                 randconfig-a016-20210615
i386                 randconfig-a012-20210615
i386                 randconfig-a014-20210615
i386                 randconfig-a011-20210615
riscv                    nommu_k210_defconfig
riscv                            allyesconfig
riscv                    nommu_virt_defconfig
riscv                             allnoconfig
riscv                               defconfig
riscv                            allmodconfig
x86_64                    rhel-8.3-kselftests
um                           x86_64_defconfig
um                             i386_defconfig
um                            kunit_defconfig
x86_64                           allyesconfig
x86_64                              defconfig
x86_64                               rhel-8.3
x86_64                      rhel-8.3-kbuiltin
x86_64                                  kexec

clang tested configs:
x86_64               randconfig-b001-20210615
x86_64               randconfig-a004-20210616
x86_64               randconfig-a001-20210616
x86_64               randconfig-a002-20210616
x86_64               randconfig-a003-20210616
x86_64               randconfig-a006-20210616
x86_64               randconfig-a005-20210616
x86_64               randconfig-a015-20210615
x86_64               randconfig-a011-20210615
x86_64               randconfig-a012-20210615
x86_64               randconfig-a014-20210615
x86_64               randconfig-a016-20210615
x86_64               randconfig-a013-20210615

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org


From kbusch at kernel.org  Wed Jun 16 10:04:11 2021
From: kbusch at kernel.org (Keith Busch)
Date: Wed, 16 Jun 2021 10:04:11 -0700
Subject: [PATCH v2 0/4] nvme: protect against possible request reference
 after completion
In-Reply-To: <777a99bc-b7e3-ce39-cd98-ab1196aebb2e@grimberg.me>
References: <20210519174340.1144806-1-sagi@grimberg.me>
 <777a99bc-b7e3-ce39-cd98-ab1196aebb2e@grimberg.me>
Message-ID: <20210616170411.GA921268@dhcp-10-100-145-180.wdc.com>

On Wed, Jun 16, 2021 at 09:28:53AM -0700, Sagi Grimberg wrote:
> 
> > Nothing in nvme protects against referencing a request after it was completed.
> > For example, in case a buggy controller sends a completion twice for the same
> > request, the host can access and modify a request that was already completed.
> > 
> > At best, this will cause a panic, but on the worst case, this can cause a silent
> > data corruption if the request was already reused and executed by the time
> > we reference it.
> > 
> > The nvme command_id is an opaque that we simply placed the request tag thus far.
> > To protect against a access after completion, we introduce a generation counter
> > to the upper 4-bits of the command_id that will increment every invocation and
> > be validated upon the reception of a completion. This will limit the maximum
> > queue depth to be effectively 4095, but we hardly ever use such long queues
> > (in fabrics the maximum is already 1024).
> 
> Keith,
> 
> Did you get a chance to look at the performance impact of this patch set? I
> think we are all in agreement that this is a useful safeguard if
> there is no major performance impact.

Yes, I was able to set up my more powerful machine for perf tests
earlier this week. I ran fio hipri tests against this patch yesterday,
and did not observe a measurable difference with either io_uring or
pvsync2 engines, so no objection from me here.

Acked-by: Keith Busch <kbusch at kernel.org>



From sagi at grimberg.me  Wed Jun 16 14:05:16 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 16 Jun 2021 14:05:16 -0700
Subject: [PATCH v2 0/4] nvme: protect against possible request reference
 after completion
In-Reply-To: <20210616170411.GA921268@dhcp-10-100-145-180.wdc.com>
References: <20210519174340.1144806-1-sagi@grimberg.me>
 <777a99bc-b7e3-ce39-cd98-ab1196aebb2e@grimberg.me>
 <20210616170411.GA921268@dhcp-10-100-145-180.wdc.com>
Message-ID: <5c0766d9-9507-b0c7-4917-4b936fd7a197@grimberg.me>


>>> Nothing in nvme protects against referencing a request after it was completed.
>>> For example, in case a buggy controller sends a completion twice for the same
>>> request, the host can access and modify a request that was already completed.
>>>
>>> At best, this will cause a panic, but on the worst case, this can cause a silent
>>> data corruption if the request was already reused and executed by the time
>>> we reference it.
>>>
>>> The nvme command_id is an opaque that we simply placed the request tag thus far.
>>> To protect against a access after completion, we introduce a generation counter
>>> to the upper 4-bits of the command_id that will increment every invocation and
>>> be validated upon the reception of a completion. This will limit the maximum
>>> queue depth to be effectively 4095, but we hardly ever use such long queues
>>> (in fabrics the maximum is already 1024).
>>
>> Keith,
>>
>> Did you get a chance to look at the performance impact of this patch set? I
>> think we are all in agreement that this is a useful safeguard if
>> there is no major performance impact.
> 
> Yes, I was able to set up my more powerful machine for perf tests
> earlier this week. I ran fio hipri tests against this patch yesterday,
> and did not observe a measurable difference with either io_uring or
> pvsync2 engines, so no objection from me here.
> 
> Acked-by: Keith Busch <kbusch at kernel.org>

Cool, Christoph, where do you stand on this? Can we move forward with
this?


From syzbot+6a8a0d93c91e8fbf2e80 at syzkaller.appspotmail.com  Wed Jun 16 14:18:17 2021
From: syzbot+6a8a0d93c91e8fbf2e80 at syzkaller.appspotmail.com (syzbot)
Date: Wed, 16 Jun 2021 14:18:17 -0700
Subject: [syzbot] possible deadlock in loop_probe
Message-ID: <00000000000015a81105c4e8a1c3@google.com>

Hello,

syzbot found the following issue on:

HEAD commit:    8ecfa36c Merge tag 'riscv-for-linus-5.13-rc6' of git://git..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=17121304300000
kernel config:  https://syzkaller.appspot.com/x/.config?x=547a5e42ca601229
dashboard link: https://syzkaller.appspot.com/bug?extid=6a8a0d93c91e8fbf2e80
compiler:       Debian clang version 11.0.1-2
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=17065d1fd00000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=12de53afd00000

The issue was bisected to:

commit 5befc7c26e5a98cd49789fb1beb52c62bd472dba
Author: Keith Busch <kbusch at kernel.org>
Date:   Wed Mar 24 23:18:05 2021 +0000

    nvme: implement non-mdts command limits

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=15e62857d00000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=17e62857d00000
console output: https://syzkaller.appspot.com/x/log.txt?x=13e62857d00000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+6a8a0d93c91e8fbf2e80 at syzkaller.appspotmail.com
Fixes: 5befc7c26e5a ("nvme: implement non-mdts command limits")

======================================================
WARNING: possible circular locking dependency detected
5.13.0-rc5-syzkaller #0 Not tainted
------------------------------------------------------
systemd-udevd/8445 is trying to acquire lock:
ffffffff8d671688 (loop_ctl_mutex){+.+.}-{3:3}, at: loop_probe+0xe1/0x220 drivers/block/loop.c:2234

but task is already holding lock:
ffffffff8d4db008 (major_names_lock){+.+.}-{3:3}, at: blk_request_module+0x2f/0x1c0 block/genhd.c:680

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (major_names_lock){+.+.}-{3:3}:
       lock_acquire+0x17f/0x720 kernel/locking/lockdep.c:5512
       __mutex_lock_common+0x1bf/0x3100 kernel/locking/mutex.c:959
       __mutex_lock kernel/locking/mutex.c:1104 [inline]
       mutex_lock_nested+0x1a/0x20 kernel/locking/mutex.c:1119
       __register_blkdev+0x2c/0x360 block/genhd.c:223
       register_mtd_blktrans+0x94/0x3d0 drivers/mtd/mtd_blkdevs.c:534
       do_one_initcall+0x1a7/0x400 init/main.c:1249
       do_initcall_level+0x14a/0x1f5 init/main.c:1322
       do_initcalls+0x4b/0x8c init/main.c:1338
       kernel_init_freeable+0x3fd/0x58a init/main.c:1560
       kernel_init+0xd/0x290 init/main.c:1447
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

-> #2 (mtd_table_mutex){+.+.}-{3:3}:
       lock_acquire+0x17f/0x720 kernel/locking/lockdep.c:5512
       __mutex_lock_common+0x1bf/0x3100 kernel/locking/mutex.c:959
       __mutex_lock kernel/locking/mutex.c:1104 [inline]
       mutex_lock_nested+0x1a/0x20 kernel/locking/mutex.c:1119
       blktrans_open+0x61/0x430 drivers/mtd/mtd_blkdevs.c:212
       __blkdev_get+0x293/0xa50 fs/block_dev.c:1311
       blkdev_get_by_dev+0xf9/0x640 fs/block_dev.c:1451
       blkdev_open+0x132/0x2c0 fs/block_dev.c:1546
       do_dentry_open+0x7cb/0x1010 fs/open.c:826
       do_open fs/namei.c:3361 [inline]
       path_openat+0x28e6/0x39b0 fs/namei.c:3494
       do_filp_open+0x221/0x460 fs/namei.c:3521
       do_sys_openat2+0x124/0x460 fs/open.c:1187
       do_sys_open fs/open.c:1203 [inline]
       __do_sys_open fs/open.c:1211 [inline]
       __se_sys_open fs/open.c:1207 [inline]
       __x64_sys_open+0x221/0x270 fs/open.c:1207
       do_syscall_64+0x3f/0xb0 arch/x86/entry/common.c:47
       entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #1 (&bdev->bd_mutex){+.+.}-{3:3}:
       lock_acquire+0x17f/0x720 kernel/locking/lockdep.c:5512
       __mutex_lock_common+0x1bf/0x3100 kernel/locking/mutex.c:959
       __mutex_lock kernel/locking/mutex.c:1104 [inline]
       mutex_lock_nested+0x1a/0x20 kernel/locking/mutex.c:1119
       del_gendisk+0x2ab/0x860 block/genhd.c:610
       loop_remove drivers/block/loop.c:2180 [inline]
       loop_control_ioctl+0x62c/0x770 drivers/block/loop.c:2280
       vfs_ioctl fs/ioctl.c:51 [inline]
       __do_sys_ioctl fs/ioctl.c:1069 [inline]
       __se_sys_ioctl+0xfb/0x170 fs/ioctl.c:1055
       do_syscall_64+0x3f/0xb0 arch/x86/entry/common.c:47
       entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #0 (loop_ctl_mutex){+.+.}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:2938 [inline]
       check_prevs_add+0x4f9/0x5b60 kernel/locking/lockdep.c:3061
       validate_chain kernel/locking/lockdep.c:3676 [inline]
       __lock_acquire+0x4307/0x6040 kernel/locking/lockdep.c:4902
       lock_acquire+0x17f/0x720 kernel/locking/lockdep.c:5512
       __mutex_lock_common+0x1bf/0x3100 kernel/locking/mutex.c:959
       __mutex_lock kernel/locking/mutex.c:1104 [inline]
       mutex_lock_nested+0x1a/0x20 kernel/locking/mutex.c:1119
       loop_probe+0xe1/0x220 drivers/block/loop.c:2234
       blk_request_module+0x19d/0x1c0 block/genhd.c:683
       blkdev_get_no_open+0x44/0x1f0 fs/block_dev.c:1374
       blkdev_get_by_dev+0x82/0x640 fs/block_dev.c:1437
       blkdev_open+0x132/0x2c0 fs/block_dev.c:1546
       do_dentry_open+0x7cb/0x1010 fs/open.c:826
       do_open fs/namei.c:3361 [inline]
       path_openat+0x28e6/0x39b0 fs/namei.c:3494
       do_filp_open+0x221/0x460 fs/namei.c:3521
       do_sys_openat2+0x124/0x460 fs/open.c:1187
       do_sys_open fs/open.c:1203 [inline]
       __do_sys_open fs/open.c:1211 [inline]
       __se_sys_open fs/open.c:1207 [inline]
       __x64_sys_open+0x221/0x270 fs/open.c:1207
       do_syscall_64+0x3f/0xb0 arch/x86/entry/common.c:47
       entry_SYSCALL_64_after_hwframe+0x44/0xae

other info that might help us debug this:

Chain exists of:
  loop_ctl_mutex --> mtd_table_mutex --> major_names_lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(major_names_lock);
                               lock(mtd_table_mutex);
                               lock(major_names_lock);
  lock(loop_ctl_mutex);

 *** DEADLOCK ***

1 lock held by systemd-udevd/8445:
 #0: ffffffff8d4db008 (major_names_lock){+.+.}-{3:3}, at: blk_request_module+0x2f/0x1c0 block/genhd.c:680

stack backtrace:
CPU: 1 PID: 8445 Comm: systemd-udevd Not tainted 5.13.0-rc5-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x202/0x31e lib/dump_stack.c:120
 print_circular_bug+0xb17/0xdc0 kernel/locking/lockdep.c:2007
 check_noncircular+0x2cc/0x390 kernel/locking/lockdep.c:2129
 check_prev_add kernel/locking/lockdep.c:2938 [inline]
 check_prevs_add+0x4f9/0x5b60 kernel/locking/lockdep.c:3061
 validate_chain kernel/locking/lockdep.c:3676 [inline]
 __lock_acquire+0x4307/0x6040 kernel/locking/lockdep.c:4902
 lock_acquire+0x17f/0x720 kernel/locking/lockdep.c:5512
 __mutex_lock_common+0x1bf/0x3100 kernel/locking/mutex.c:959
 __mutex_lock kernel/locking/mutex.c:1104 [inline]
 mutex_lock_nested+0x1a/0x20 kernel/locking/mutex.c:1119
 loop_probe+0xe1/0x220 drivers/block/loop.c:2234
 blk_request_module+0x19d/0x1c0 block/genhd.c:683
 blkdev_get_no_open+0x44/0x1f0 fs/block_dev.c:1374
 blkdev_get_by_dev+0x82/0x640 fs/block_dev.c:1437
 blkdev_open+0x132/0x2c0 fs/block_dev.c:1546
 do_dentry_open+0x7cb/0x1010 fs/open.c:826
 do_open fs/namei.c:3361 [inline]
 path_openat+0x28e6/0x39b0 fs/namei.c:3494
 do_filp_open+0x221/0x460 fs/namei.c:3521
 do_sys_openat2+0x124/0x460 fs/open.c:1187
 do_sys_open fs/open.c:1203 [inline]
 __do_sys_open fs/open.c:1211 [inline]
 __se_sys_open fs/open.c:1207 [inline]
 __x64_sys_open+0x221/0x270 fs/open.c:1207
 do_syscall_64+0x3f/0xb0 arch/x86/entry/common.c:47
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f666c06a840
Code: 73 01 c3 48 8b 0d 68 77 20 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d 89 bb 20 00 00 75 10 b8 02 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 1e f6 ff ff 48 89 04 24
RSP: 002b:00007ffc880be598 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
RAX: ffffffffffffffda RBX: 000055dd2c8278f0 RCX: 00007f666c06a840
RDX: 000055dd2c52efe3 RSI: 00000000000a0800 RDI: 000055dd2c827830
RBP: 00007ffc880be710 R08: 000055dd2c52e670 R09: 0000000000000010
R10: 000055dd2c52ed0c R11: 0000000000000246 R12: 00007ffc880be660
R13: 000055dd2c829010 R14: 0000000000000003 R15: 000000000000000e


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller at googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches


From sagi at grimberg.me  Wed Jun 16 14:19:32 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 16 Jun 2021 14:19:32 -0700
Subject: [PATCH v3 0/4] nvme: protect against possible request reference after
 completion
Message-ID: <20210616211936.451128-1-sagi@grimberg.me>

Nothing in nvme protects against referencing a request after it was completed.
For example, in case a buggy controller sends a completion twice for the same
request, the host can access and modify a request that was already completed.

At best, this will cause a panic, but on the worst case, this can cause a silent
data corruption if the request was already reused and executed by the time
we reference it.

The nvme command_id is an opaque that we simply placed the request tag thus far.
To protect against a access after completion, we introduce a generation counter
to the upper 4-bits of the command_id that will increment every invocation and
be validated upon the reception of a completion. This will limit the maximum
queue depth to be effectively 4095, but we hardly ever use such long queues
(in fabrics the maximum is already 1024).

Changes from v2:
- cc linux-nfs,linux-kernel for patch 1/4
- fix expected genctr print in patch 4/4
- match param_set_uint_minmax indentation
- collected review tags

Changes from v1:
- lift param_set_uint_minmax and reuse it
- simplify initialization in patch 3/4

Sagi Grimberg (4):
  params: lift param_set_uint_minmax to common code
  nvme-pci: limit maximum queue depth to 4095
  nvme-tcp: don't check blk_mq_tag_to_rq when receiving pdu data
  nvme: code command_id with a genctr for use-after-free validation

 drivers/nvme/host/core.c    |  3 ++-
 drivers/nvme/host/nvme.h    | 47 ++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/pci.c     | 17 ++++++--------
 drivers/nvme/host/rdma.c    |  4 ++--
 drivers/nvme/host/tcp.c     | 38 ++++++++++++------------------
 drivers/nvme/target/loop.c  |  4 ++--
 include/linux/moduleparam.h |  3 +++
 kernel/params.c             | 19 +++++++++++++++
 net/sunrpc/xprtsock.c       | 18 --------------
 9 files changed, 96 insertions(+), 57 deletions(-)

-- 
2.27.0



From sagi at grimberg.me  Wed Jun 16 14:19:33 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 16 Jun 2021 14:19:33 -0700
Subject: [PATCH v3 1/4] params: lift param_set_uint_minmax to common code
In-Reply-To: <20210616211936.451128-1-sagi@grimberg.me>
References: <20210616211936.451128-1-sagi@grimberg.me>
Message-ID: <20210616211936.451128-2-sagi@grimberg.me>

It is a useful helper hence move it to common code so others can enjoy
it.

Suggested-by: Christoph Hellwig <hch at lst.de>
Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Cc: linux-nfs at vger.kernel.org
Cc: linux-kernel at vger.kernel.org
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 include/linux/moduleparam.h |  3 +++
 kernel/params.c             | 19 +++++++++++++++++++
 net/sunrpc/xprtsock.c       | 18 ------------------
 3 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/include/linux/moduleparam.h b/include/linux/moduleparam.h
index eed280fae433..b36ece513616 100644
--- a/include/linux/moduleparam.h
+++ b/include/linux/moduleparam.h
@@ -432,6 +432,9 @@ extern const struct kernel_param_ops param_ops_uint;
 extern int param_set_uint(const char *val, const struct kernel_param *kp);
 extern int param_get_uint(char *buffer, const struct kernel_param *kp);
 #define param_check_uint(name, p) __param_check(name, p, unsigned int)
+int param_set_uint_minmax(const char *val,
+			  const struct kernel_param *kp,
+			  unsigned int min, unsigned int max);
 
 extern const struct kernel_param_ops param_ops_long;
 extern int param_set_long(const char *val, const struct kernel_param *kp);
diff --git a/kernel/params.c b/kernel/params.c
index 2daa2780a92c..7d3c61e64140 100644
--- a/kernel/params.c
+++ b/kernel/params.c
@@ -243,6 +243,25 @@ STANDARD_PARAM_DEF(ulong,	unsigned long,		"%lu",		kstrtoul);
 STANDARD_PARAM_DEF(ullong,	unsigned long long,	"%llu",		kstrtoull);
 STANDARD_PARAM_DEF(hexint,	unsigned int,		"%#08x", 	kstrtouint);
 
+int param_set_uint_minmax(const char *val,
+			  const struct kernel_param *kp,
+			  unsigned int min, unsigned int max)
+{
+	unsigned int num;
+	int ret;
+
+	if (!val)
+		return -EINVAL;
+	ret = kstrtouint(val, 0, &num);
+	if (ret)
+		return ret;
+	if (num < min || num > max)
+		return -EINVAL;
+	*((unsigned int *)kp->arg) = num;
+	return 0;
+}
+EXPORT_SYMBOL(param_set_uint_minmax);
+
 int param_set_charp(const char *val, const struct kernel_param *kp)
 {
 	if (strlen(val) > 1024) {
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 47aa47a2b07c..0cfbf618e8c2 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -3130,24 +3130,6 @@ void cleanup_socket_xprt(void)
 	xprt_unregister_transport(&xs_bc_tcp_transport);
 }
 
-static int param_set_uint_minmax(const char *val,
-		const struct kernel_param *kp,
-		unsigned int min, unsigned int max)
-{
-	unsigned int num;
-	int ret;
-
-	if (!val)
-		return -EINVAL;
-	ret = kstrtouint(val, 0, &num);
-	if (ret)
-		return ret;
-	if (num < min || num > max)
-		return -EINVAL;
-	*((unsigned int *)kp->arg) = num;
-	return 0;
-}
-
 static int param_set_portnr(const char *val, const struct kernel_param *kp)
 {
 	return param_set_uint_minmax(val, kp,
-- 
2.27.0



From sagi at grimberg.me  Wed Jun 16 14:19:35 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 16 Jun 2021 14:19:35 -0700
Subject: [PATCH v3 3/4] nvme-tcp: don't check blk_mq_tag_to_rq when receiving
 pdu data
In-Reply-To: <20210616211936.451128-1-sagi@grimberg.me>
References: <20210616211936.451128-1-sagi@grimberg.me>
Message-ID: <20210616211936.451128-4-sagi@grimberg.me>

We already validate it when receiving the c2hdata pdu header
and this is not changing so this is a redundant check.

Reviewed-by: Hannes Reinecke <hare at suse.de>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/tcp.c | 14 +++-----------
 1 file changed, 3 insertions(+), 11 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index c7bd37103cf4..3ad65f42fc1e 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -703,17 +703,9 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 			      unsigned int *offset, size_t *len)
 {
 	struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu;
-	struct nvme_tcp_request *req;
-	struct request *rq;
-
-	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
-	if (!rq) {
-		dev_err(queue->ctrl->ctrl.device,
-			"queue %d tag %#x not found\n",
-			nvme_tcp_queue_id(queue), pdu->command_id);
-		return -ENOENT;
-	}
-	req = blk_mq_rq_to_pdu(rq);
+	struct request *rq =
+		blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
+	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
 
 	while (true) {
 		int recv_len, ret;
-- 
2.27.0



From sagi at grimberg.me  Wed Jun 16 14:19:34 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 16 Jun 2021 14:19:34 -0700
Subject: [PATCH v3 2/4] nvme-pci: limit maximum queue depth to 4095
In-Reply-To: <20210616211936.451128-1-sagi@grimberg.me>
References: <20210616211936.451128-1-sagi@grimberg.me>
Message-ID: <20210616211936.451128-3-sagi@grimberg.me>

We are going to use the upper 4-bits of the command_id for a generation
counter, so enforce the new queue depth upper limit. As we enforce
both min and max queue depth, use param_set_uint_minmax istead of
open coding it.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/pci.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 9cdf2099027a..d2ee9e46f5d3 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -60,6 +60,8 @@ MODULE_PARM_DESC(sgl_threshold,
 		"Use SGLs when average request segment size is larger or equal to "
 		"this size. Use 0 to disable SGLs.");
 
+#define NVME_PCI_MIN_QUEUE_SIZE 2
+#define NVME_PCI_MAX_QUEUE_SIZE 4095
 static int io_queue_depth_set(const char *val, const struct kernel_param *kp);
 static const struct kernel_param_ops io_queue_depth_ops = {
 	.set = io_queue_depth_set,
@@ -68,7 +70,7 @@ static const struct kernel_param_ops io_queue_depth_ops = {
 
 static unsigned int io_queue_depth = 1024;
 module_param_cb(io_queue_depth, &io_queue_depth_ops, &io_queue_depth, 0644);
-MODULE_PARM_DESC(io_queue_depth, "set io queue depth, should >= 2");
+MODULE_PARM_DESC(io_queue_depth, "set io queue depth, should >= 2 and < 4096");
 
 static int io_queue_count_set(const char *val, const struct kernel_param *kp)
 {
@@ -157,14 +159,9 @@ struct nvme_dev {
 
 static int io_queue_depth_set(const char *val, const struct kernel_param *kp)
 {
-	int ret;
-	u32 n;
-
-	ret = kstrtou32(val, 10, &n);
-	if (ret != 0 || n < 2)
-		return -EINVAL;
-
-	return param_set_uint(val, kp);
+	return param_set_uint_minmax(val, kp,
+			NVME_PCI_MIN_QUEUE_SIZE,
+			NVME_PCI_MAX_QUEUE_SIZE);
 }
 
 static inline unsigned int sq_idx(unsigned int qid, u32 stride)
-- 
2.27.0



From sagi at grimberg.me  Wed Jun 16 14:19:36 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 16 Jun 2021 14:19:36 -0700
Subject: [PATCH v3 4/4] nvme: code command_id with a genctr for use-after-free
 validation
In-Reply-To: <20210616211936.451128-1-sagi@grimberg.me>
References: <20210616211936.451128-1-sagi@grimberg.me>
Message-ID: <20210616211936.451128-5-sagi@grimberg.me>

We cannot detect a (perhaps buggy) controller that is sending us
a completion for a request that was already completed (for example
sending a completion twice), this phenomenon was seen in the wild
a few times.

So to protect against this, we use the upper 4 msbits of the nvme sqe
command_id to use as a 4-bit generation counter and verify it matches
the existing request generation that is incrementing on every execution.

The 16-bit command_id structure now is constructed by:
| xxxx | xxxxxxxxxxxx |
  gen    request tag

This means that we are giving up some possible queue depth as 12 bits
allow for a maximum queue depth of 4095 instead of 65536, however we
never create such long queues anyways so no real harm done.

Suggested-by: Keith Busch <kbusch at kernel.org>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Acked-by: Keith Busch <kbusch at kernel.org>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c   |  3 ++-
 drivers/nvme/host/nvme.h   | 47 +++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/pci.c    |  2 +-
 drivers/nvme/host/rdma.c   |  4 ++--
 drivers/nvme/host/tcp.c    | 26 ++++++++++-----------
 drivers/nvme/target/loop.c |  4 ++--
 6 files changed, 66 insertions(+), 20 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 25ad9027f928..a75876dfa38c 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1026,7 +1026,8 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req)
 		return BLK_STS_IOERR;
 	}
 
-	cmd->common.command_id = req->tag;
+	nvme_req(req)->genctr++;
+	cmd->common.command_id = nvme_cid(req);
 	trace_nvme_setup_cmd(req, cmd);
 	return ret;
 }
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1aab74128d40..ad518b1c0fac 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -158,6 +158,7 @@ enum nvme_quirks {
 struct nvme_request {
 	struct nvme_command	*cmd;
 	union nvme_result	result;
+	u8			genctr;
 	u8			retries;
 	u8			flags;
 	u16			status;
@@ -497,6 +498,49 @@ struct nvme_ctrl_ops {
 	int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size);
 };
 
+/*
+ * nvme command_id is constructed as such:
+ * | xxxx | xxxxxxxxxxxx |
+ *   gen    request tag
+ */
+#define nvme_genctr_mask(gen)			(gen & 0xf)
+#define nvme_cid_install_genctr(gen)		(nvme_genctr_mask(gen) << 12)
+#define nvme_genctr_from_cid(cid)		((cid & 0xf000) >> 12)
+#define nvme_tag_from_cid(cid)			(cid & 0xfff)
+
+static inline u16 nvme_cid(struct request *rq)
+{
+	return nvme_cid_install_genctr(nvme_req(rq)->genctr) | rq->tag;
+}
+
+static inline struct request *nvme_find_rq(struct blk_mq_tags *tags,
+		u16 command_id)
+{
+	u8 genctr = nvme_genctr_from_cid(command_id);
+	u16 tag = nvme_tag_from_cid(command_id);
+	struct request *rq;
+
+	rq = blk_mq_tag_to_rq(tags, tag);
+	if (unlikely(!rq)) {
+		pr_err("could not locate request for tag %#x\n",
+			tag);
+		return NULL;
+	}
+	if (unlikely(nvme_genctr_mask(nvme_req(rq)->genctr) != genctr)) {
+		dev_err(nvme_req(rq)->ctrl->device,
+			"request %#x genctr mismatch (got %#x expected %#x)\n",
+			tag, genctr, nvme_genctr_mask(nvme_req(rq)->genctr));
+		return NULL;
+	}
+	return rq;
+}
+
+static inline struct request *nvme_cid_to_rq(struct blk_mq_tags *tags,
+                u16 command_id)
+{
+	return blk_mq_tag_to_rq(tags, nvme_tag_from_cid(command_id));
+}
+
 #ifdef CONFIG_FAULT_INJECTION_DEBUG_FS
 void nvme_fault_inject_init(struct nvme_fault_inject *fault_inj,
 			    const char *dev_name);
@@ -594,7 +638,8 @@ static inline void nvme_put_ctrl(struct nvme_ctrl *ctrl)
 
 static inline bool nvme_is_aen_req(u16 qid, __u16 command_id)
 {
-	return !qid && command_id >= NVME_AQ_BLK_MQ_DEPTH;
+	return !qid &&
+		nvme_tag_from_cid(command_id) >= NVME_AQ_BLK_MQ_DEPTH;
 }
 
 void nvme_complete_rq(struct request *req);
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index d2ee9e46f5d3..e111532c2082 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1012,7 +1012,7 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq, u16 idx)
 		return;
 	}
 
-	req = blk_mq_tag_to_rq(nvme_queue_tagset(nvmeq), command_id);
+	req = nvme_find_rq(nvme_queue_tagset(nvmeq), command_id);
 	if (unlikely(!req)) {
 		dev_warn(nvmeq->dev->ctrl.device,
 			"invalid id %d completed on queue %d\n",
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 74bf2c7f2b80..14d5023603d7 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1730,10 +1730,10 @@ static void nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue,
 	struct request *rq;
 	struct nvme_rdma_request *req;
 
-	rq = blk_mq_tag_to_rq(nvme_rdma_tagset(queue), cqe->command_id);
+	rq = nvme_find_rq(nvme_rdma_tagset(queue), cqe->command_id);
 	if (!rq) {
 		dev_err(queue->ctrl->ctrl.device,
-			"tag 0x%x on QP %#x not found\n",
+			"got bad command_id %#x on QP %#x\n",
 			cqe->command_id, queue->qp->qp_num);
 		nvme_rdma_error_recovery(queue->ctrl);
 		return;
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 3ad65f42fc1e..b906ee41449e 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -488,11 +488,11 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
 {
 	struct request *rq;
 
-	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), cqe->command_id);
+	rq = nvme_find_rq(nvme_tcp_tagset(queue), cqe->command_id);
 	if (!rq) {
 		dev_err(queue->ctrl->ctrl.device,
-			"queue %d tag 0x%x not found\n",
-			nvme_tcp_queue_id(queue), cqe->command_id);
+			"got bad cqe.command_id %#x on queue %d\n",
+			cqe->command_id, nvme_tcp_queue_id(queue));
 		nvme_tcp_error_recovery(&queue->ctrl->ctrl);
 		return -EINVAL;
 	}
@@ -509,11 +509,11 @@ static int nvme_tcp_handle_c2h_data(struct nvme_tcp_queue *queue,
 {
 	struct request *rq;
 
-	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
+	rq = nvme_find_rq(nvme_tcp_tagset(queue), pdu->command_id);
 	if (!rq) {
 		dev_err(queue->ctrl->ctrl.device,
-			"queue %d tag %#x not found\n",
-			nvme_tcp_queue_id(queue), pdu->command_id);
+			"got bad c2hdata.command_id %#x on queue %d\n",
+			pdu->command_id, nvme_tcp_queue_id(queue));
 		return -ENOENT;
 	}
 
@@ -607,7 +607,7 @@ static int nvme_tcp_setup_h2c_data_pdu(struct nvme_tcp_request *req,
 	data->hdr.plen =
 		cpu_to_le32(data->hdr.hlen + hdgst + req->pdu_len + ddgst);
 	data->ttag = pdu->ttag;
-	data->command_id = rq->tag;
+	data->command_id = nvme_cid(rq);
 	data->data_offset = cpu_to_le32(req->data_sent);
 	data->data_length = cpu_to_le32(req->pdu_len);
 	return 0;
@@ -620,11 +620,11 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,
 	struct request *rq;
 	int ret;
 
-	rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
+	rq = nvme_find_rq(nvme_tcp_tagset(queue), pdu->command_id);
 	if (!rq) {
 		dev_err(queue->ctrl->ctrl.device,
-			"queue %d tag %#x not found\n",
-			nvme_tcp_queue_id(queue), pdu->command_id);
+			"got bad r2t.command_id %#x on queue %d\n",
+			pdu->command_id, nvme_tcp_queue_id(queue));
 		return -ENOENT;
 	}
 	req = blk_mq_rq_to_pdu(rq);
@@ -704,7 +704,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 {
 	struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu;
 	struct request *rq =
-		blk_mq_tag_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
+		nvme_cid_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
 	struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
 
 	while (true) {
@@ -797,8 +797,8 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
 	}
 
 	if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {
-		struct request *rq = blk_mq_tag_to_rq(nvme_tcp_tagset(queue),
-						pdu->command_id);
+		struct request *rq = nvme_cid_to_rq(nvme_tcp_tagset(queue),
+					pdu->command_id);
 
 		nvme_tcp_end_request(rq, NVME_SC_SUCCESS);
 		queue->nr_cqe++;
diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index cb30cb942e1d..88d34212b6dc 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -107,10 +107,10 @@ static void nvme_loop_queue_response(struct nvmet_req *req)
 	} else {
 		struct request *rq;
 
-		rq = blk_mq_tag_to_rq(nvme_loop_tagset(queue), cqe->command_id);
+		rq = nvme_find_rq(nvme_loop_tagset(queue), cqe->command_id);
 		if (!rq) {
 			dev_err(queue->ctrl->ctrl.device,
-				"tag 0x%x on queue %d not found\n",
+				"got bad command_id %#x on queue %d\n",
 				cqe->command_id, nvme_loop_queue_idx(queue));
 			return;
 		}
-- 
2.27.0



From chaitanya.kulkarni at wdc.com  Wed Jun 16 15:15:50 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed, 16 Jun 2021 15:15:50 -0700
Subject: [PATCH 0/3] nvme: remove struct zeroout memset calls
Message-ID: <20210616221553.55914-1-chaitanya.kulkarni@wdc.com>

Hi,

Initialize structure so we can remove the zerout memset calls.

-ck

Chaitanya Kulkarni (3):
  nvmet: remove zeroout memset call for struct
  nvme: remove zeroout memset call for struct
  nvme-pci: remove zeroout memset call for struct

 drivers/nvme/host/core.c   | 19 ++++++-------------
 drivers/nvme/host/pci.c    | 24 ++++++++----------------
 drivers/nvme/target/rdma.c |  3 +--
 3 files changed, 15 insertions(+), 31 deletions(-)

-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun 16 15:15:51 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed, 16 Jun 2021 15:15:51 -0700
Subject: [PATCH 1/3] nvmet: remove zeroout memset call for struct
In-Reply-To: <20210616221553.55914-1-chaitanya.kulkarni@wdc.com>
References: <20210616221553.55914-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210616221553.55914-2-chaitanya.kulkarni@wdc.com>

Declare and initialize structure variables to zero values so that we can
remove zeroout memset calls in the target/rdma.c.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/rdma.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 7d607f435e36..891174ccd44b 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -1257,7 +1257,7 @@ nvmet_rdma_find_get_device(struct rdma_cm_id *cm_id)
 
 static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue)
 {
-	struct ib_qp_init_attr qp_attr;
+	struct ib_qp_init_attr qp_attr = { };
 	struct nvmet_rdma_device *ndev = queue->dev;
 	int nr_cqe, ret, i, factor;
 
@@ -1275,7 +1275,6 @@ static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue)
 		goto out;
 	}
 
-	memset(&qp_attr, 0, sizeof(qp_attr));
 	qp_attr.qp_context = queue;
 	qp_attr.event_handler = nvmet_rdma_qp_event;
 	qp_attr.send_cq = queue->cq;
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun 16 15:15:52 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed, 16 Jun 2021 15:15:52 -0700
Subject: [PATCH 2/3] nvme: remove zeroout memset call for struct
In-Reply-To: <20210616221553.55914-1-chaitanya.kulkarni@wdc.com>
References: <20210616221553.55914-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210616221553.55914-3-chaitanya.kulkarni@wdc.com>

Declare and initialize structure variables to zero values so that we can
remove zeroout memset calls in the host/core.c.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/core.c | 19 ++++++-------------
 1 file changed, 6 insertions(+), 13 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index cb1a76a3d42a..14d96ee4d341 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -724,9 +724,7 @@ EXPORT_SYMBOL_GPL(__nvme_check_ready);
 
 static int nvme_toggle_streams(struct nvme_ctrl *ctrl, bool enable)
 {
-	struct nvme_command c;
-
-	memset(&c, 0, sizeof(c));
+	struct nvme_command c = { };
 
 	c.directive.opcode = nvme_admin_directive_send;
 	c.directive.nsid = cpu_to_le32(NVME_NSID_ALL);
@@ -751,9 +749,8 @@ static int nvme_enable_streams(struct nvme_ctrl *ctrl)
 static int nvme_get_stream_params(struct nvme_ctrl *ctrl,
 				  struct streams_directive_params *s, u32 nsid)
 {
-	struct nvme_command c;
+	struct nvme_command c = { };
 
-	memset(&c, 0, sizeof(c));
 	memset(s, 0, sizeof(*s));
 
 	c.directive.opcode = nvme_admin_directive_recv;
@@ -1463,10 +1460,9 @@ static int nvme_features(struct nvme_ctrl *dev, u8 op, unsigned int fid,
 		unsigned int dword11, void *buffer, size_t buflen, u32 *result)
 {
 	union nvme_result res = { 0 };
-	struct nvme_command c;
+	struct nvme_command c = { };
 	int ret;
 
-	memset(&c, 0, sizeof(c));
 	c.features.opcode = op;
 	c.features.fid = cpu_to_le32(fid);
 	c.features.dword11 = cpu_to_le32(dword11);
@@ -1594,9 +1590,8 @@ int nvme_getgeo(struct block_device *bdev, struct hd_geometry *geo)
 static void nvme_init_integrity(struct gendisk *disk, u16 ms, u8 pi_type,
 				u32 max_integrity_segments)
 {
-	struct blk_integrity integrity;
+	struct blk_integrity integrity = { };
 
-	memset(&integrity, 0, sizeof(integrity));
 	switch (pi_type) {
 	case NVME_NS_DPS_PI_TYPE3:
 		integrity.profile = &t10_pi_type3_crc;
@@ -1967,13 +1962,12 @@ static int nvme_send_ns_pr_command(struct nvme_ns *ns, struct nvme_command *c,
 static int nvme_pr_command(struct block_device *bdev, u32 cdw10,
 				u64 key, u64 sa_key, u8 op)
 {
-	struct nvme_command c;
+	struct nvme_command c = { };
 	u8 data[16] = { 0, };
 
 	put_unaligned_le64(key, &data[0]);
 	put_unaligned_le64(sa_key, &data[8]);
 
-	memset(&c, 0, sizeof(c));
 	c.common.opcode = op;
 	c.common.cdw10 = cpu_to_le32(cdw10);
 
@@ -2045,9 +2039,8 @@ int nvme_sec_submit(void *data, u16 spsp, u8 secp, void *buffer, size_t len,
 		bool send)
 {
 	struct nvme_ctrl *ctrl = data;
-	struct nvme_command cmd;
+	struct nvme_command cmd = { };
 
-	memset(&cmd, 0, sizeof(cmd));
 	if (send)
 		cmd.common.opcode = nvme_admin_security_send;
 	else
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Wed Jun 16 15:15:53 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Wed, 16 Jun 2021 15:15:53 -0700
Subject: [PATCH 3/3] nvme-pci: remove zeroout memset call for struct
In-Reply-To: <20210616221553.55914-1-chaitanya.kulkarni@wdc.com>
References: <20210616221553.55914-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210616221553.55914-4-chaitanya.kulkarni@wdc.com>

Declare and initialize structure variables to zero values so that we can
remove zeroout memset calls in the host/pci.c.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/pci.c | 24 ++++++++----------------
 1 file changed, 8 insertions(+), 16 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 9cdf2099027a..c20acaeac3d0 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -307,13 +307,12 @@ static void nvme_dbbuf_free(struct nvme_queue *nvmeq)
 
 static void nvme_dbbuf_set(struct nvme_dev *dev)
 {
-	struct nvme_command c;
+	struct nvme_command c = { };
 	unsigned int i;
 
 	if (!dev->dbbuf_dbs)
 		return;
 
-	memset(&c, 0, sizeof(c));
 	c.dbbuf.opcode = nvme_admin_dbbuf;
 	c.dbbuf.prp1 = cpu_to_le64(dev->dbbuf_dbs_dma_addr);
 	c.dbbuf.prp2 = cpu_to_le64(dev->dbbuf_eis_dma_addr);
@@ -1112,9 +1111,8 @@ static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl)
 {
 	struct nvme_dev *dev = to_nvme_dev(ctrl);
 	struct nvme_queue *nvmeq = &dev->queues[0];
-	struct nvme_command c;
+	struct nvme_command c = { };
 
-	memset(&c, 0, sizeof(c));
 	c.common.opcode = nvme_admin_async_event;
 	c.common.command_id = NVME_AQ_BLK_MQ_DEPTH;
 	nvme_submit_cmd(nvmeq, &c, true);
@@ -1122,9 +1120,8 @@ static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl)
 
 static int adapter_delete_queue(struct nvme_dev *dev, u8 opcode, u16 id)
 {
-	struct nvme_command c;
+	struct nvme_command c = { };
 
-	memset(&c, 0, sizeof(c));
 	c.delete_queue.opcode = opcode;
 	c.delete_queue.qid = cpu_to_le16(id);
 
@@ -1134,7 +1131,7 @@ static int adapter_delete_queue(struct nvme_dev *dev, u8 opcode, u16 id)
 static int adapter_alloc_cq(struct nvme_dev *dev, u16 qid,
 		struct nvme_queue *nvmeq, s16 vector)
 {
-	struct nvme_command c;
+	struct nvme_command c = { };
 	int flags = NVME_QUEUE_PHYS_CONTIG;
 
 	if (!test_bit(NVMEQ_POLLED, &nvmeq->flags))
@@ -1144,7 +1141,6 @@ static int adapter_alloc_cq(struct nvme_dev *dev, u16 qid,
 	 * Note: we (ab)use the fact that the prp fields survive if no data
 	 * is attached to the request.
 	 */
-	memset(&c, 0, sizeof(c));
 	c.create_cq.opcode = nvme_admin_create_cq;
 	c.create_cq.prp1 = cpu_to_le64(nvmeq->cq_dma_addr);
 	c.create_cq.cqid = cpu_to_le16(qid);
@@ -1159,7 +1155,7 @@ static int adapter_alloc_sq(struct nvme_dev *dev, u16 qid,
 						struct nvme_queue *nvmeq)
 {
 	struct nvme_ctrl *ctrl = &dev->ctrl;
-	struct nvme_command c;
+	struct nvme_command c = { };
 	int flags = NVME_QUEUE_PHYS_CONTIG;
 
 	/*
@@ -1174,7 +1170,6 @@ static int adapter_alloc_sq(struct nvme_dev *dev, u16 qid,
 	 * Note: we (ab)use the fact that the prp fields survive if no data
 	 * is attached to the request.
 	 */
-	memset(&c, 0, sizeof(c));
 	c.create_sq.opcode = nvme_admin_create_sq;
 	c.create_sq.prp1 = cpu_to_le64(nvmeq->sq_dma_addr);
 	c.create_sq.sqid = cpu_to_le16(qid);
@@ -1255,7 +1250,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
 	struct nvme_queue *nvmeq = iod->nvmeq;
 	struct nvme_dev *dev = nvmeq->dev;
 	struct request *abort_req;
-	struct nvme_command cmd;
+	struct nvme_command cmd = { };
 	u32 csts = readl(dev->bar + NVME_REG_CSTS);
 
 	/* If PCI error recovery process is happening, we cannot reset or
@@ -1335,7 +1330,6 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
 	}
 	iod->aborted = 1;
 
-	memset(&cmd, 0, sizeof(cmd));
 	cmd.abort.opcode = nvme_admin_abort_cmd;
 	cmd.abort.cid = req->tag;
 	cmd.abort.sqid = cpu_to_le16(nvmeq->qid);
@@ -1886,10 +1880,9 @@ static int nvme_set_host_mem(struct nvme_dev *dev, u32 bits)
 {
 	u32 host_mem_size = dev->host_mem_size >> NVME_CTRL_PAGE_SHIFT;
 	u64 dma_addr = dev->host_mem_descs_dma;
-	struct nvme_command c;
+	struct nvme_command c = { };
 	int ret;
 
-	memset(&c, 0, sizeof(c));
 	c.features.opcode	= nvme_admin_set_features;
 	c.features.fid		= cpu_to_le32(NVME_FEAT_HOST_MEM_BUF);
 	c.features.dword11	= cpu_to_le32(bits);
@@ -2263,9 +2256,8 @@ static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode)
 {
 	struct request_queue *q = nvmeq->dev->ctrl.admin_q;
 	struct request *req;
-	struct nvme_command cmd;
+	struct nvme_command cmd = { };
 
-	memset(&cmd, 0, sizeof(cmd));
 	cmd.delete_queue.opcode = opcode;
 	cmd.delete_queue.qid = cpu_to_le16(nvmeq->qid);
 
-- 
2.22.1



From prabhakar.pkin at gmail.com  Wed Jun 16 18:36:06 2021
From: prabhakar.pkin at gmail.com (Prabhakar Kushwaha)
Date: Thu, 17 Jun 2021 07:06:06 +0530
Subject: [PATCH 4/4] nvme-tcp: use ctrl sgl check helper
In-Reply-To: <20210610012826.53398-5-chaitanya.kulkarni@wdc.com>
References: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
 <20210610012826.53398-5-chaitanya.kulkarni@wdc.com>
Message-ID: <CAJ2QiJLSfoZ0qXJoJ_n_1Z=7JzL_TZp0ZY2Wb0EgB-bQNzatjw@mail.gmail.com>

Hi Chaitanya,

On Thu, Jun 10, 2021 at 7:04 AM Chaitanya Kulkarni
<chaitanya.kulkarni at wdc.com> wrote:
>
> Use the helper to check NVMe controller's SGL support.
>
> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> ---
>  drivers/nvme/host/tcp.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> index 6a65b0516180..c7bd37103cf4 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -1993,7 +1993,7 @@ static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new)
>                 goto destroy_admin;
>         }
>
> -       if (!(ctrl->sgls & ((1 << 0) | (1 << 1)))) {
> +       if (!nvme_ctrl_sgl_supported(ctrl)) {
>                 ret = -EOPNOTSUPP;
>                 dev_err(ctrl->device, "Mandatory sgls are not supported!\n");
>                 goto destroy_admin;
> --
> 2.22.1
>
>

base patch defines nvme_ctrl_sgl_supported as below
 +static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 +{
 +    if (!(ctrl->sgls & ((1 << 0) | (1 << 1))))
 +        return true;
 +
 +    return false;
 +}

The "!" is already part of nvme_ctrl_sgl_supported(). There is no
point in keeping "!" at other places.
So, either remove "!" from other places or from nvme_ctrl_sgl_supported().

I request you to check the logic again.
In case you are planning to fix, the best place would be
nvme_ctrl_sgl_supported() to have minimal changes.

 -pk


From Chaitanya.Kulkarni at wdc.com  Wed Jun 16 21:59:11 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Thu, 17 Jun 2021 04:59:11 +0000
Subject: [PATCH 4/4] nvme-tcp: use ctrl sgl check helper
In-Reply-To: <CAJ2QiJLSfoZ0qXJoJ_n_1Z=7JzL_TZp0ZY2Wb0EgB-bQNzatjw@mail.gmail.com>
References: <20210610012826.53398-1-chaitanya.kulkarni@wdc.com>
 <20210610012826.53398-5-chaitanya.kulkarni@wdc.com>,
 <CAJ2QiJLSfoZ0qXJoJ_n_1Z=7JzL_TZp0ZY2Wb0EgB-bQNzatjw@mail.gmail.com>
Message-ID: <132CA045-F44B-432B-A995-779A5D108E28@wdc.com>



> On Jun 16, 2021, at 6:37 PM, Prabhakar Kushwaha <prabhakar.pkin at gmail.com> wrote:
> 
> ?Hi Chaitanya,
> 
>> On Thu, Jun 10, 2021 at 7:04 AM Chaitanya Kulkarni
>> <chaitanya.kulkarni at wdc.com> wrote:
>> 
>> Use the helper to check NVMe controller's SGL support.
>> 
>> Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
>> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
>> ---
>> drivers/nvme/host/tcp.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
>> index 6a65b0516180..c7bd37103cf4 100644
>> --- a/drivers/nvme/host/tcp.c
>> +++ b/drivers/nvme/host/tcp.c
>> @@ -1993,7 +1993,7 @@ static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new)
>>                goto destroy_admin;
>>        }
>> 
>> -       if (!(ctrl->sgls & ((1 << 0) | (1 << 1)))) {
>> +       if (!nvme_ctrl_sgl_supported(ctrl)) {
>>                ret = -EOPNOTSUPP;
>>                dev_err(ctrl->device, "Mandatory sgls are not supported!\n");
>>                goto destroy_admin;
>> --
>> 2.22.1
>> 
>> 
> 
> base patch defines nvme_ctrl_sgl_supported as below
> +static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
> +{
> +    if (!(ctrl->sgls & ((1 << 0) | (1 << 1))))
> +        return true;
> +
> +    return false;
> +}
> 
> The "!" is already part of nvme_ctrl_sgl_supported(). There is no
> point in keeping "!" at other places.
> So, either remove "!" from other places or from nvme_ctrl_sgl_supported().
> 
> I request you to check the logic again.
> In case you are planning to fix, the best place would be
> nvme_ctrl_sgl_supported() to have minimal changes.
> 
> -pk


Thanks for pointing this out. I'll send out the fix shortly. 


From hare at suse.de  Wed Jun 16 22:45:39 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 17 Jun 2021 07:45:39 +0200
Subject: [PATCH v3 1/4] params: lift param_set_uint_minmax to common code
In-Reply-To: <20210616211936.451128-2-sagi@grimberg.me>
References: <20210616211936.451128-1-sagi@grimberg.me>
 <20210616211936.451128-2-sagi@grimberg.me>
Message-ID: <9855eaf9-52c0-05d7-bc04-d777cdfca262@suse.de>

On 6/16/21 11:19 PM, Sagi Grimberg wrote:
> It is a useful helper hence move it to common code so others can enjoy
> it.
> 
> Suggested-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> Cc: linux-nfs at vger.kernel.org
> Cc: linux-kernel at vger.kernel.org
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>   include/linux/moduleparam.h |  3 +++
>   kernel/params.c             | 19 +++++++++++++++++++
>   net/sunrpc/xprtsock.c       | 18 ------------------
>   3 files changed, 22 insertions(+), 18 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.com>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From hare at suse.de  Wed Jun 16 22:46:08 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 17 Jun 2021 07:46:08 +0200
Subject: [PATCH v3 2/4] nvme-pci: limit maximum queue depth to 4095
In-Reply-To: <20210616211936.451128-3-sagi@grimberg.me>
References: <20210616211936.451128-1-sagi@grimberg.me>
 <20210616211936.451128-3-sagi@grimberg.me>
Message-ID: <6b71ca0d-c2b8-db04-65d8-3055690cbd22@suse.de>

On 6/16/21 11:19 PM, Sagi Grimberg wrote:
> We are going to use the upper 4-bits of the command_id for a generation
> counter, so enforce the new queue depth upper limit. As we enforce
> both min and max queue depth, use param_set_uint_minmax istead of
> open coding it.
> 
> Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
> ---
>   drivers/nvme/host/pci.c | 15 ++++++---------
>   1 file changed, 6 insertions(+), 9 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From jongkang.kim2 at gmail.com  Wed Jun 16 23:02:17 2021
From: jongkang.kim2 at gmail.com (JK Kim)
Date: Thu, 17 Jun 2021 15:02:17 +0900
Subject: [PATCH] nvme-pci: fix var. type for increasing cq_head
Message-ID: <YMrlaTjfyy9slGg6@localhost>

nvmeq->cq_head is compared with nvmeq->q_depth and changed the value
and cq_phase for handling the next cq db.

but, nvmeq->q_depth's type is u32 and max. value is 0x10000 when
CQP.MSQE is 0xffff and io_queue_depth is 0x10000.

current temp. variable for comparing with nvmeq->q_depth is overflowed
when previous nvmeq->cq_head is 0xffff.

in this case, nvmeq->cq_phase is not updated.
so, fix data type for temp. variable to u32.

Signed-off-by: JK Kim <jongkang.kim2 at gmail.com>
---
 drivers/nvme/host/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index a29b170701fc..2995e87ce776 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1032,7 +1032,7 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq, u16 idx)
 
 static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
 {
-	u16 tmp = nvmeq->cq_head + 1;
+	u32 tmp = nvmeq->cq_head + 1;
 
 	if (tmp == nvmeq->q_depth) {
 		nvmeq->cq_head = 0;
-- 
2.30.1



From dwagner at suse.de  Thu Jun 17 01:00:04 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Thu, 17 Jun 2021 10:00:04 +0200
Subject: [PATCH v3 1/4] params: lift param_set_uint_minmax to common code
In-Reply-To: <20210616211936.451128-2-sagi@grimberg.me>
References: <20210616211936.451128-1-sagi@grimberg.me>
 <20210616211936.451128-2-sagi@grimberg.me>
Message-ID: <20210617080004.ujxn73bntthjuoqu@beryllium.lan>

On Wed, Jun 16, 2021 at 02:19:33PM -0700, Sagi Grimberg wrote:
> It is a useful helper hence move it to common code so others can enjoy
> it.
> 
> Suggested-by: Christoph Hellwig <hch at lst.de>
> Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> Cc: linux-nfs at vger.kernel.org
> Cc: linux-kernel at vger.kernel.org
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>

[...]

> +int param_set_uint_minmax(const char *val,
> +			  const struct kernel_param *kp,
> +			  unsigned int min, unsigned int max)
> +{
> +	unsigned int num;
> +	int ret;
> +
> +	if (!val)
> +		return -EINVAL;
> +	ret = kstrtouint(val, 0, &num);
> +	if (ret)
> +		return ret;
> +	if (num < min || num > max)
> +		return -EINVAL;
> +	*((unsigned int *)kp->arg) = num;
> +	return 0;
> +}
> +EXPORT_SYMBOL(param_set_uint_minmax);

Couldn't this be EXPORT_SYMBOL_GPL?


From dwagner at suse.de  Thu Jun 17 01:04:28 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Thu, 17 Jun 2021 10:04:28 +0200
Subject: [PATCH v3 2/4] nvme-pci: limit maximum queue depth to 4095
In-Reply-To: <20210616211936.451128-3-sagi@grimberg.me>
References: <20210616211936.451128-1-sagi@grimberg.me>
 <20210616211936.451128-3-sagi@grimberg.me>
Message-ID: <20210617080428.dv7qbo3yuvkhkkmw@beryllium.lan>

On Wed, Jun 16, 2021 at 02:19:34PM -0700, Sagi Grimberg wrote:
> We are going to use the upper 4-bits of the command_id for a generation
> counter, so enforce the new queue depth upper limit. As we enforce
> both min and max queue depth, use param_set_uint_minmax istead of
> open coding it.
> 
> Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
> ---

Reviewed-by: Daniel Wagner <dwagner at suse.de>


From dwagner at suse.de  Thu Jun 17 01:11:25 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Thu, 17 Jun 2021 10:11:25 +0200
Subject: [PATCH v3 3/4] nvme-tcp: don't check blk_mq_tag_to_rq when
 receiving pdu data
In-Reply-To: <20210616211936.451128-4-sagi@grimberg.me>
References: <20210616211936.451128-1-sagi@grimberg.me>
 <20210616211936.451128-4-sagi@grimberg.me>
Message-ID: <20210617081125.ufn6ldgzzhzy6grn@beryllium.lan>

On Wed, Jun 16, 2021 at 02:19:35PM -0700, Sagi Grimberg wrote:
> We already validate it when receiving the c2hdata pdu header
> and this is not changing so this is a redundant check.
> 
> Reviewed-by: Hannes Reinecke <hare at suse.de>
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>

Reviewed-by: Daniel Wagner <dwagner at suse.de>


From dwagner at suse.de  Thu Jun 17 01:56:27 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Thu, 17 Jun 2021 10:56:27 +0200
Subject: [PATCH v3 4/4] nvme: code command_id with a genctr for
 use-after-free validation
In-Reply-To: <20210616211936.451128-5-sagi@grimberg.me>
References: <20210616211936.451128-1-sagi@grimberg.me>
 <20210616211936.451128-5-sagi@grimberg.me>
Message-ID: <20210617085627.mhqobcbhsmrusuyz@beryllium.lan>

On Wed, Jun 16, 2021 at 02:19:36PM -0700, Sagi Grimberg wrote:
> We cannot detect a (perhaps buggy) controller that is sending us
> a completion for a request that was already completed (for example
> sending a completion twice), this phenomenon was seen in the wild
> a few times.
> 
> So to protect against this, we use the upper 4 msbits of the nvme sqe
> command_id to use as a 4-bit generation counter and verify it matches
> the existing request generation that is incrementing on every execution.
> 
> The 16-bit command_id structure now is constructed by:
> | xxxx | xxxxxxxxxxxx |
>   gen    request tag
> 
> This means that we are giving up some possible queue depth as 12 bits
> allow for a maximum queue depth of 4095 instead of 65536, however we
> never create such long queues anyways so no real harm done.
> 
> Suggested-by: Keith Busch <kbusch at kernel.org>
> Reviewed-by: Hannes Reinecke <hare at suse.de>
> Acked-by: Keith Busch <kbusch at kernel.org>
> Signed-off-by: Sagi Grimberg <sagi at grimberg.me>

I've tested (only functional) this on FC (NetApp target). All looks
good.

Tested-by: Daniel Wagner <dwagner at suse.de>
Reviewed-by: Daniel Wagner <dwagner at suse.de>


From hch at lst.de  Thu Jun 17 06:48:22 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 17 Jun 2021 15:48:22 +0200
Subject: [PATCH v3 1/4] params: lift param_set_uint_minmax to common code
In-Reply-To: <20210616211936.451128-2-sagi@grimberg.me>
References: <20210616211936.451128-1-sagi@grimberg.me>
 <20210616211936.451128-2-sagi@grimberg.me>
Message-ID: <20210617134822.GA27880@lst.de>

This needs a Cc to linux-kernel..

> @@ -432,6 +432,9 @@ extern const struct kernel_param_ops param_ops_uint;
>  extern int param_set_uint(const char *val, const struct kernel_param *kp);
>  extern int param_get_uint(char *buffer, const struct kernel_param *kp);
>  #define param_check_uint(name, p) __param_check(name, p, unsigned int)
> +int param_set_uint_minmax(const char *val,
> +			  const struct kernel_param *kp,
> +			  unsigned int min, unsigned int max);

Super minor nitpick, but for consistency I'd move it above the
param_check_uint definition/


From hch at lst.de  Thu Jun 17 06:54:46 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 17 Jun 2021 15:54:46 +0200
Subject: [PATCH] nvme.h: add missing nvme_lba_range_type endianness
 annotations
In-Reply-To: <20210616052508.78970-1-wesley.sheng@amd.com>
References: <20210616052508.78970-1-wesley.sheng@amd.com>
Message-ID: <20210617135446.GA28419@lst.de>

Thanks,

applied to nvme-5.14.

(and I hope you do not actually plan to use the horribly ill defined
LBA ranges for anything..)


From hch at lst.de  Thu Jun 17 06:54:55 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 17 Jun 2021 15:54:55 +0200
Subject: [PATCH 0/3] nvme: remove struct zeroout memset calls
In-Reply-To: <20210616221553.55914-1-chaitanya.kulkarni@wdc.com>
References: <20210616221553.55914-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210617135455.GB28419@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Thu Jun 17 06:55:07 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 17 Jun 2021 15:55:07 +0200
Subject: [PATCH] nvme-pci: fix var. type for increasing cq_head
In-Reply-To: <YMrlaTjfyy9slGg6@localhost>
References: <YMrlaTjfyy9slGg6@localhost>
Message-ID: <20210617135507.GC28419@lst.de>

Thanks,

applied to nvme-5.14.


From sagi at grimberg.me  Thu Jun 17 12:06:11 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Thu, 17 Jun 2021 12:06:11 -0700
Subject: [PATCH 2/3] nvmet: prevent creating ctrl with no namespace
In-Reply-To: <20210613200252.9167-3-chaitanya.kulkarni@wdc.com>
References: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
 <20210613200252.9167-3-chaitanya.kulkarni@wdc.com>
Message-ID: <b6b62366-fff4-81e8-5b3b-c3062211389b@grimberg.me>


>  From Spec :-
> 
> If the controller supports Asymmetric Namespace Access Reporting, then
> MNAN shall be set to a non-zero value that is less than or equal
> to the NN value.
> 
> In current implementation if subsys has 0 namespaces then we report
> wrong MNAN value for the controller if we allow controller creation.
> 
> Prevent new controller creation if the subsys associated with the new
> controller has no namespaces.

This is a perfectly valid thing to do, preventing this is not the
solution here.

Why not just set both nn and mnan to NVMET_MAX_NAMESPACES and be done
with it?

> 
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> ---
>   drivers/nvme/target/core.c | 9 +++++++++
>   1 file changed, 9 insertions(+)
> 
> diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
> index 146909486b8f..26d93ac9bd2e 100644
> --- a/drivers/nvme/target/core.c
> +++ b/drivers/nvme/target/core.c
> @@ -1325,6 +1325,15 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
>   		goto out;
>   	}
>   
> +	mutex_lock(&subsys->lock);
> +	if (!subsys->max_nsid) {
> +		mutex_unlock(&subsys->lock);
> +		pr_err("invalid mnan value for the ana enabled controller\n");
> +		status = NVME_SC_INTERNAL;
> +		goto out_put_subsystem;
> +	}
> +	mutex_unlock(&subsys->lock);
> +
>   	down_read(&nvmet_config_sem);
>   	if (!nvmet_host_allowed(subsys, hostnqn)) {
>   		pr_info("connect by host %s for subsystem %s not allowed\n",
> 


From lkp at intel.com  Thu Jun 17 19:08:35 2021
From: lkp at intel.com (kernel test robot)
Date: Fri, 18 Jun 2021 10:08:35 +0800
Subject: [linux-nvme:nvme-5.14] BUILD SUCCESS
 8cf486e131b351db4f224078bef8e1efedcf0340
Message-ID: <60cc0023.a3gxG2NSPUI22eRV%lkp@intel.com>

tree/branch: git://git.infradead.org/nvme.git nvme-5.14
branch HEAD: 8cf486e131b351db4f224078bef8e1efedcf0340  nvme.h: add missing nvme_lba_range_type endianness annotations

elapsed time: 723m

configs tested: 130
configs skipped: 3

The following configs have been built successfully.
More configs may be tested in the coming days.

gcc tested configs:
arm                                 defconfig
arm64                            allyesconfig
arm64                               defconfig
arm                              allyesconfig
arm                              allmodconfig
sh                           se7705_defconfig
arm                         s3c6400_defconfig
powerpc                      arches_defconfig
arm                          ep93xx_defconfig
um                            kunit_defconfig
arm                         cm_x300_defconfig
mips                           ip27_defconfig
mips                      bmips_stb_defconfig
arm                        mini2440_defconfig
mips                        bcm63xx_defconfig
arm                          pxa3xx_defconfig
sh                          rsk7203_defconfig
sh                         ap325rxa_defconfig
parisc                           alldefconfig
h8300                    h8300h-sim_defconfig
powerpc                 mpc834x_mds_defconfig
arm64                            alldefconfig
arm                        keystone_defconfig
riscv                          rv32_defconfig
arm                        trizeps4_defconfig
m68k                            q40_defconfig
arm                       aspeed_g5_defconfig
m68k                       m5249evb_defconfig
riscv                    nommu_virt_defconfig
arm                          lpd270_defconfig
arm                           sama5_defconfig
arm                            pleb_defconfig
mips                           xway_defconfig
powerpc                       holly_defconfig
powerpc                      walnut_defconfig
arm                           h3600_defconfig
sh                  sh7785lcr_32bit_defconfig
arm                          pxa910_defconfig
mips                      maltasmvp_defconfig
arm                         hackkit_defconfig
sh                           se7206_defconfig
mips                           rs90_defconfig
powerpc                     tqm8555_defconfig
powerpc                     taishan_defconfig
sh                           se7750_defconfig
arc                            hsdk_defconfig
openrisc                 simple_smp_defconfig
arm                          moxart_defconfig
powerpc                     redwood_defconfig
powerpc                       ppc64_defconfig
powerpc                      chrp32_defconfig
mips                           ip28_defconfig
m68k                        mvme147_defconfig
mips                  decstation_64_defconfig
mips                        workpad_defconfig
mips                           gcw0_defconfig
arm                         lubbock_defconfig
arc                              alldefconfig
xtensa                       common_defconfig
x86_64                            allnoconfig
ia64                             allmodconfig
ia64                                defconfig
ia64                             allyesconfig
m68k                             allmodconfig
m68k                                defconfig
m68k                             allyesconfig
nios2                               defconfig
arc                              allyesconfig
nds32                             allnoconfig
nds32                               defconfig
nios2                            allyesconfig
csky                                defconfig
alpha                               defconfig
alpha                            allyesconfig
xtensa                           allyesconfig
h8300                            allyesconfig
arc                                 defconfig
sh                               allmodconfig
parisc                              defconfig
s390                             allyesconfig
s390                             allmodconfig
parisc                           allyesconfig
s390                                defconfig
i386                             allyesconfig
sparc                            allyesconfig
sparc                               defconfig
i386                                defconfig
mips                             allyesconfig
mips                             allmodconfig
powerpc                          allyesconfig
powerpc                          allmodconfig
powerpc                           allnoconfig
x86_64               randconfig-a004-20210617
x86_64               randconfig-a001-20210617
x86_64               randconfig-a002-20210617
x86_64               randconfig-a003-20210617
x86_64               randconfig-a006-20210617
x86_64               randconfig-a005-20210617
i386                 randconfig-a002-20210617
i386                 randconfig-a006-20210617
i386                 randconfig-a001-20210617
i386                 randconfig-a004-20210617
i386                 randconfig-a005-20210617
i386                 randconfig-a003-20210617
i386                 randconfig-a015-20210617
i386                 randconfig-a013-20210617
i386                 randconfig-a016-20210617
i386                 randconfig-a012-20210617
i386                 randconfig-a014-20210617
i386                 randconfig-a011-20210617
riscv                    nommu_k210_defconfig
riscv                            allyesconfig
riscv                             allnoconfig
riscv                               defconfig
riscv                            allmodconfig
x86_64                    rhel-8.3-kselftests
um                           x86_64_defconfig
um                             i386_defconfig
x86_64                           allyesconfig
x86_64                              defconfig
x86_64                               rhel-8.3
x86_64                      rhel-8.3-kbuiltin
x86_64                                  kexec

clang tested configs:
x86_64               randconfig-b001-20210617
x86_64               randconfig-a015-20210617
x86_64               randconfig-a011-20210617
x86_64               randconfig-a014-20210617
x86_64               randconfig-a012-20210617
x86_64               randconfig-a013-20210617
x86_64               randconfig-a016-20210617

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org


From hch at lst.de  Thu Jun 17 21:28:14 2021
From: hch at lst.de (Christoph Hellwig)
Date: Fri, 18 Jun 2021 06:28:14 +0200
Subject: [PATCH 2/3] nvmet: prevent creating ctrl with no namespace
In-Reply-To: <b6b62366-fff4-81e8-5b3b-c3062211389b@grimberg.me>
References: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
 <20210613200252.9167-3-chaitanya.kulkarni@wdc.com>
 <b6b62366-fff4-81e8-5b3b-c3062211389b@grimberg.me>
Message-ID: <20210618042814.GA18021@lst.de>

On Thu, Jun 17, 2021 at 12:06:11PM -0700, Sagi Grimberg wrote:
> Why not just set both nn and mnan to NVMET_MAX_NAMESPACES and be done
> with it?

Yes, I'm pretty sure I suggested that before somewhere..


From pkushwaha at marvell.com  Fri Jun 18 06:39:56 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Fri, 18 Jun 2021 16:39:56 +0300
Subject: [PATCH] nvme-tcp: Use __dev_get_by_name instead dev_get_by_name for
 OPT_HOST_IFACE
Message-ID: <20210618133956.10637-1-pkushwaha@marvell.com>

dev_get_by_name() finds network device by name but it also increases
reference count.
Increasing the ref count,
If nvme-tcp queue is present and the network device driver is removed
before nvme_tcp, we will face the following continuous log:
  "kernel:unregister_netdevice: waiting for <eth> to become
  free. Usage count = 2"
And rmmod further halts. Similar case arises during reboot/shutdown
with nvme-tcp queue present and both never completes.

As a fix we will use __dev_get_by_name() which find network device by
name without increasing any reference counter.

Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
---
 drivers/nvme/host/tcp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index c7bd37103cf4..f9b527e71c13 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2533,7 +2533,7 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev,
 	}
 
 	if (opts->mask & NVMF_OPT_HOST_IFACE) {
-		ctrl->ndev = dev_get_by_name(&init_net, opts->host_iface);
+		ctrl->ndev = __dev_get_by_name(&init_net, opts->host_iface);
 		if (!ctrl->ndev) {
 			pr_err("invalid interface passed: %s\n",
 			       opts->host_iface);
-- 
2.30.1



From hch at lst.de  Fri Jun 18 06:42:44 2021
From: hch at lst.de (Christoph Hellwig)
Date: Fri, 18 Jun 2021 15:42:44 +0200
Subject: [PATCH] nvme-tcp: Use __dev_get_by_name instead
 dev_get_by_name for OPT_HOST_IFACE
In-Reply-To: <20210618133956.10637-1-pkushwaha@marvell.com>
References: <20210618133956.10637-1-pkushwaha@marvell.com>
Message-ID: <20210618134244.GA16005@lst.de>

On Fri, Jun 18, 2021 at 04:39:56PM +0300, Prabhakar Kushwaha wrote:
> dev_get_by_name() finds network device by name but it also increases
> reference count.
> Increasing the ref count,
> If nvme-tcp queue is present and the network device driver is removed
> before nvme_tcp, we will face the following continuous log:
>   "kernel:unregister_netdevice: waiting for <eth> to become
>   free. Usage count = 2"
> And rmmod further halts. Similar case arises during reboot/shutdown
> with nvme-tcp queue present and both never completes.
> 
> As a fix we will use __dev_get_by_name() which find network device by
> name without increasing any reference counter.

And when you remove it without the refcount we'll now have a stale
kernel pointer?


From hch at lst.de  Fri Jun 18 07:01:47 2021
From: hch at lst.de (Christoph Hellwig)
Date: Fri, 18 Jun 2021 16:01:47 +0200
Subject: [PATCH 13/16] block: switch polling to be bio based
In-Reply-To: <YMliP6sFVuPhMbOB@T590>
References: <20210615131034.752623-1-hch@lst.de>
 <20210615131034.752623-14-hch@lst.de> <YMliP6sFVuPhMbOB@T590>
Message-ID: <20210618140147.GA16258@lst.de>

On Wed, Jun 16, 2021 at 10:30:23AM +0800, Ming Lei wrote:
> Not sure disk is valid, we only hold the disk when opening a bdev, but
> the bdev can be closed during polling.

How?  On a block device the caller needs to hold the block device open
to read/write from it.  On a file systems the file systems needs to
be mounted, which also holds a bdev reference.

> Also disk always holds one
> reference on request queue, so if disk is valid, no need to grab queue's
> refcnt in bio_poll().

But we need to avoid going into the lowlevel blk-mq polling code to not
reference the potentially freed hctxs or tags as correctly pointed by
yourself on the previous iteration.


From zhouchuangao at vivo.com  Fri Jun 18 07:10:58 2021
From: zhouchuangao at vivo.com (zhouchuangao)
Date: Fri, 18 Jun 2021 07:10:58 -0700
Subject: [PATCH] drivers/nvme/host: Use kobj_to_dev() API
Message-ID: <1624025458-56465-1-git-send-email-zhouchuangao@vivo.com>

Use kobj_to_dev() API instead of container_of().

Signed-off-by: zhouchuangao <zhouchuangao at vivo.com>
---
 drivers/nvme/host/core.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 468d264..bec6965 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3195,7 +3195,7 @@ static struct attribute *nvme_ns_id_attrs[] = {
 static umode_t nvme_ns_id_attrs_are_visible(struct kobject *kobj,
 		struct attribute *a, int n)
 {
-	struct device *dev = container_of(kobj, struct device, kobj);
+	struct device *dev = kobj_to_dev(kobj);
 	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
 
 	if (a == &dev_attr_uuid.attr) {
@@ -3467,7 +3467,7 @@ static struct attribute *nvme_dev_attrs[] = {
 static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		struct attribute *a, int n)
 {
-	struct device *dev = container_of(kobj, struct device, kobj);
+	struct device *dev = kobj_to_dev(kobj);
 	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
 
 	if (a == &dev_attr_delete_controller.attr && !ctrl->ops->delete_ctrl)
-- 
2.7.4



From hch at lst.de  Fri Jun 18 07:18:17 2021
From: hch at lst.de (Christoph Hellwig)
Date: Fri, 18 Jun 2021 16:18:17 +0200
Subject: [PATCH] drivers/nvme/host: Use kobj_to_dev() API
In-Reply-To: <1624025458-56465-1-git-send-email-zhouchuangao@vivo.com>
References: <1624025458-56465-1-git-send-email-zhouchuangao@vivo.com>
Message-ID: <20210618141817.GA18781@lst.de>

On Fri, Jun 18, 2021 at 07:10:58AM -0700, zhouchuangao wrote:
> Use kobj_to_dev() API instead of container_of().

Why?  That just makes the code harder to read.


From ming.lei at redhat.com  Fri Jun 18 07:27:20 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Fri, 18 Jun 2021 22:27:20 +0800
Subject: [PATCH 13/16] block: switch polling to be bio based
In-Reply-To: <20210618140147.GA16258@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
 <20210615131034.752623-14-hch@lst.de> <YMliP6sFVuPhMbOB@T590>
 <20210618140147.GA16258@lst.de>
Message-ID: <YMytSIBm7k2pqFlc@T590>

On Fri, Jun 18, 2021 at 04:01:47PM +0200, Christoph Hellwig wrote:
> On Wed, Jun 16, 2021 at 10:30:23AM +0800, Ming Lei wrote:
> > Not sure disk is valid, we only hold the disk when opening a bdev, but
> > the bdev can be closed during polling.
> 
> How?  On a block device the caller needs to hold the block device open
> to read/write from it.  On a file systems the file systems needs to
> be mounted, which also holds a bdev reference.

+       rcu_read_lock();
+       bio = READ_ONCE(kiocb->private);
+       if (bio && bio->bi_bdev)

The bio may be ended now from another polling job, then the disk is
closed & deleted, and released. Then request queue & hctxs are released.

+               ret = bio_poll(bio, flags);

But disk & request queue & hctx can still be referred in above bio_poll().

+       rcu_read_unlock();

> 
> > Also disk always holds one
> > reference on request queue, so if disk is valid, no need to grab queue's
> > refcnt in bio_poll().
> 
> But we need to avoid going into the lowlevel blk-mq polling code to not
> reference the potentially freed hctxs or tags as correctly pointed by
> yourself on the previous iteration.

If request queue isn't released, hctx won't be freed too. Tagset can be
freed, but it is supposed to not be touched after request queue is cleanup.


Thanks, 
Ming



From sagi at grimberg.me  Fri Jun 18 12:14:01 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Fri, 18 Jun 2021 12:14:01 -0700
Subject: [PATCH] nvme-tcp: Use __dev_get_by_name instead dev_get_by_name
 for OPT_HOST_IFACE
In-Reply-To: <20210618134244.GA16005@lst.de>
References: <20210618133956.10637-1-pkushwaha@marvell.com>
 <20210618134244.GA16005@lst.de>
Message-ID: <78d5833d-2038-811c-f6a4-2f14a3996fc5@grimberg.me>


>> dev_get_by_name() finds network device by name but it also increases
>> reference count.
>> Increasing the ref count,
>> If nvme-tcp queue is present and the network device driver is removed
>> before nvme_tcp, we will face the following continuous log:
>>    "kernel:unregister_netdevice: waiting for <eth> to become
>>    free. Usage count = 2"
>> And rmmod further halts. Similar case arises during reboot/shutdown
>> with nvme-tcp queue present and both never completes.
>>
>> As a fix we will use __dev_get_by_name() which find network device by
>> name without increasing any reference counter.
> 
> And when you remove it without the refcount we'll now have a stale
> kernel pointer?

The netdev is not actually needed, its to make sure that it exist as
reference in the host_iface afaict (the netdev is never referenced
afterwards), and I'm pretty sure there was a matching put somewhere in
the former versions of this patch.


From tglx at linutronix.de  Fri Jun 18 12:19:12 2021
From: tglx at linutronix.de (Thomas Gleixner)
Date: Fri, 18 Jun 2021 21:19:12 +0200
Subject: [patch v6 3/7] genirq/affinity: Add new callback for
 (re)calculating interrupt sets
In-Reply-To: <20210615195707.GA2909907@bjorn-Precision-5520>
References: <20210615195707.GA2909907@bjorn-Precision-5520>
Message-ID: <878s37f0b3.ffs@nanos.tec.linutronix.de>

On Tue, Jun 15 2021 at 14:57, Bjorn Helgaas wrote:
>
>> @@ -1196,6 +1196,13 @@ int pci_alloc_irq_vectors_affinity(struc
>>  	/* use legacy irq if allowed */
>>  	if (flags & PCI_IRQ_LEGACY) {
>>  		if (min_vecs == 1 && dev->irq) {
>> +			/*
>> +			 * Invoke the affinity spreading logic to ensure that
>> +			 * the device driver can adjust queue configuration
>> +			 * for the single interrupt case.
>> +			 */
>> +			if (affd)
>> +				irq_create_affinity_masks(1, affd);
>
> This looks like a leak because irq_create_affinity_masks() returns a
> pointer to kcalloc()ed space, but we throw away the pointer.
>
> Or is there something very subtle going on here, like this special
> case doesn't allocate anything?  I do see the "Nothing to assign?"
> case that returns NULL with no alloc, but it's not completely trivial
> to verify that we take that case here.

Yes, it's subtle and it's subtle crap. Sorry that I did not catch that.

Thanks,

        tglx


From tglx at linutronix.de  Fri Jun 18 12:32:28 2021
From: tglx at linutronix.de (Thomas Gleixner)
Date: Fri, 18 Jun 2021 21:32:28 +0200
Subject: [patch v6 3/7] genirq/affinity: Add new callback for
 (re)calculating interrupt sets
In-Reply-To: <YMlIbt3EPyRJHNWf@T590>
References: <20190216172228.512444498@linutronix.de>
 <20210615195707.GA2909907@bjorn-Precision-5520> <YMlIbt3EPyRJHNWf@T590>
Message-ID: <875yybezoz.ffs@nanos.tec.linutronix.de>

On Wed, Jun 16 2021 at 08:40, Ming Lei wrote:
> On Tue, Jun 15, 2021 at 02:57:07PM -0500, Bjorn Helgaas wrote:

> +static inline void irq_affinity_calc_sets_legacy(struct irq_affinity *affd)

This function name sucks because the function is really a wrapper around
irq_affinity_calc_sets(). What's so legacy about this? The fact that
it's called from the legacy PCI single interrupt code path?

> @@ -405,6 +405,30 @@ static void default_calc_sets(struct irq_affinity *affd, unsigned int affvecs)
>  	affd->set_size[0] = affvecs;
>  }
>  
> +static void irq_affinity_calc_sets(unsigned int affvecs,
> +		struct irq_affinity *affd)

Please align the arguments when you need a line break.

> +{
> +	/*
> +	 * Simple invocations do not provide a calc_sets() callback. Install
> +	 * the generic one.
> +	 */
> +	if (!affd->calc_sets)
> +		affd->calc_sets = default_calc_sets;
> +
> +	/* Recalculate the sets */
> +	affd->calc_sets(affd, affvecs);
> +
> +	WARN_ON_ONCE(affd->nr_sets > IRQ_AFFINITY_MAX_SETS);

Hrm. That function really should return an error code to tell the caller
that something went wrong.

> +}
> +
> +/* Provide a chance to call ->calc_sets for legacy */

What does this comment tell? Close to zero. 

> +void irq_affinity_calc_sets_legacy(struct irq_affinity *affd)
> +{
> +	if (!affd)
> +		return;
> +	irq_affinity_calc_sets(0, affd);
> +}

What's wrong with just exposing irq_affinity_calc_sets() have that
NULL pointer check in the function and add proper function documentation
which explains what this is about?

Thanks,

        tglx


From zhouchuangao at vivo.com  Fri Jun 18 22:01:34 2021
From: zhouchuangao at vivo.com (=?UTF-8?B?5ZGo5Lyg6auY?=)
Date: Sat, 19 Jun 2021 13:01:34 +0800 (GMT+08:00)
Subject: =?UTF-8?B?UmU6UmU6IFtQQVRDSF0gZHJpdmVycy9udm1lL2hvc3Q6IFVzZSBrb2JqX3RvX2RldigpIEFQSQ==?=
In-Reply-To: <20210618141817.GA18781@lst.de>
Message-ID: <ABYAKQDNDmr9B7YYOVwFFKqU.3.1624078894067.Hmail.zhouchuangao@vivo.com>


>On Fri, Jun 18, 2021 at 07:10:58AM -0700, zhouchuangao wrote:
>> Use kobj_to_dev() API instead of container_of().
>
>Why?  That just makes the code harder to read.

In my opinion, the kobj_to_dev() interface is provided by the kernel so that
we can get device based on kobj without having to pass more parameters.
I think it's easier to use.

Thansk,
zhouchaungao


From mingli199x at qq.com  Sat Jun 19 00:40:18 2021
From: mingli199x at qq.com (Ming.Li)
Date: Sat, 19 Jun 2021 15:40:18 +0800
Subject: [PATCH] remove all blanks in front of goto label
Message-ID: <tencent_B28F4F86583B12ADE15402FA0EBA8CE87806@qq.com>

There is a format style mistake. Some blanks in front of goto label.

Signed-off-by: Ming.Li <mingli199x at qq.com>
---
 drivers/nvme/host/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index a29b170701fc..bf3deda1561e 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2780,7 +2780,7 @@ static int nvme_dev_map(struct nvme_dev *dev)
 		goto release;
 
 	return 0;
-  release:
+release:
 	pci_release_mem_regions(pdev);
 	return -ENODEV;
 }
-- 
2.24.3 (Apple Git-128)



From kbusch at kernel.org  Sun Jun 20 10:27:54 2021
From: kbusch at kernel.org (Keith Busch)
Date: Sun, 20 Jun 2021 10:27:54 -0700
Subject: [PATCH] drivers/nvme/host: Use kobj_to_dev() API
In-Reply-To: <ABYAKQDNDmr9B7YYOVwFFKqU.3.1624078894067.Hmail.zhouchuangao@vivo.com>
References: <20210618141817.GA18781@lst.de>
 <ABYAKQDNDmr9B7YYOVwFFKqU.3.1624078894067.Hmail.zhouchuangao@vivo.com>
Message-ID: <20210620172754.GB1137891@dhcp-10-100-145-180.wdc.com>

On Sat, Jun 19, 2021 at 01:01:34PM +0800, ??? wrote:
> >On Fri, Jun 18, 2021 at 07:10:58AM -0700, zhouchuangao wrote:
> >> Use kobj_to_dev() API instead of container_of().
> >
> >Why?  That just makes the code harder to read.
> 
> In my opinion, the kobj_to_dev() interface is provided by the kernel so that
> we can get device based on kobj without having to pass more parameters.
> I think it's easier to use.

This same patch has been proposed and rejected numerous times. Do we
need to place a comment in the code to prevent future requests?

http://lists.infradead.org/pipermail/linux-nvme/2021-March/023316.html
http://lists.infradead.org/pipermail/linux-nvme/2021-February/023060.html
http://lists.infradead.org/pipermail/linux-nvme/2020-September/019462.html


From prabhakar.pkin at gmail.com  Sun Jun 20 18:43:51 2021
From: prabhakar.pkin at gmail.com (Prabhakar Kushwaha)
Date: Mon, 21 Jun 2021 07:13:51 +0530
Subject: [PATCH] nvme-tcp: Use __dev_get_by_name instead dev_get_by_name
 for OPT_HOST_IFACE
In-Reply-To: <78d5833d-2038-811c-f6a4-2f14a3996fc5@grimberg.me>
References: <20210618133956.10637-1-pkushwaha@marvell.com>
 <20210618134244.GA16005@lst.de>
 <78d5833d-2038-811c-f6a4-2f14a3996fc5@grimberg.me>
Message-ID: <CAJ2QiJLWh3_gWf8Bx8c201hGzVGcqfEhT-J_OLD=2740w1+Qsw@mail.gmail.com>

On Sat, Jun 19, 2021 at 12:44 AM Sagi Grimberg <sagi at grimberg.me> wrote:
>
>
> >> dev_get_by_name() finds network device by name but it also increases
> >> reference count.
> >> Increasing the ref count,
> >> If nvme-tcp queue is present and the network device driver is removed
> >> before nvme_tcp, we will face the following continuous log:
> >>    "kernel:unregister_netdevice: waiting for <eth> to become
> >>    free. Usage count = 2"
> >> And rmmod further halts. Similar case arises during reboot/shutdown
> >> with nvme-tcp queue present and both never completes.
> >>
> >> As a fix we will use __dev_get_by_name() which find network device by
> >> name without increasing any reference counter.
> >
> > And when you remove it without the refcount we'll now have a stale
> > kernel pointer?
>
> The netdev is not actually needed, its to make sure that it exist as
> reference in the host_iface afaict (the netdev is never referenced
> afterwards), and I'm pretty sure there was a matching put somewhere in
> the former versions of this patch.

Patch 3ede8f72a9a2 (?nvme-tcp: allow selecting the network interface for
connections") exposed to this problem in the following cases:
A)  create nvme-tcp queue, disconnect queue and remove network driver
B)  create nvme-tcp queue and remove network driver

?A? can be solved by putting dev_put() before kfree(ctrl):
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index c7bd37103cf4..93639b5b795f 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2155,6 +2155,8 @@ static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl)
        nvmf_free_options(nctrl->opts);
free_ctrl:
        kfree(ctrl->queues);
+       if (ctrl->ndev)
+               dev_put(ctrl->ndev);
        kfree(ctrl);
}

@@ -2586,6 +2588,8 @@ static struct nvme_ctrl
*nvme_tcp_create_ctrl(struct device *dev,
out_kfree_queues:
        kfree(ctrl->queues);
out_free_ctrl:
+       if (ctrl->ndev)
+               dev_put(ctrl->ndev);
        kfree(ctrl);
        return ERR_PTR(ret);
}

But even using dev_put() will not solve ?B? in case of the removal of
the network
driver before nvme-tcp.

This was our initial motivation for using  __dev_get_by_name().
It will solve both ?A? and ?B? and it will behave the same with or without the
usage of the OPT_HOST_IFACE option.
i.e. If we remove the network driver while controllers are connected it will
end with nvme timeouts.

--pk


From Chaitanya.Kulkarni at wdc.com  Sun Jun 20 19:49:33 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 21 Jun 2021 02:49:33 +0000
Subject: [PATCH] remove all blanks in front of goto label
References: <tencent_B28F4F86583B12ADE15402FA0EBA8CE87806@qq.com>
Message-ID: <BYAPR04MB4965105142E9D0EB9FBCFDB8860A9@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/19/21 00:49, Ming.Li wrote:
> There is a format style mistake. Some blanks in front of goto label.
>
> Signed-off-by: Ming.Li <mingli199x at qq.com>

Looks good.

Reviewed-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>




From chaitanya.kulkarni at wdc.com  Sun Jun 20 20:01:09 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 20 Jun 2021 20:01:09 -0700
Subject: [PATCH] nvmet: use NVMET_MAX_NAMESPACES to set nn value
Message-ID: <20210621030109.20695-1-chaitanya.kulkarni@wdc.com>

For Spec regarding MNAN value:-

If the controller supports Asymmetric Namespace Access Reporting, then
this field shall be set to a non-zero value that is less than or equal
to the NN value.

Instead of using subsys->max_nsid that gets calculated dynamically,
use NVMET_MAX_NAMESPACES value to report NN. This way we will maintain
the MNAN value spec compliant with NN.

Without this patch, code results in the following error :-

[337976.409142] nvme nvme1: Invalid MNAN value 1024

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 363e357d2f20..0cb98f2bbc8c 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -422,7 +422,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	/* no enforcement soft-limit for maxcmd - pick arbitrary high value */
 	id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
 
-	id->nn = cpu_to_le32(ctrl->subsys->max_nsid);
+	id->nn = cpu_to_le32(NVMET_MAX_NAMESPACES);
 	id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
 	id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
 			NVME_CTRL_ONCS_WRITE_ZEROES);
-- 
2.22.1



From Chaitanya.Kulkarni at wdc.com  Sun Jun 20 20:02:22 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 21 Jun 2021 03:02:22 +0000
Subject: [PATCH 2/3] nvmet: prevent creating ctrl with no namespace
References: <20210613200252.9167-1-chaitanya.kulkarni@wdc.com>
 <20210613200252.9167-3-chaitanya.kulkarni@wdc.com>
 <b6b62366-fff4-81e8-5b3b-c3062211389b@grimberg.me>
 <20210618042814.GA18021@lst.de>
Message-ID: <BYAPR04MB49652E3E4188D7FB861FE908860A9@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/17/21 21:28, Christoph Hellwig wrote:
> On Thu, Jun 17, 2021 at 12:06:11PM -0700, Sagi Grimberg wrote:
>> Why not just set both nn and mnan to NVMET_MAX_NAMESPACES and be done
>> with it?
> Yes, I'm pretty sure I suggested that before somewhere..
>

Sent out the patch with NVMET_MAX_NAMESPACES, please have a look.




From chaitanya.kulkarni at wdc.com  Sun Jun 20 20:09:40 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 20 Jun 2021 20:09:40 -0700
Subject: [PATCH] nvme: add comment to not use kobj_to_dev()
Message-ID: <20210621030940.22286-1-chaitanya.kulkarni@wdc.com>

Add a comment in nvme_ns_id_attrs_are_visible() and
nvme_dev_attrs_are_visible() not to replace container_of() with
kobj_to_dev(). Since use of kobj_to_dev() makes code harder to read.

Suggested-by: Keith Busch <kbusch at kernel.org>
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/host/core.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 14d96ee4d341..559637ebda25 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3191,6 +3191,7 @@ static struct attribute *nvme_ns_id_attrs[] = {
 static umode_t nvme_ns_id_attrs_are_visible(struct kobject *kobj,
 		struct attribute *a, int n)
 {
+	/* Don't use kobj_to_dev(), since it makes code harder to read. */
 	struct device *dev = container_of(kobj, struct device, kobj);
 	struct nvme_ns_ids *ids = &dev_to_ns_head(dev)->ids;
 
@@ -3463,6 +3464,7 @@ static struct attribute *nvme_dev_attrs[] = {
 static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		struct attribute *a, int n)
 {
+	/* Don't use kobj_to_dev(), since it makes code harder to read. */
 	struct device *dev = container_of(kobj, struct device, kobj);
 	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
 
-- 
2.22.1



From Chaitanya.Kulkarni at wdc.com  Sun Jun 20 20:11:05 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 21 Jun 2021 03:11:05 +0000
Subject: [PATCH] drivers/nvme/host: Use kobj_to_dev() API
References: <20210618141817.GA18781@lst.de>
 <ABYAKQDNDmr9B7YYOVwFFKqU.3.1624078894067.Hmail.zhouchuangao@vivo.com>
 <20210620172754.GB1137891@dhcp-10-100-145-180.wdc.com>
Message-ID: <BYAPR04MB4965F9291CC991093FD485F7860A9@BYAPR04MB4965.namprd04.prod.outlook.com>

Keith,

On 6/20/21 10:34, Keith Busch wrote:
> On Sat, Jun 19, 2021 at 01:01:34PM +0800, ??? wrote:
>>> On Fri, Jun 18, 2021 at 07:10:58AM -0700, zhouchuangao wrote:
>>>> Use kobj_to_dev() API instead of container_of().
>>> Why?  That just makes the code harder to read.
>> In my opinion, the kobj_to_dev() interface is provided by the kernel so that
>> we can get device based on kobj without having to pass more parameters.
>> I think it's easier to use.
> This same patch has been proposed and rejected numerous times. Do we
> need to place a comment in the code to prevent future requests?
>
> http://lists.infradead.org/pipermail/linux-nvme/2021-March/023316.html
> http://lists.infradead.org/pipermail/linux-nvme/2021-February/023060.html
> http://lists.infradead.org/pipermail/linux-nvme/2020-September/019462.html
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
>

Just sent a patch with your suggested-by see if that is useful.



From Chaitanya.Kulkarni at wdc.com  Sun Jun 20 20:23:34 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Mon, 21 Jun 2021 03:23:34 +0000
Subject: [PATCH] remove all blanks in front of goto label
References: <tencent_B28F4F86583B12ADE15402FA0EBA8CE87806@qq.com>
Message-ID: <BYAPR04MB49650E07DD0D902EADE35995860A9@BYAPR04MB4965.namprd04.prod.outlook.com>

Keith/Christoph,

On 6/19/21 00:49, Ming.Li wrote:
> There is a format style mistake. Some blanks in front of goto label.
>
> Signed-off-by: Ming.Li <mingli199x at qq.com>
> ---
>  drivers/nvme/host/pci.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index a29b170701fc..bf3deda1561e 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -2780,7 +2780,7 @@ static int nvme_dev_map(struct nvme_dev *dev)
>  		goto release;
>  
>  	return 0;
> -  release:
> +release:
>  	pci_release_mem_regions(pdev);
>  	return -ENODEV;
>  }
> -- 2.24.3 (Apple Git-128)

While reviewing this patch I found that on the host side there
are many places where we've spaces before goto labeldefinition,
perhaps following cleanup is useful ?


diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 559637ebda25..03c8201d15b2 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1095,7 +1095,7 @@ int __nvme_submit_sync_cmd(struct request_queue
*q, struct nvme_command *cmd,
                ret = -EINTR;
        else
                ret = nvme_req(req)->status;
- out:
+out:
        blk_mq_free_request(req);
        return ret;
 }
@@ -3997,9 +3997,9 @@ static int nvme_scan_ns_list(struct nvme_ctrl *ctrl)
                                nvme_ns_remove_by_nsid(ctrl, prev);
                }
        }
- out:
+out:
        nvme_remove_invalid_namespaces(ctrl, prev);
- free:
+free:
        kfree(ns_list);
        return ret;
 }
diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c
index d93928d1e5bd..49a9e376251c 100644
--- a/drivers/nvme/host/ioctl.c
+++ b/drivers/nvme/host/ioctl.c
@@ -105,10 +105,10 @@ static int nvme_submit_user_cmd(struct
request_queue *q,
                        ret = -EFAULT;
        }
        kfree(meta);
- out_unmap:
+out_unmap:
        if (bio)
                blk_rq_unmap_user(bio);
- out:
+out:
        blk_mq_free_request(req);
        return ret;
 }
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index d3c5086673bc..1519433d541f 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2188,7 +2188,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
        } while (1);
        adminq->q_db = dev->dbs;
 
- retry:
+retry:
        /* Deregister the admin queue's interrupt */
        pci_free_irq(pdev, 0, adminq);
 
@@ -2279,7 +2279,7 @@ static bool __nvme_disable_io_queues(struct
nvme_dev *dev, u8 opcode)
        int nr_queues = dev->online_queues - 1, sent = 0;
        unsigned long timeout;
 
- retry:
+retry:
        timeout = NVME_ADMIN_TIMEOUT;
        while (nr_queues > 0) {
                if (nvme_delete_queue(&dev->queues[nr_queues], opcode))
@@ -2428,7 +2428,7 @@ static int nvme_pci_enable(struct nvme_dev *dev)
        pci_save_state(pdev);
        return 0;
 
- disable:
+disable:
        pci_disable_device(pdev);
        return result;
 }
@@ -2702,9 +2702,9 @@ static void nvme_reset_work(struct work_struct *work)
        nvme_start_ctrl(&dev->ctrl);
        return;
 
- out_unlock:
+out_unlock:
        mutex_unlock(&dev->shutdown_lock);
- out:
+out:
        if (result)
                dev_warn(dev->ctrl.device,
                         "Removing after probe failure status: %d\n",
result);
@@ -2905,15 +2905,15 @@ static int nvme_probe(struct pci_dev *pdev,
const struct pci_device_id *id)
 
        return 0;
 
- release_mempool:
+release_mempool:
        mempool_destroy(dev->iod_mempool);
- release_pools:
+release_pools:
        nvme_release_prp_pools(dev);
- unmap:
+unmap:
        nvme_dev_unmap(dev);
  put_pci:
        put_device(dev->dev);
- free:
+free:
        kfree(dev->queues);
        kfree(dev);
        return result;



From smalin at marvell.com  Sun Jun 20 22:10:38 2021
From: smalin at marvell.com (Shai Malin)
Date: Mon, 21 Jun 2021 08:10:38 +0300
Subject: [PATCH v2 0/8] NVMeTCP Offload ULP
Message-ID: <20210621051046.28783-1-smalin@marvell.com>

With the goal of enabling a generic infrastructure that allows NVMe/TCP
offload devices like NICs to seamlessly plug into the NVMe-oF stack, this
patch series introduces the nvme-tcp-offload ULP host layer, which will
be a new transport type called "tcp-offload" and will serve as an 
abstraction layer to work with vendor specific nvme-tcp offload drivers.

NVMeTCP offload is a full offload of the NVMeTCP protocol, this includes 
both the TCP level and the NVMeTCP level.

The nvme-tcp-offload transport can co-exist with the existing tcp and 
other transports. The tcp offload was designed so that stack changes are 
kept to a bare minimum: only registering new transports.
All other APIs, ops etc. are identical to the regular tcp transport.
Representing the TCP offload as a new transport allows clear and manageable
differentiation between the connections which should use the offload path 
and those that are not offloaded (even on the same device).

The nvme-tcp-offload layers and API compared to nvme-tcp and nvme-rdma:

* NVMe layer: *

       [ nvme/nvme-fabrics/blk-mq ]
             |
        (nvme API and blk-mq API)
             |
             |			 
* Vendor agnostic transport layer: *

      [ nvme-rdma ] [ nvme-tcp ] [ nvme-tcp-offload ]
             |        |             |
           (Verbs) 
             |        |             |
             |     (Socket)
             |        |             |
             |        |        (nvme-tcp-offload API)
             |        |             |
             |        |             |
* Vendor Specific Driver: *

             |        |             |
      [ RDMA driver ]       
                      |             |
             [ Network driver ]
                                    |
                       [ NVMeTCP Offload driver ]


Usage:
======
The user will interact with the network-device in order to configure 
the ip/vlan - Logically similar to the RDMA model.
The NVMeTCP configuration is populated as part of the 
nvme connect command.

Example:
Assign IP to the net-device (from any existing Linux tool):

    ip addr add 100.100.0.101/24 dev p1p1

This IP will be used by both net-device and offload-device.

In order to connect from "sw" nvme-tcp through the net-device:

    nvme connect -t tcp -s 4420 -a 100.100.0.100 -n testnqn

In order to connect from "offload" nvme-tcp through the offload-device:

    nvme connect -t tcp_offload -s 4420 -a 100.100.0.100 -n testnqn
	
An alternative approach, and as a future enhancement that will not impact this 
series will be to modify nvme-cli with a new flag that will determine 
if "-t tcp" should be the regular nvme-tcp (which will be the default) 
or nvme-tcp-offload.
Exmaple:
    nvme connect -t tcp -s 4420 -a 100.100.0.100 -n testnqn -[new flag]


Queue Initialization Design:
============================
The nvme-tcp-offload ULP module shall register with the existing 
nvmf_transport_ops (.name = "tcp_offload"), nvme_ctrl_ops and blk_mq_ops.
The nvme-tcp-offload vendor driver shall register to nvme-tcp-offload ULP
with the following ops:
- claim_dev() - in order to resolve the route to the target according to
                the paired net_dev.
- create_queue() - in order to create offloaded nvme-tcp queue.

The nvme-tcp-offload ULP module shall manage all the controller level
functionalities, call claim_dev and based on the return values shall call
the relevant module create_queue in order to create the admin queue and
the IO queues.


IO-path Design:
===============
The nvme-tcp-offload shall work at the IO-level - the nvme-tcp-offload 
ULP module shall pass the request (the IO) to the nvme-tcp-offload vendor
driver and later, the nvme-tcp-offload vendor driver returns the request
completion (the IO completion).
No additional handling is needed in between; this design will reduce the
CPU utilization as we will describe below.

The nvme-tcp-offload vendor driver shall register to nvme-tcp-offload ULP
with the following IO-path ops:
- send_req() - in order to pass the request to the handling of the
               offload driver that shall pass it to the vendor specific device.
- poll_queue()

Once the IO completes, the nvme-tcp-offload vendor driver shall call 
command.done() that will invoke the nvme-tcp-offload ULP layer to
complete the request.


Teardown and errors:
====================
In case of NVMeTCP queue error the nvme-tcp-offload vendor driver shall
call the nvme_tcp_ofld_report_queue_err.
The nvme-tcp-offload vendor driver shall register to nvme-tcp-offload ULP
with the following teardown ops:
- drain_queue()
- destroy_queue()


Changes since RFC v1:
=====================
- nvme-tcp-offload: Fix nvme_tcp_ofld_ops return values.
- nvme-tcp-offload: Remove NVMF_TRTYPE_TCP_OFFLOAD.
- nvme-tcp-offload: Add nvme_tcp_ofld_poll() implementation.
- nvme-tcp-offload: Fix nvme_tcp_ofld_queue_rq() to check map_sg() and 
  send_req() return values.

Changes since RFC v2:
=====================
- nvme-tcp-offload: Fixes in controller and queue level (patches 3-6).
- qedn: Add the Marvell's NVMeTCP HW offload vendor driver init and probe
  (patches 8-11).
  
Changes since RFC v3:
=====================
- nvme-tcp-offload: Add the full implementation of the nvme-tcp-offload layer 
  including the new ops: setup_ctrl(), release_ctrl(), commit_rqs() and new 
  flows (ASYNC and timeout).
- nvme-tcp-offload: Add device maximums: max_hw_sectors, max_segments.
- nvme-tcp-offload: layer design and optimization changes.

Changes since RFC v4:
=====================
(Many thanks to Hannes Reinecke for his feedback)
- nvme_tcp_offload: Add num_hw_vectors in order to limit the number of queues.
- nvme_tcp_offload: Add per device private_data.
- nvme_tcp_offload: Fix header digest, data digest and tos initialization.

Changes since RFC v5:
=====================
(Many thanks to Sagi Grimberg for his feedback)
- nvme-fabrics: Expose nvmf_check_required_opts() globally (as a new patch).
- nvme_tcp_offload: Remove io-queues BLK_MQ_F_BLOCKING.
- nvme_tcp_offload: Fix the nvme_tcp_ofld_stop_queue (drain_queue) flow.
- nvme_tcp_offload: Fix the nvme_tcp_ofld_free_queue (destroy_queue) flow.
- nvme_tcp_offload: Change rwsem to mutex.
- nvme_tcp_offload: remove redundant fields.
- nvme_tcp_offload: Remove the "new" from setup_ctrl().
- nvme_tcp_offload: Remove the init_req() and commit_rqs() ops.
- nvme_tcp_offload: Minor fixes in nvme_tcp_ofld_create_ctrl() ansd 
  nvme_tcp_ofld_free_queue().
- nvme_tcp_offload: Patch 8 (timeout and async) was squeashed into 
  patch 7 (io level).

Changes since RFC v6:
=====================
- No changes in nvme_tcp_offload (only in qedn).

Changes since v0:
=====================
- nvme_tcp_offload: Add support for NVME_OPT_HOST_IFACE.
- nvme_tcp_offload: Kconfig fix (thanks to Petr Mladek).
- nvme_tcp_offload: return code fix (thanks to Dan Carpenter).


Arie Gershberg (2):
  nvme-tcp-offload: Add controller level implementation
  nvme-tcp-offload: Add controller level error recovery implementation

Dean Balandin (3):
  nvme-tcp-offload: Add device scan implementation
  nvme-tcp-offload: Add queue level implementation
  nvme-tcp-offload: Add IO level implementation

Prabhakar Kushwaha (2):
  nvme-fabrics: Move NVMF_ALLOWED_OPTS and NVMF_REQUIRED_OPTS
    definitions
  nvme-fabrics: Expose nvmf_check_required_opts() globally

Shai Malin (1):
  nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW offload ULP

 MAINTAINERS                     |    8 +
 drivers/nvme/host/Kconfig       |   15 +
 drivers/nvme/host/Makefile      |    3 +
 drivers/nvme/host/fabrics.c     |   12 +-
 drivers/nvme/host/fabrics.h     |    9 +
 drivers/nvme/host/tcp-offload.c | 1333 +++++++++++++++++++++++++++++++
 drivers/nvme/host/tcp-offload.h |  207 +++++
 7 files changed, 1578 insertions(+), 9 deletions(-)
 create mode 100644 drivers/nvme/host/tcp-offload.c
 create mode 100644 drivers/nvme/host/tcp-offload.h

-- 
2.22.0



From smalin at marvell.com  Sun Jun 20 22:10:41 2021
From: smalin at marvell.com (Shai Malin)
Date: Mon, 21 Jun 2021 08:10:41 +0300
Subject: [PATCH v2 3/8] nvme-fabrics: Expose nvmf_check_required_opts()
 globally
In-Reply-To: <20210621051046.28783-1-smalin@marvell.com>
References: <20210621051046.28783-1-smalin@marvell.com>
Message-ID: <20210621051046.28783-4-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

nvmf_check_required_opts() is used to check if user provided opts has
the required_opts or not. if not, it will log which options are not
provided.

It can be leveraged by nvme-tcp-offload to check if provided opts are
supported by this specific vendor driver or not.

So expose nvmf_check_required_opts() globally.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/fabrics.c | 5 +++--
 drivers/nvme/host/fabrics.h | 2 ++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 21d67775d2af..7830d2540492 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -864,8 +864,8 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	return ret;
 }
 
-static int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
-		unsigned int required_opts)
+int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
+			     unsigned int required_opts)
 {
 	if ((opts->mask & required_opts) != required_opts) {
 		int i;
@@ -883,6 +883,7 @@ static int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(nvmf_check_required_opts);
 
 bool nvmf_ip_options_match(struct nvme_ctrl *ctrl,
 		struct nvmf_ctrl_options *opts)
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 38ac7b757d78..15d9c15ef8a6 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -197,5 +197,7 @@ int nvmf_get_address(struct nvme_ctrl *ctrl, char *buf, int size);
 bool nvmf_should_reconnect(struct nvme_ctrl *ctrl);
 bool nvmf_ip_options_match(struct nvme_ctrl *ctrl,
 		struct nvmf_ctrl_options *opts);
+int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
+			     unsigned int required_opts);
 
 #endif /* _NVME_FABRICS_H */
-- 
2.22.0



From smalin at marvell.com  Sun Jun 20 22:10:39 2021
From: smalin at marvell.com (Shai Malin)
Date: Mon, 21 Jun 2021 08:10:39 +0300
Subject: [PATCH v2 1/8] nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW
 offload ULP
In-Reply-To: <20210621051046.28783-1-smalin@marvell.com>
References: <20210621051046.28783-1-smalin@marvell.com>
Message-ID: <20210621051046.28783-2-smalin@marvell.com>

This patch will present the structure for the NVMeTCP offload common
layer driver. This module is added under "drivers/nvme/host/" and future
offload drivers which will register to it will be placed under
"drivers/nvme/hw".
This new driver will be enabled by the Kconfig "NVM Express over Fabrics
TCP offload commmon layer".
In order to support the new transport type, for host mode, no change is
needed.

Each new vendor-specific offload driver will register to this ULP during
its probe function, by filling out the nvme_tcp_ofld_dev->ops and
nvme_tcp_ofld_dev->private_data and calling nvme_tcp_ofld_register_dev
with the initialized struct.

The internal implementation:
- tcp-offload.h:
  Includes all common structs and ops to be used and shared by offload
  drivers.

- tcp-offload.c:
  Includes the init function which registers as a NVMf transport just
  like any other transport.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 MAINTAINERS                     |   8 ++
 drivers/nvme/host/Kconfig       |  15 +++
 drivers/nvme/host/Makefile      |   3 +
 drivers/nvme/host/tcp-offload.c | 124 ++++++++++++++++++++
 drivers/nvme/host/tcp-offload.h | 199 ++++++++++++++++++++++++++++++++
 5 files changed, 349 insertions(+)
 create mode 100644 drivers/nvme/host/tcp-offload.c
 create mode 100644 drivers/nvme/host/tcp-offload.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 81e1edeceae4..01fbebdc7722 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13093,6 +13093,14 @@ F:	drivers/nvme/host/
 F:	include/linux/nvme.h
 F:	include/uapi/linux/nvme_ioctl.h
 
+NVM EXPRESS TCP OFFLOAD TRANSPORT DRIVERS
+M:	Shai Malin <smalin at marvell.com>
+M:	Ariel Elior <aelior at marvell.com>
+L:	linux-nvme at lists.infradead.org
+S:	Supported
+F:	drivers/nvme/host/tcp-offload.c
+F:	drivers/nvme/host/tcp-offload.h
+
 NVM EXPRESS FC TRANSPORT DRIVERS
 M:	James Smart <james.smart at broadcom.com>
 L:	linux-nvme at lists.infradead.org
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 102292289cdf..1993734d0104 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -84,3 +84,18 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_TCP_OFFLOAD
+	tristate "NVM Express over Fabrics TCP offload common layer"
+	depends on INET
+	depends on BLK_DEV_NVME
+	select NVME_FABRICS
+	help
+	  This provides support for the NVMe over Fabrics protocol using
+	  the TCP offload transport. This allows you to use remote block devices
+	  exported using the NVMe protocol set.
+
+	  To configure a NVMe over Fabrics controller use the nvme-cli tool
+	  from https://github.com/linux-nvme/nvme-cli.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index cbc509784b2e..3c3fdf83ce38 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -8,6 +8,7 @@ obj-$(CONFIG_NVME_FABRICS)		+= nvme-fabrics.o
 obj-$(CONFIG_NVME_RDMA)			+= nvme-rdma.o
 obj-$(CONFIG_NVME_FC)			+= nvme-fc.o
 obj-$(CONFIG_NVME_TCP)			+= nvme-tcp.o
+obj-$(CONFIG_NVME_TCP_OFFLOAD)	+= nvme-tcp-offload.o
 
 nvme-core-y				:= core.o ioctl.o
 nvme-core-$(CONFIG_TRACING)		+= trace.o
@@ -26,3 +27,5 @@ nvme-rdma-y				+= rdma.o
 nvme-fc-y				+= fc.o
 
 nvme-tcp-y				+= tcp.o
+
+nvme-tcp-offload-y		+= tcp-offload.o
diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
new file mode 100644
index 000000000000..b9b6ffa10747
--- /dev/null
+++ b/drivers/nvme/host/tcp-offload.c
@@ -0,0 +1,124 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+/* Kernel includes */
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+/* Driver includes */
+#include "tcp-offload.h"
+
+static LIST_HEAD(nvme_tcp_ofld_devices);
+static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
+
+/**
+ * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
+ * function.
+ * @dev:	NVMeTCP offload device instance to be registered to the
+ *		common tcp offload instance.
+ *
+ * API function that registers the type of vendor specific driver
+ * being implemented to the common NVMe over TCP offload library. Part of
+ * the overall init sequence of starting up an offload driver.
+ */
+int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev)
+{
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+
+	if (!ops->claim_dev ||
+	    !ops->setup_ctrl ||
+	    !ops->release_ctrl ||
+	    !ops->create_queue ||
+	    !ops->drain_queue ||
+	    !ops->destroy_queue ||
+	    !ops->poll_queue ||
+	    !ops->send_req)
+		return -EINVAL;
+
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_add_tail(&dev->entry, &nvme_tcp_ofld_devices);
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_register_dev);
+
+/**
+ * nvme_tcp_ofld_unregister_dev() - NVMeTCP Offload Library unregistration
+ * function.
+ * @dev:	NVMeTCP offload device instance to be unregistered from the
+ *		common tcp offload instance.
+ *
+ * API function that unregisters the type of vendor specific driver being
+ * implemented from the common NVMe over TCP offload library.
+ * Part of the overall exit sequence of unloading the implemented driver.
+ */
+void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev)
+{
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_del(&dev->entry);
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
+
+/**
+ * nvme_tcp_ofld_report_queue_err() - NVMeTCP Offload report error event
+ * callback function. Pointed to by nvme_tcp_ofld_queue->report_err.
+ * @queue:	NVMeTCP offload queue instance on which the error has occurred.
+ *
+ * API function that allows the vendor specific offload driver to reports errors
+ * to the common offload layer, to invoke error recovery.
+ */
+int nvme_tcp_ofld_report_queue_err(struct nvme_tcp_ofld_queue *queue)
+{
+	/* Placeholder - invoke error recovery flow */
+
+	return 0;
+}
+
+/**
+ * nvme_tcp_ofld_req_done() - NVMeTCP Offload request done callback
+ * function. Pointed to by nvme_tcp_ofld_req->done.
+ * Handles both NVME_TCP_F_DATA_SUCCESS flag and NVMe CQ.
+ * @req:	NVMeTCP offload request to complete.
+ * @result:     The nvme_result.
+ * @status:     The completion status.
+ *
+ * API function that allows the vendor specific offload driver to report request
+ * completions to the common offload layer.
+ */
+void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
+			    union nvme_result *result,
+			    __le16 status)
+{
+	/* Placeholder - complete request with/without error */
+}
+
+static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
+	.name		= "tcp_offload",
+	.module		= THIS_MODULE,
+	.required_opts	= NVMF_OPT_TRADDR,
+	.allowed_opts	= NVMF_OPT_TRSVCID | NVMF_OPT_NR_WRITE_QUEUES  |
+			  NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO |
+			  NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HDR_DIGEST |
+			  NVMF_OPT_DATA_DIGEST | NVMF_OPT_NR_POLL_QUEUES |
+			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE,
+};
+
+static int __init nvme_tcp_ofld_init_module(void)
+{
+	nvmf_register_transport(&nvme_tcp_ofld_transport);
+
+	return 0;
+}
+
+static void __exit nvme_tcp_ofld_cleanup_module(void)
+{
+	nvmf_unregister_transport(&nvme_tcp_ofld_transport);
+}
+
+module_init(nvme_tcp_ofld_init_module);
+module_exit(nvme_tcp_ofld_cleanup_module);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
new file mode 100644
index 000000000000..4964e18a5885
--- /dev/null
+++ b/drivers/nvme/host/tcp-offload.h
@@ -0,0 +1,199 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+/* Linux includes */
+#include <linux/dma-mapping.h>
+#include <linux/scatterlist.h>
+#include <linux/types.h>
+#include <linux/nvme-tcp.h>
+
+/* Driver includes */
+#include "nvme.h"
+#include "fabrics.h"
+
+/* Forward declarations */
+struct nvme_tcp_ofld_ops;
+
+/* Representation of a vendor-specific device. This is the struct used to
+ * register to the offload layer by the vendor-specific driver during its probe
+ * function.
+ * Allocated by vendor-specific driver.
+ */
+struct nvme_tcp_ofld_dev {
+	struct list_head entry;
+	struct nvme_tcp_ofld_ops *ops;
+
+	/* Vendor specific driver context */
+	int num_hw_vectors;
+};
+
+/* Per IO struct holding the nvme_request and command
+ * Allocated by blk-mq.
+ */
+struct nvme_tcp_ofld_req {
+	struct nvme_request req;
+	struct nvme_command nvme_cmd;
+	struct list_head queue_entry;
+	struct nvme_tcp_ofld_queue *queue;
+
+	/* Vendor specific driver context */
+	void *private_data;
+
+	/* async flag is used to distinguish between async and IO flow
+	 * in common send_req() of nvme_tcp_ofld_ops.
+	 */
+	bool async;
+
+	void (*done)(struct nvme_tcp_ofld_req *req,
+		     union nvme_result *result,
+		     __le16 status);
+};
+
+enum nvme_tcp_ofld_queue_flags {
+	NVME_TCP_OFLD_Q_ALLOCATED = 0,
+	NVME_TCP_OFLD_Q_LIVE = 1,
+};
+
+/* Allocated by nvme_tcp_ofld */
+struct nvme_tcp_ofld_queue {
+	/* Offload device associated to this queue */
+	struct nvme_tcp_ofld_dev *dev;
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	unsigned long flags;
+	size_t cmnd_capsule_len;
+
+	u8 hdr_digest;
+	u8 data_digest;
+	u8 tos;
+
+	/* Vendor specific driver context */
+	void *private_data;
+
+	/* Error callback function */
+	int (*report_err)(struct nvme_tcp_ofld_queue *queue);
+};
+
+/* Connectivity (routing) params used for establishing a connection */
+struct nvme_tcp_ofld_ctrl_con_params {
+	struct sockaddr_storage remote_ip_addr;
+
+	/* If NVMF_OPT_HOST_TRADDR is provided it will be set in local_ip_addr
+	 * in nvme_tcp_ofld_create_ctrl().
+	 * If NVMF_OPT_HOST_TRADDR is not provided the local_ip_addr will be
+	 * initialized by claim_dev().
+	 */
+	struct sockaddr_storage local_ip_addr;
+};
+
+/* Allocated by nvme_tcp_ofld */
+struct nvme_tcp_ofld_ctrl {
+	struct nvme_ctrl nctrl;
+	struct list_head list;
+	struct net_device *ndev;
+	struct nvme_tcp_ofld_dev *dev;
+
+	/* admin and IO queues */
+	struct blk_mq_tag_set tag_set;
+	struct blk_mq_tag_set admin_tag_set;
+	struct nvme_tcp_ofld_queue *queues;
+
+	struct work_struct err_work;
+	struct delayed_work connect_work;
+
+	/*
+	 * Each entry in the array indicates the number of queues of
+	 * corresponding type.
+	 */
+	u32 io_queues[HCTX_MAX_TYPES];
+
+	/* Connectivity params */
+	struct nvme_tcp_ofld_ctrl_con_params conn_params;
+
+	/* Vendor specific driver context */
+	void *private_data;
+};
+
+struct nvme_tcp_ofld_ops {
+	const char *name;
+	struct module *module;
+
+	/* For vendor-specific driver to report what opts it supports.
+	 * It could be different than the ULP supported opts due to hardware
+	 * limitations. Also it could be different among different vendor
+	 * drivers.
+	 */
+	int required_opts; /* bitmap using enum nvmf_parsing_opts */
+	int allowed_opts; /* bitmap using enum nvmf_parsing_opts */
+
+	/* For vendor-specific max num of segments and IO sizes */
+	u32 max_hw_sectors;
+	u32 max_segments;
+
+	/**
+	 * claim_dev: Return True if addr is reachable via offload device.
+	 * @dev: The offload device to check.
+	 * @ctrl: The offload ctrl have the conn_params field. The
+	 * conn_params is to be filled with routing params by the lower
+	 * driver.
+	 */
+	int (*claim_dev)(struct nvme_tcp_ofld_dev *dev,
+			 struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * setup_ctrl: Setup device specific controller structures.
+	 * @ctrl: The offload ctrl.
+	 */
+	int (*setup_ctrl)(struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * release_ctrl: Release/Free device specific controller structures.
+	 * @ctrl: The offload ctrl.
+	 */
+	int (*release_ctrl)(struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * create_queue: Create offload queue and establish TCP + NVMeTCP
+	 * (icreq+icresp) connection. Return true on successful connection.
+	 * Based on nvme_tcp_alloc_queue.
+	 * @queue: The queue itself - used as input and output.
+	 * @qid: The queue ID associated with the requested queue.
+	 * @q_size: The queue depth.
+	 */
+	int (*create_queue)(struct nvme_tcp_ofld_queue *queue, int qid,
+			    size_t queue_size);
+
+	/**
+	 * drain_queue: Drain a given queue - blocking function call.
+	 * Return from this function ensures that no additional
+	 * completions will arrive on this queue and that the HW will
+	 * not access host memory.
+	 * @queue: The queue to drain.
+	 */
+	void (*drain_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * destroy_queue: Close the TCP + NVMeTCP connection of a given queue
+	 * and make sure its no longer active (no completions will arrive on the
+	 * queue).
+	 * @queue: The queue to destroy.
+	 */
+	void (*destroy_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * poll_queue: Poll a given queue for completions.
+	 * @queue: The queue to poll.
+	 */
+	int (*poll_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * send_req: Dispatch a request. Returns the execution status.
+	 * @req: Ptr to request to be sent.
+	 */
+	int (*send_req)(struct nvme_tcp_ofld_req *req);
+};
+
+/* Exported functions for lower vendor specific offload drivers */
+int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
+void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
-- 
2.22.0



From smalin at marvell.com  Sun Jun 20 22:10:40 2021
From: smalin at marvell.com (Shai Malin)
Date: Mon, 21 Jun 2021 08:10:40 +0300
Subject: [PATCH v2 2/8] nvme-fabrics: Move NVMF_ALLOWED_OPTS and
 NVMF_REQUIRED_OPTS definitions
In-Reply-To: <20210621051046.28783-1-smalin@marvell.com>
References: <20210621051046.28783-1-smalin@marvell.com>
Message-ID: <20210621051046.28783-3-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

Move NVMF_ALLOWED_OPTS and NVMF_REQUIRED_OPTS definitions
to header file, so it can be used by the different HW devices.

NVMeTCP offload devices might have different limitations of the
allowed options, for example, a device that does not support all the
queue types. With tcp and rdma, only the nvme-tcp and nvme-rdma layers
handle those attributes and the HW devices do not create any limitations
for the allowed options.

An alternative design could be to add separate fields in
nvme_tcp_ofld_ops such as max_hw_sectors and max_segments that
we already have in this series.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Acked-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/fabrics.c | 7 -------
 drivers/nvme/host/fabrics.h | 7 +++++++
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 1e6a7cc056ca..21d67775d2af 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -947,13 +947,6 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
 
-#define NVMF_REQUIRED_OPTS	(NVMF_OPT_TRANSPORT | NVMF_OPT_NQN)
-#define NVMF_ALLOWED_OPTS	(NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \
-				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
-				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
-				 NVMF_OPT_DISABLE_SQFLOW |\
-				 NVMF_OPT_FAIL_FAST_TMO)
-
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
 {
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index c31dad69a773..38ac7b757d78 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -69,6 +69,13 @@ enum {
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 };
 
+#define NVMF_REQUIRED_OPTS	(NVMF_OPT_TRANSPORT | NVMF_OPT_NQN)
+#define NVMF_ALLOWED_OPTS	(NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \
+				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
+				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
+				 NVMF_OPT_DISABLE_SQFLOW |\
+				 NVMF_OPT_FAIL_FAST_TMO)
+
 /**
  * struct nvmf_ctrl_options - Used to hold the options specified
  *			      with the parsing opts enum.
-- 
2.22.0



From smalin at marvell.com  Sun Jun 20 22:10:43 2021
From: smalin at marvell.com (Shai Malin)
Date: Mon, 21 Jun 2021 08:10:43 +0300
Subject: [PATCH v2 5/8] nvme-tcp-offload: Add controller level implementation
In-Reply-To: <20210621051046.28783-1-smalin@marvell.com>
References: <20210621051046.28783-1-smalin@marvell.com>
Message-ID: <20210621051046.28783-6-smalin@marvell.com>

From: Arie Gershberg <agershberg at marvell.com>

In this patch we implement controller level functionality including:
- create_ctrl.
- delete_ctrl.
- free_ctrl.

The implementation is similar to other nvme fabrics modules, the main
difference being that the nvme-tcp-offload ULP calls the vendor specific
claim_dev() op with the given TCP/IP parameters to determine which device
will be used for this controller.
Once found, the vendor specific device and controller will be paired and
kept in a controller list managed by the ULP.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/tcp-offload.c | 482 +++++++++++++++++++++++++++++++-
 1 file changed, 477 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 15f018e45c60..ef7a0b89de9d 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -12,6 +12,10 @@
 
 static LIST_HEAD(nvme_tcp_ofld_devices);
 static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
+static LIST_HEAD(nvme_tcp_ofld_ctrl_list);
+static DEFINE_MUTEX(nvme_tcp_ofld_ctrl_mutex);
+static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops;
+static struct blk_mq_ops nvme_tcp_ofld_mq_ops;
 
 static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nctrl)
 {
@@ -133,21 +137,440 @@ nvme_tcp_ofld_lookup_dev(struct nvme_tcp_ofld_ctrl *ctrl)
 	return dev;
 }
 
+static struct blk_mq_tag_set *
+nvme_tcp_ofld_alloc_tagset(struct nvme_ctrl *nctrl, bool admin)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct blk_mq_tag_set *set;
+	int rc;
+
+	if (admin) {
+		set = &ctrl->admin_tag_set;
+		memset(set, 0, sizeof(*set));
+		set->ops = &nvme_tcp_ofld_admin_mq_ops;
+		set->queue_depth = NVME_AQ_MQ_TAG_DEPTH;
+		set->reserved_tags = NVMF_RESERVED_TAGS;
+		set->numa_node = nctrl->numa_node;
+		set->flags = BLK_MQ_F_BLOCKING;
+		set->cmd_size = sizeof(struct nvme_tcp_ofld_req);
+		set->driver_data = ctrl;
+		set->nr_hw_queues = 1;
+		set->timeout = NVME_ADMIN_TIMEOUT;
+	} else {
+		set = &ctrl->tag_set;
+		memset(set, 0, sizeof(*set));
+		set->ops = &nvme_tcp_ofld_mq_ops;
+		set->queue_depth = nctrl->sqsize + 1;
+		set->reserved_tags = NVMF_RESERVED_TAGS;
+		set->numa_node = nctrl->numa_node;
+		set->flags = BLK_MQ_F_SHOULD_MERGE;
+		set->cmd_size = sizeof(struct nvme_tcp_ofld_req);
+		set->driver_data = ctrl;
+		set->nr_hw_queues = nctrl->queue_count - 1;
+		set->timeout = NVME_IO_TIMEOUT;
+		set->nr_maps = nctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2;
+	}
+
+	rc = blk_mq_alloc_tag_set(set);
+	if (rc)
+		return ERR_PTR(rc);
+
+	return set;
+}
+
+static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
+					       bool new)
+{
+	int rc;
+
+	/* Placeholder - alloc_admin_queue */
+	if (new) {
+		nctrl->admin_tagset =
+				nvme_tcp_ofld_alloc_tagset(nctrl, true);
+		if (IS_ERR(nctrl->admin_tagset)) {
+			rc = PTR_ERR(nctrl->admin_tagset);
+			nctrl->admin_tagset = NULL;
+			goto out_destroy_queue;
+		}
+
+		nctrl->fabrics_q = blk_mq_init_queue(nctrl->admin_tagset);
+		if (IS_ERR(nctrl->fabrics_q)) {
+			rc = PTR_ERR(nctrl->fabrics_q);
+			nctrl->fabrics_q = NULL;
+			goto out_free_tagset;
+		}
+
+		nctrl->admin_q = blk_mq_init_queue(nctrl->admin_tagset);
+		if (IS_ERR(nctrl->admin_q)) {
+			rc = PTR_ERR(nctrl->admin_q);
+			nctrl->admin_q = NULL;
+			goto out_cleanup_fabrics_q;
+		}
+	}
+
+	/* Placeholder - nvme_tcp_ofld_start_queue */
+
+	rc = nvme_enable_ctrl(nctrl);
+	if (rc)
+		goto out_stop_queue;
+
+	blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	rc = nvme_init_ctrl_finish(nctrl);
+	if (rc)
+		goto out_quiesce_queue;
+
+	return 0;
+
+out_quiesce_queue:
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+
+out_stop_queue:
+	/* Placeholder - stop offload queue */
+	nvme_cancel_admin_tagset(nctrl);
+
+out_cleanup_fabrics_q:
+	if (new)
+		blk_cleanup_queue(nctrl->fabrics_q);
+out_free_tagset:
+	if (new)
+		blk_mq_free_tag_set(nctrl->admin_tagset);
+out_destroy_queue:
+	/* Placeholder - free admin queue */
+
+	return rc;
+}
+
+static int
+nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
+{
+	int rc;
+
+	/* Placeholder - alloc_io_queues */
+
+	if (new) {
+		nctrl->tagset = nvme_tcp_ofld_alloc_tagset(nctrl, false);
+		if (IS_ERR(nctrl->tagset)) {
+			rc = PTR_ERR(nctrl->tagset);
+			nctrl->tagset = NULL;
+			goto out_free_io_queues;
+		}
+
+		nctrl->connect_q = blk_mq_init_queue(nctrl->tagset);
+		if (IS_ERR(nctrl->connect_q)) {
+			rc = PTR_ERR(nctrl->connect_q);
+			nctrl->connect_q = NULL;
+			goto out_free_tag_set;
+		}
+	}
+
+	/* Placeholder - start_io_queues */
+
+	if (!new) {
+		nvme_start_queues(nctrl);
+		if (!nvme_wait_freeze_timeout(nctrl, NVME_IO_TIMEOUT)) {
+			/*
+			 * If we timed out waiting for freeze we are likely to
+			 * be stuck.  Fail the controller initialization just
+			 * to be safe.
+			 */
+			rc = -ENODEV;
+			goto out_wait_freeze_timed_out;
+		}
+		blk_mq_update_nr_hw_queues(nctrl->tagset, nctrl->queue_count - 1);
+		nvme_unfreeze(nctrl);
+	}
+
+	return 0;
+
+out_wait_freeze_timed_out:
+	nvme_stop_queues(nctrl);
+	nvme_sync_io_queues(nctrl);
+
+	/* Placeholder - Stop IO queues */
+
+	if (new)
+		blk_cleanup_queue(nctrl->connect_q);
+out_free_tag_set:
+	if (new)
+		blk_mq_free_tag_set(nctrl->tagset);
+out_free_io_queues:
+	/* Placeholder - free_io_queues */
+
+	return rc;
+}
+
+static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvmf_ctrl_options *opts = nctrl->opts;
+	int rc = 0;
+
+	rc = ctrl->dev->ops->setup_ctrl(ctrl);
+	if (rc)
+		return rc;
+
+	rc = nvme_tcp_ofld_configure_admin_queue(nctrl, new);
+	if (rc)
+		goto out_release_ctrl;
+
+	if (nctrl->icdoff) {
+		dev_err(nctrl->device, "icdoff is not supported!\n");
+		rc = -EOPNOTSUPP;
+		goto destroy_admin;
+	}
+
+	if (!nvme_ctrl_sgl_supported(nctrl)) {
+		dev_err(nctrl->device, "Mandatory sgls are not supported!\n");
+		rc = -EOPNOTSUPP;
+		goto destroy_admin;
+	}
+
+	if (opts->queue_size > nctrl->sqsize + 1)
+		dev_warn(nctrl->device,
+			 "queue_size %zu > ctrl sqsize %u, clamping down\n",
+			 opts->queue_size, nctrl->sqsize + 1);
+
+	if (nctrl->sqsize + 1 > nctrl->maxcmd) {
+		dev_warn(nctrl->device,
+			 "sqsize %u > ctrl maxcmd %u, clamping down\n",
+			 nctrl->sqsize + 1, nctrl->maxcmd);
+		nctrl->sqsize = nctrl->maxcmd - 1;
+	}
+
+	if (nctrl->queue_count > 1) {
+		rc = nvme_tcp_ofld_configure_io_queues(nctrl, new);
+		if (rc)
+			goto destroy_admin;
+	}
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_LIVE)) {
+		/*
+		 * state change failure is ok if we started ctrl delete,
+		 * unless we're during creation of a new controller to
+		 * avoid races with teardown flow.
+		 */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+		WARN_ON_ONCE(new);
+		rc = -EINVAL;
+		goto destroy_io;
+	}
+
+	nvme_start_ctrl(nctrl);
+
+	return 0;
+
+destroy_io:
+	/* Placeholder - stop and destroy io queues*/
+destroy_admin:
+	/* Placeholder - stop and destroy admin queue*/
+out_release_ctrl:
+	ctrl->dev->ops->release_ctrl(ctrl);
+
+	return rc;
+}
+
+static int
+nvme_tcp_ofld_check_dev_opts(struct nvmf_ctrl_options *opts,
+			     struct nvme_tcp_ofld_ops *ofld_ops)
+{
+	unsigned int nvme_tcp_ofld_opt_mask = NVMF_ALLOWED_OPTS |
+			ofld_ops->allowed_opts | ofld_ops->required_opts;
+	struct nvmf_ctrl_options dev_opts_mask;
+
+	if (opts->mask & ~nvme_tcp_ofld_opt_mask) {
+		pr_warn("One or more nvmf options missing from ofld drvr %s.\n",
+			ofld_ops->name);
+
+		dev_opts_mask.mask = nvme_tcp_ofld_opt_mask;
+
+		return nvmf_check_required_opts(&dev_opts_mask, opts->mask);
+	}
+
+	return 0;
+}
+
+static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+
+	if (list_empty(&ctrl->list))
+		goto free_ctrl;
+
+	ctrl->dev->ops->release_ctrl(ctrl);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_del(&ctrl->list);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+
+	nvmf_free_options(nctrl->opts);
+free_ctrl:
+	module_put(dev->ops->module);
+	kfree(ctrl->queues);
+	kfree(ctrl);
+}
+
+static void
+nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove)
+{
+	/* Placeholder - teardown_admin_queue */
+}
+
+static void
+nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
+{
+	/* Placeholder - teardown_io_queues */
+}
+
+static void
+nvme_tcp_ofld_teardown_ctrl(struct nvme_ctrl *nctrl, bool shutdown)
+{
+	/* Placeholder - err_work and connect_work */
+	nvme_tcp_ofld_teardown_io_queues(nctrl, shutdown);
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	if (shutdown)
+		nvme_shutdown_ctrl(nctrl);
+	else
+		nvme_disable_ctrl(nctrl);
+	nvme_tcp_ofld_teardown_admin_queue(nctrl, shutdown);
+}
+
+static void nvme_tcp_ofld_delete_ctrl(struct nvme_ctrl *nctrl)
+{
+	nvme_tcp_ofld_teardown_ctrl(nctrl, true);
+}
+
+static int
+nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
+			   struct request *rq,
+			   unsigned int hctx_idx,
+			   unsigned int numa_node)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+
+	/* Placeholder - init request */
+
+	req->done = nvme_tcp_ofld_req_done;
+
+	return 0;
+}
+
+static blk_status_t
+nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
+		       const struct blk_mq_queue_data *bd)
+{
+	/* Call nvme_setup_cmd(...) */
+
+	/* Call ops->send_req(...) */
+
+	return BLK_STS_OK;
+}
+
+static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
+	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.init_request	= nvme_tcp_ofld_init_request,
+	/*
+	 * All additional ops will be also implemented and registered similar to
+	 * tcp.c
+	 */
+};
+
+static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
+	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.init_request	= nvme_tcp_ofld_init_request,
+	/*
+	 * All additional ops will be also implemented and registered similar to
+	 * tcp.c
+	 */
+};
+
+static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
+	.name			= "tcp_offload",
+	.module			= THIS_MODULE,
+	.flags			= NVME_F_FABRICS,
+	.reg_read32		= nvmf_reg_read32,
+	.reg_read64		= nvmf_reg_read64,
+	.reg_write32		= nvmf_reg_write32,
+	.free_ctrl		= nvme_tcp_ofld_free_ctrl,
+	.delete_ctrl		= nvme_tcp_ofld_delete_ctrl,
+	.get_address		= nvmf_get_address,
+};
+
+static bool
+nvme_tcp_ofld_existing_controller(struct nvmf_ctrl_options *opts)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	bool found = false;
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_for_each_entry(ctrl, &nvme_tcp_ofld_ctrl_list, list) {
+		found = nvmf_ip_options_match(&ctrl->nctrl, opts);
+		if (found)
+			break;
+	}
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+
+	return found;
+}
+
 static struct nvme_ctrl *
 nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 {
+	struct nvme_tcp_ofld_queue *queue;
 	struct nvme_tcp_ofld_ctrl *ctrl;
 	struct nvme_tcp_ofld_dev *dev;
 	struct nvme_ctrl *nctrl;
-	int rc = 0;
+	int i, rc = 0;
 
 	ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
 	if (!ctrl)
 		return ERR_PTR(-ENOMEM);
 
+	INIT_LIST_HEAD(&ctrl->list);
 	nctrl = &ctrl->nctrl;
+	nctrl->opts = opts;
+	nctrl->queue_count = opts->nr_io_queues + opts->nr_write_queues +
+			     opts->nr_poll_queues + 1;
+	nctrl->sqsize = opts->queue_size - 1;
+	nctrl->kato = opts->kato;
+	if (!(opts->mask & NVMF_OPT_TRSVCID)) {
+		opts->trsvcid =
+			kstrdup(__stringify(NVME_TCP_DISC_PORT), GFP_KERNEL);
+		if (!opts->trsvcid) {
+			rc = -ENOMEM;
+			goto out_free_ctrl;
+		}
+		opts->mask |= NVMF_OPT_TRSVCID;
+	}
+
+	rc = inet_pton_with_scope(&init_net, AF_UNSPEC, opts->traddr,
+				  opts->trsvcid,
+				  &ctrl->conn_params.remote_ip_addr);
+	if (rc) {
+		pr_err("malformed address passed: %s:%s\n",
+		       opts->traddr, opts->trsvcid);
+		goto out_free_ctrl;
+	}
 
-	/* Init nvme_tcp_ofld_ctrl and nvme_ctrl params based on received opts */
+	if (opts->mask & NVMF_OPT_HOST_TRADDR) {
+		rc = inet_pton_with_scope(&init_net, AF_UNSPEC,
+					  opts->host_traddr, NULL,
+					  &ctrl->conn_params.local_ip_addr);
+		if (rc) {
+			pr_err("malformed src address passed: %s\n",
+			       opts->host_traddr);
+			goto out_free_ctrl;
+		}
+	}
+
+	if (!opts->duplicate_connect &&
+	    nvme_tcp_ofld_existing_controller(opts)) {
+		rc = -EALREADY;
+		goto out_free_ctrl;
+	}
 
 	/* Find device that can reach the dest addr */
 	dev = nvme_tcp_ofld_lookup_dev(ctrl);
@@ -165,6 +588,10 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 		goto out_free_ctrl;
 	}
 
+	rc = nvme_tcp_ofld_check_dev_opts(opts, dev->ops);
+	if (rc)
+		goto out_module_put;
+
 	ctrl->dev = dev;
 
 	if (ctrl->dev->ops->max_hw_sectors)
@@ -172,14 +599,51 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 	if (ctrl->dev->ops->max_segments)
 		nctrl->max_segments = ctrl->dev->ops->max_segments;
 
-	/* Init queues */
+	ctrl->queues = kcalloc(nctrl->queue_count,
+			       sizeof(struct nvme_tcp_ofld_queue),
+			       GFP_KERNEL);
+	if (!ctrl->queues) {
+		rc = -ENOMEM;
+		goto out_module_put;
+	}
+
+	for (i = 0; i < nctrl->queue_count; ++i) {
+		queue = &ctrl->queues[i];
+		queue->ctrl = ctrl;
+		queue->dev = dev;
+		queue->report_err = nvme_tcp_ofld_report_queue_err;
+	}
+
+	rc = nvme_init_ctrl(nctrl, ndev, &nvme_tcp_ofld_ctrl_ops, 0);
+	if (rc)
+		goto out_free_queues;
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		WARN_ON_ONCE(1);
+		rc = -EINTR;
+		goto out_uninit_ctrl;
+	}
 
-	/* Call nvme_init_ctrl */
+	rc = nvme_tcp_ofld_setup_ctrl(nctrl, true);
+	if (rc)
+		goto out_uninit_ctrl;
 
-	/* Setup ctrl */
+	dev_info(nctrl->device, "new ctrl: NQN \"%s\", addr %pISp\n",
+		 opts->subsysnqn, &ctrl->conn_params.remote_ip_addr);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_add_tail(&ctrl->list, &nvme_tcp_ofld_ctrl_list);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
 
 	return nctrl;
 
+out_uninit_ctrl:
+	nvme_uninit_ctrl(nctrl);
+	nvme_put_ctrl(nctrl);
+out_free_queues:
+	kfree(ctrl->queues);
+out_module_put:
+	module_put(dev->ops->module);
 out_free_ctrl:
 	kfree(ctrl);
 
@@ -207,7 +671,15 @@ static int __init nvme_tcp_ofld_init_module(void)
 
 static void __exit nvme_tcp_ofld_cleanup_module(void)
 {
+	struct nvme_tcp_ofld_ctrl *ctrl;
+
 	nvmf_unregister_transport(&nvme_tcp_ofld_transport);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_for_each_entry(ctrl, &nvme_tcp_ofld_ctrl_list, list)
+		nvme_delete_ctrl(&ctrl->nctrl);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+	flush_workqueue(nvme_delete_wq);
 }
 
 module_init(nvme_tcp_ofld_init_module);
-- 
2.22.0



From smalin at marvell.com  Sun Jun 20 22:10:42 2021
From: smalin at marvell.com (Shai Malin)
Date: Mon, 21 Jun 2021 08:10:42 +0300
Subject: [PATCH v2 4/8] nvme-tcp-offload: Add device scan implementation
In-Reply-To: <20210621051046.28783-1-smalin@marvell.com>
References: <20210621051046.28783-1-smalin@marvell.com>
Message-ID: <20210621051046.28783-5-smalin@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

As part of create_ctrl(), it scans the registered devices and calls
the claim_dev op on each of them, to find the first devices that matches
the connection params. Once the correct devices is found (claim_dev
returns true), we raise the refcnt of that device and return that device
as the device to be used for ctrl currently being created.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/tcp-offload.c | 91 +++++++++++++++++++++++++++++++++
 drivers/nvme/host/tcp-offload.h |  1 +
 2 files changed, 92 insertions(+)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index b9b6ffa10747..15f018e45c60 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -13,6 +13,11 @@
 static LIST_HEAD(nvme_tcp_ofld_devices);
 static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
 
+static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nctrl)
+{
+	return container_of(nctrl, struct nvme_tcp_ofld_ctrl, nctrl);
+}
+
 /**
  * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
  * function.
@@ -96,6 +101,91 @@ void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
 	/* Placeholder - complete request with/without error */
 }
 
+static struct nvme_tcp_ofld_dev *
+nvme_tcp_ofld_lookup_dev(struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+	struct nvme_tcp_ofld_dev *dev;
+	char *iface;
+
+	if (nctrl->opts->mask & NVMF_OPT_HOST_IFACE) {
+		iface = nctrl->opts->host_iface;
+		ctrl->ndev = dev_get_by_name(&init_net, iface);
+		if (!ctrl->ndev) {
+			pr_err("invalid interface passed: %s\n", iface);
+			return NULL;
+		}
+	}
+
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_for_each_entry(dev, &nvme_tcp_ofld_devices, entry) {
+	/* ctrl includes the destination ip, source ip (if provided) and
+	 * network interface (if provided).
+	 */
+		if (dev->ops->claim_dev(dev, ctrl))
+			goto out;
+	}
+
+	dev = NULL;
+out:
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+
+	return dev;
+}
+
+static struct nvme_ctrl *
+nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	struct nvme_tcp_ofld_dev *dev;
+	struct nvme_ctrl *nctrl;
+	int rc = 0;
+
+	ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
+	if (!ctrl)
+		return ERR_PTR(-ENOMEM);
+
+	nctrl = &ctrl->nctrl;
+
+	/* Init nvme_tcp_ofld_ctrl and nvme_ctrl params based on received opts */
+
+	/* Find device that can reach the dest addr */
+	dev = nvme_tcp_ofld_lookup_dev(ctrl);
+	if (!dev) {
+		pr_info("no device found for addr %s:%s.\n",
+			opts->traddr, opts->trsvcid);
+		rc = -EINVAL;
+		goto out_free_ctrl;
+	}
+
+	/* Increase driver refcnt */
+	if (!try_module_get(dev->ops->module)) {
+		pr_err("try_module_get failed\n");
+		rc = -ENODEV;
+		goto out_free_ctrl;
+	}
+
+	ctrl->dev = dev;
+
+	if (ctrl->dev->ops->max_hw_sectors)
+		nctrl->max_hw_sectors = ctrl->dev->ops->max_hw_sectors;
+	if (ctrl->dev->ops->max_segments)
+		nctrl->max_segments = ctrl->dev->ops->max_segments;
+
+	/* Init queues */
+
+	/* Call nvme_init_ctrl */
+
+	/* Setup ctrl */
+
+	return nctrl;
+
+out_free_ctrl:
+	kfree(ctrl);
+
+	return ERR_PTR(rc);
+}
+
 static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
 	.name		= "tcp_offload",
 	.module		= THIS_MODULE,
@@ -105,6 +195,7 @@ static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
 			  NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HDR_DIGEST |
 			  NVMF_OPT_DATA_DIGEST | NVMF_OPT_NR_POLL_QUEUES |
 			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE,
+	.create_ctrl	= nvme_tcp_ofld_create_ctrl,
 };
 
 static int __init nvme_tcp_ofld_init_module(void)
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index 4964e18a5885..11e2a20c4c55 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -8,6 +8,7 @@
 #include <linux/scatterlist.h>
 #include <linux/types.h>
 #include <linux/nvme-tcp.h>
+#include <linux/netdevice.h>
 
 /* Driver includes */
 #include "nvme.h"
-- 
2.22.0



From smalin at marvell.com  Sun Jun 20 22:10:44 2021
From: smalin at marvell.com (Shai Malin)
Date: Mon, 21 Jun 2021 08:10:44 +0300
Subject: [PATCH v2 6/8] nvme-tcp-offload: Add controller level error recovery
 implementation
In-Reply-To: <20210621051046.28783-1-smalin@marvell.com>
References: <20210621051046.28783-1-smalin@marvell.com>
Message-ID: <20210621051046.28783-7-smalin@marvell.com>

From: Arie Gershberg <agershberg at marvell.com>

In this patch, we implement controller level error handling and recovery.
Upon an error discovered by the ULP or reset controller initiated by the
nvme-core (using reset_ctrl workqueue), the ULP will initiate a controller
recovery which includes teardown and re-connect of all queues.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 drivers/nvme/host/tcp-offload.c | 127 +++++++++++++++++++++++++++++++-
 drivers/nvme/host/tcp-offload.h |   1 +
 2 files changed, 126 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index ef7a0b89de9d..ccec011bf7eb 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -72,6 +72,23 @@ void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev)
 }
 EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
 
+/**
+ * nvme_tcp_ofld_error_recovery() - NVMeTCP Offload library error recovery.
+ * function.
+ * @nctrl:	NVMe controller instance to change to resetting.
+ *
+ * API function that change the controller state to resseting.
+ * Part of the overall controller reset sequence.
+ */
+void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl)
+{
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_RESETTING))
+		return;
+
+	queue_work(nvme_reset_wq, &to_tcp_ofld_ctrl(nctrl)->err_work);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_error_recovery);
+
 /**
  * nvme_tcp_ofld_report_queue_err() - NVMeTCP Offload report error event
  * callback function. Pointed to by nvme_tcp_ofld_queue->report_err.
@@ -82,7 +99,8 @@ EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
  */
 int nvme_tcp_ofld_report_queue_err(struct nvme_tcp_ofld_queue *queue)
 {
-	/* Placeholder - invoke error recovery flow */
+	pr_err("nvme-tcp-offload queue error\n");
+	nvme_tcp_ofld_error_recovery(&queue->ctrl->nctrl);
 
 	return 0;
 }
@@ -301,6 +319,28 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 	return rc;
 }
 
+static void nvme_tcp_ofld_reconnect_or_remove(struct nvme_ctrl *nctrl)
+{
+	/* If we are resetting/deleting then do nothing */
+	if (nctrl->state != NVME_CTRL_CONNECTING) {
+		WARN_ON_ONCE(nctrl->state == NVME_CTRL_NEW ||
+			     nctrl->state == NVME_CTRL_LIVE);
+
+		return;
+	}
+
+	if (nvmf_should_reconnect(nctrl)) {
+		dev_info(nctrl->device, "Reconnecting in %d seconds...\n",
+			 nctrl->opts->reconnect_delay);
+		queue_delayed_work(nvme_wq,
+				   &to_tcp_ofld_ctrl(nctrl)->connect_work,
+				   nctrl->opts->reconnect_delay * HZ);
+	} else {
+		dev_info(nctrl->device, "Removing controller...\n");
+		nvme_delete_ctrl(nctrl);
+	}
+}
+
 static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 {
 	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
@@ -425,10 +465,63 @@ nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
 	/* Placeholder - teardown_io_queues */
 }
 
+static void nvme_tcp_ofld_reconnect_ctrl_work(struct work_struct *work)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl =
+				container_of(to_delayed_work(work),
+					     struct nvme_tcp_ofld_ctrl,
+					     connect_work);
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+
+	++nctrl->nr_reconnects;
+
+	if (nvme_tcp_ofld_setup_ctrl(nctrl, false))
+		goto requeue;
+
+	dev_info(nctrl->device, "Successfully reconnected (%d attempt)\n",
+		 nctrl->nr_reconnects);
+
+	nctrl->nr_reconnects = 0;
+
+	return;
+
+requeue:
+	dev_info(nctrl->device, "Failed reconnect attempt %d\n",
+		 nctrl->nr_reconnects);
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
+static void nvme_tcp_ofld_error_recovery_work(struct work_struct *work)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl =
+		container_of(work, struct nvme_tcp_ofld_ctrl, err_work);
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+
+	nvme_stop_keep_alive(nctrl);
+	nvme_tcp_ofld_teardown_io_queues(nctrl, false);
+	/* unquiesce to fail fast pending requests */
+	nvme_start_queues(nctrl);
+	nvme_tcp_ofld_teardown_admin_queue(nctrl, false);
+	blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		/* state change failure is ok if we started nctrl delete */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+
+		return;
+	}
+
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
 static void
 nvme_tcp_ofld_teardown_ctrl(struct nvme_ctrl *nctrl, bool shutdown)
 {
-	/* Placeholder - err_work and connect_work */
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+
+	cancel_work_sync(&ctrl->err_work);
+	cancel_delayed_work_sync(&ctrl->connect_work);
 	nvme_tcp_ofld_teardown_io_queues(nctrl, shutdown);
 	blk_mq_quiesce_queue(nctrl->admin_q);
 	if (shutdown)
@@ -443,6 +536,32 @@ static void nvme_tcp_ofld_delete_ctrl(struct nvme_ctrl *nctrl)
 	nvme_tcp_ofld_teardown_ctrl(nctrl, true);
 }
 
+static void nvme_tcp_ofld_reset_ctrl_work(struct work_struct *work)
+{
+	struct nvme_ctrl *nctrl =
+		container_of(work, struct nvme_ctrl, reset_work);
+
+	nvme_stop_ctrl(nctrl);
+	nvme_tcp_ofld_teardown_ctrl(nctrl, false);
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		/* state change failure is ok if we started ctrl delete */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+
+		return;
+	}
+
+	if (nvme_tcp_ofld_setup_ctrl(nctrl, false))
+		goto out_fail;
+
+	return;
+
+out_fail:
+	++nctrl->nr_reconnects;
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
 static int
 nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 			   struct request *rq,
@@ -536,6 +655,10 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 			     opts->nr_poll_queues + 1;
 	nctrl->sqsize = opts->queue_size - 1;
 	nctrl->kato = opts->kato;
+	INIT_DELAYED_WORK(&ctrl->connect_work,
+			  nvme_tcp_ofld_reconnect_ctrl_work);
+	INIT_WORK(&ctrl->err_work, nvme_tcp_ofld_error_recovery_work);
+	INIT_WORK(&nctrl->reset_work, nvme_tcp_ofld_reset_ctrl_work);
 	if (!(opts->mask & NVMF_OPT_TRSVCID)) {
 		opts->trsvcid =
 			kstrdup(__stringify(NVME_TCP_DISC_PORT), GFP_KERNEL);
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index 11e2a20c4c55..b583e1824ef7 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -198,3 +198,4 @@ struct nvme_tcp_ofld_ops {
 /* Exported functions for lower vendor specific offload drivers */
 int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
+void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl);
-- 
2.22.0



From smalin at marvell.com  Sun Jun 20 22:10:46 2021
From: smalin at marvell.com (Shai Malin)
Date: Mon, 21 Jun 2021 08:10:46 +0300
Subject: [PATCH v2 8/8] nvme-tcp-offload: Add IO level implementation
In-Reply-To: <20210621051046.28783-1-smalin@marvell.com>
References: <20210621051046.28783-1-smalin@marvell.com>
Message-ID: <20210621051046.28783-9-smalin@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

In this patch, we present the IO level functionality.
The nvme-tcp-offload shall work on the IO-level, meaning the
nvme-tcp-offload ULP module shall pass the request to the nvme-tcp-offload
vendor driver and shall expect for the request completion.
No additional handling is needed in between, this design will reduce the
CPU utilization as we will describe below.

The nvme-tcp-offload vendor driver shall register to nvme-tcp-offload ULP
with the following IO-path ops:
 - send_req - in order to pass the request to the handling of the offload
   driver that shall pass it to the vendor specific device
 - poll_queue

The vendor driver will manage the context from which the request will be
executed and the request aggregations.
Once the IO completed, the nvme-tcp-offload vendor driver shall call
command.done() that shall invoke the nvme-tcp-offload ULP layer for
completing the request.

This patch also add support for the nvme-tcp-offload timeout and
nvme-tcp-offload ASYNC flow.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 drivers/nvme/host/tcp-offload.c | 176 ++++++++++++++++++++++++++++++--
 drivers/nvme/host/tcp-offload.h |   2 +
 2 files changed, 171 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index ecac5ef740ff..338c4d56159e 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -125,7 +125,30 @@ void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
 			    union nvme_result *result,
 			    __le16 status)
 {
-	/* Placeholder - complete request with/without error */
+	struct request *rq = blk_mq_rq_from_pdu(req);
+
+	if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), *result))
+		nvme_complete_rq(rq);
+}
+
+/**
+ * nvme_tcp_ofld_async_req_done() - NVMeTCP Offload request done callback
+ * function for async request. Pointed to by nvme_tcp_ofld_req->done.
+ * Handles both NVME_TCP_F_DATA_SUCCESS flag and NVMe CQ.
+ * @req:	NVMeTCP offload request to complete.
+ * @result:     The nvme_result.
+ * @status:     The completion status.
+ *
+ * API function that allows the vendor specific offload driver to report request
+ * completions to the common offload layer.
+ */
+void nvme_tcp_ofld_async_req_done(struct nvme_tcp_ofld_req *req,
+				  union nvme_result *result, __le16 status)
+{
+	struct nvme_tcp_ofld_queue *queue = req->queue;
+	struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl;
+
+	nvme_complete_async_event(&ctrl->nctrl, status, result);
 }
 
 static struct nvme_tcp_ofld_dev *
@@ -713,6 +736,54 @@ static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
 	kfree(ctrl);
 }
 
+static void nvme_tcp_ofld_set_sg_null(struct nvme_command *c)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = 0;
+	sg->length = 0;
+	sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) | NVME_SGL_FMT_TRANSPORT_A;
+}
+
+inline void nvme_tcp_ofld_set_sg_inline(struct nvme_tcp_ofld_queue *queue,
+					struct nvme_command *c, u32 data_len)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = cpu_to_le64(queue->ctrl->nctrl.icdoff);
+	sg->length = cpu_to_le32(data_len);
+	sg->type = (NVME_SGL_FMT_DATA_DESC << 4) | NVME_SGL_FMT_OFFSET;
+}
+
+static void nvme_tcp_ofld_map_data(struct nvme_command *c, u32 data_len)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = 0;
+	sg->length = cpu_to_le32(data_len);
+	sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) | NVME_SGL_FMT_TRANSPORT_A;
+}
+
+static void nvme_tcp_ofld_submit_async_event(struct nvme_ctrl *arg)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(arg);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[0];
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+
+	ctrl->async_req.nvme_cmd.common.opcode = nvme_admin_async_event;
+	ctrl->async_req.nvme_cmd.common.command_id = NVME_AQ_BLK_MQ_DEPTH;
+	ctrl->async_req.nvme_cmd.common.flags |= NVME_CMD_SGL_METABUF;
+
+	nvme_tcp_ofld_set_sg_null(&ctrl->async_req.nvme_cmd);
+
+	ctrl->async_req.async = true;
+	ctrl->async_req.queue = queue;
+	ctrl->async_req.done = nvme_tcp_ofld_async_req_done;
+
+	ops->send_req(&ctrl->async_req);
+}
+
 static void
 nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *nctrl, bool remove)
 {
@@ -851,9 +922,13 @@ nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 			   unsigned int numa_node)
 {
 	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_ofld_ctrl *ctrl = set->driver_data;
+	int qid;
 
-	/* Placeholder - init request */
-
+	qid = (set == &ctrl->tag_set) ? hctx_idx + 1 : 0;
+	req->queue = &ctrl->queues[qid];
+	nvme_req(rq)->ctrl = &ctrl->nctrl;
+	nvme_req(rq)->cmd = &req->nvme_cmd;
 	req->done = nvme_tcp_ofld_req_done;
 
 	return 0;
@@ -869,9 +944,46 @@ static blk_status_t
 nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 		       const struct blk_mq_queue_data *bd)
 {
-	/* Call nvme_setup_cmd(...) */
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(bd->rq);
+	struct nvme_tcp_ofld_queue *queue = hctx->driver_data;
+	struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl;
+	struct nvme_ns *ns = hctx->queue->queuedata;
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+	struct nvme_command *nvme_cmd;
+	struct request *rq = bd->rq;
+	bool queue_ready;
+	u32 data_len;
+	int rc;
+
+	queue_ready = test_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags);
+
+	req->async = false;
+
+	if (!nvme_check_ready(&ctrl->nctrl, rq, queue_ready))
+		return nvme_fail_nonready_command(&ctrl->nctrl, rq);
+
+	rc = nvme_setup_cmd(ns, rq);
+	if (unlikely(rc))
+		return rc;
 
-	/* Call ops->send_req(...) */
+	blk_mq_start_request(rq);
+
+	nvme_cmd = &req->nvme_cmd;
+	nvme_cmd->common.flags |= NVME_CMD_SGL_METABUF;
+
+	data_len = blk_rq_nr_phys_segments(rq) ? blk_rq_payload_bytes(rq) : 0;
+	if (!data_len)
+		nvme_tcp_ofld_set_sg_null(&req->nvme_cmd);
+	else if ((rq_data_dir(rq) == WRITE) &&
+		 data_len <= nvme_tcp_ofld_inline_data_size(queue))
+		nvme_tcp_ofld_set_sg_inline(queue, nvme_cmd, data_len);
+	else
+		nvme_tcp_ofld_map_data(nvme_cmd, data_len);
+
+	rc = ops->send_req(req);
+	if (unlikely(rc))
+		return rc;
 
 	return BLK_STS_OK;
 }
@@ -944,9 +1056,56 @@ static int nvme_tcp_ofld_map_queues(struct blk_mq_tag_set *set)
 
 static int nvme_tcp_ofld_poll(struct blk_mq_hw_ctx *hctx)
 {
-	/* Placeholder - Implement polling mechanism */
+	struct nvme_tcp_ofld_queue *queue = hctx->driver_data;
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
 
-	return 0;
+	return ops->poll_queue(queue);
+}
+
+static void nvme_tcp_ofld_complete_timed_out(struct request *rq)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_ctrl *nctrl = &req->queue->ctrl->nctrl;
+
+	nvme_tcp_ofld_stop_queue(nctrl, nvme_tcp_ofld_qid(req->queue));
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq)) {
+		nvme_req(rq)->status = NVME_SC_HOST_ABORTED_CMD;
+		blk_mq_complete_request(rq);
+	}
+}
+
+static enum blk_eh_timer_return nvme_tcp_ofld_timeout(struct request *rq, bool reserved)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_ofld_ctrl *ctrl = req->queue->ctrl;
+
+	dev_warn(ctrl->nctrl.device,
+		 "queue %d: timeout request %#x type %d\n",
+		 nvme_tcp_ofld_qid(req->queue), rq->tag, req->nvme_cmd.common.opcode);
+
+	if (ctrl->nctrl.state != NVME_CTRL_LIVE) {
+		/*
+		 * If we are resetting, connecting or deleting we should
+		 * complete immediately because we may block controller
+		 * teardown or setup sequence
+		 * - ctrl disable/shutdown fabrics requests
+		 * - connect requests
+		 * - initialization admin requests
+		 * - I/O requests that entered after unquiescing and
+		 *   the controller stopped responding
+		 *
+		 * All other requests should be cancelled by the error
+		 * recovery work, so it's fine that we fail it here.
+		 */
+		nvme_tcp_ofld_complete_timed_out(rq);
+
+		return BLK_EH_DONE;
+	}
+
+	nvme_tcp_ofld_error_recovery(&ctrl->nctrl);
+
+	return BLK_EH_RESET_TIMER;
 }
 
 static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
@@ -955,6 +1114,7 @@ static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
 	.init_request	= nvme_tcp_ofld_init_request,
 	.exit_request	= nvme_tcp_ofld_exit_request,
 	.init_hctx	= nvme_tcp_ofld_init_hctx,
+	.timeout	= nvme_tcp_ofld_timeout,
 	.map_queues	= nvme_tcp_ofld_map_queues,
 	.poll		= nvme_tcp_ofld_poll,
 };
@@ -965,6 +1125,7 @@ static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
 	.init_request	= nvme_tcp_ofld_init_request,
 	.exit_request	= nvme_tcp_ofld_exit_request,
 	.init_hctx	= nvme_tcp_ofld_init_admin_hctx,
+	.timeout	= nvme_tcp_ofld_timeout,
 };
 
 static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
@@ -975,6 +1136,7 @@ static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
 	.reg_read64		= nvmf_reg_read64,
 	.reg_write32		= nvmf_reg_write32,
 	.free_ctrl		= nvme_tcp_ofld_free_ctrl,
+	.submit_async_event     = nvme_tcp_ofld_submit_async_event,
 	.delete_ctrl		= nvme_tcp_ofld_delete_ctrl,
 	.get_address		= nvmf_get_address,
 };
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index 0f88f34a1a8e..ef350e1e3897 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -115,6 +115,8 @@ struct nvme_tcp_ofld_ctrl {
 	/* Connectivity params */
 	struct nvme_tcp_ofld_ctrl_con_params conn_params;
 
+	struct nvme_tcp_ofld_req async_req;
+
 	/* Vendor specific driver context */
 	void *private_data;
 };
-- 
2.22.0



From smalin at marvell.com  Sun Jun 20 22:10:45 2021
From: smalin at marvell.com (Shai Malin)
Date: Mon, 21 Jun 2021 08:10:45 +0300
Subject: [PATCH v2 7/8] nvme-tcp-offload: Add queue level implementation
In-Reply-To: <20210621051046.28783-1-smalin@marvell.com>
References: <20210621051046.28783-1-smalin@marvell.com>
Message-ID: <20210621051046.28783-8-smalin@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

In this patch we implement queue level functionality.
The implementation is similar to the nvme-tcp module, the main
difference being that we call the vendor specific create_queue op which
creates the TCP connection, and NVMeTPC connection including
icreq+icresp negotiation.
Once create_queue returns successfully, we can move on to the fabrics
connect.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/tcp-offload.c | 417 +++++++++++++++++++++++++++++---
 drivers/nvme/host/tcp-offload.h |   4 +
 2 files changed, 393 insertions(+), 28 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index ccec011bf7eb..ecac5ef740ff 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -22,6 +22,11 @@ static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nctr
 	return container_of(nctrl, struct nvme_tcp_ofld_ctrl, nctrl);
 }
 
+static inline int nvme_tcp_ofld_qid(struct nvme_tcp_ofld_queue *queue)
+{
+	return queue - queue->ctrl->queues;
+}
+
 /**
  * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
  * function.
@@ -196,19 +201,124 @@ nvme_tcp_ofld_alloc_tagset(struct nvme_ctrl *nctrl, bool admin)
 	return set;
 }
 
+static void __nvme_tcp_ofld_stop_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	queue->dev->ops->drain_queue(queue);
+}
+
+static void nvme_tcp_ofld_stop_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+
+	mutex_lock(&queue->queue_lock);
+	if (test_and_clear_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags))
+		__nvme_tcp_ofld_stop_queue(queue);
+	mutex_unlock(&queue->queue_lock);
+}
+
+static void nvme_tcp_ofld_stop_io_queues(struct nvme_ctrl *ctrl)
+{
+	int i;
+
+	for (i = 1; i < ctrl->queue_count; i++)
+		nvme_tcp_ofld_stop_queue(ctrl, i);
+}
+
+static void __nvme_tcp_ofld_free_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	queue->dev->ops->destroy_queue(queue);
+}
+
+static void nvme_tcp_ofld_free_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+
+	if (test_and_clear_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags)) {
+		__nvme_tcp_ofld_free_queue(queue);
+		mutex_destroy(&queue->queue_lock);
+	}
+}
+
+static void
+nvme_tcp_ofld_free_io_queues(struct nvme_ctrl *nctrl)
+{
+	int i;
+
+	for (i = 1; i < nctrl->queue_count; i++)
+		nvme_tcp_ofld_free_queue(nctrl, i);
+}
+
+static void nvme_tcp_ofld_destroy_io_queues(struct nvme_ctrl *nctrl, bool remove)
+{
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+	if (remove) {
+		blk_cleanup_queue(nctrl->connect_q);
+		blk_mq_free_tag_set(nctrl->tagset);
+	}
+	nvme_tcp_ofld_free_io_queues(nctrl);
+}
+
+static void nvme_tcp_ofld_destroy_admin_queue(struct nvme_ctrl *nctrl, bool remove)
+{
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	if (remove) {
+		blk_cleanup_queue(nctrl->admin_q);
+		blk_cleanup_queue(nctrl->fabrics_q);
+		blk_mq_free_tag_set(nctrl->admin_tagset);
+	}
+	nvme_tcp_ofld_free_queue(nctrl, 0);
+}
+
+static int nvme_tcp_ofld_start_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+	int rc;
+
+	queue = &ctrl->queues[qid];
+	if (qid) {
+		queue->cmnd_capsule_len = nctrl->ioccsz * 16;
+		rc = nvmf_connect_io_queue(nctrl, qid, false);
+	} else {
+		queue->cmnd_capsule_len = sizeof(struct nvme_command) + NVME_TCP_ADMIN_CCSZ;
+		rc = nvmf_connect_admin_queue(nctrl);
+	}
+
+	if (!rc) {
+		set_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags);
+	} else {
+		if (test_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags))
+			__nvme_tcp_ofld_stop_queue(queue);
+		dev_err(nctrl->device,
+			"failed to connect queue: %d ret=%d\n", qid, rc);
+	}
+
+	return rc;
+}
+
 static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 					       bool new)
 {
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[0];
 	int rc;
 
-	/* Placeholder - alloc_admin_queue */
+	mutex_init(&queue->queue_lock);
+
+	rc = ctrl->dev->ops->create_queue(queue, 0, NVME_AQ_DEPTH);
+	if (rc)
+		return rc;
+
+	set_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags);
 	if (new) {
 		nctrl->admin_tagset =
 				nvme_tcp_ofld_alloc_tagset(nctrl, true);
 		if (IS_ERR(nctrl->admin_tagset)) {
 			rc = PTR_ERR(nctrl->admin_tagset);
 			nctrl->admin_tagset = NULL;
-			goto out_destroy_queue;
+			goto out_free_queue;
 		}
 
 		nctrl->fabrics_q = blk_mq_init_queue(nctrl->admin_tagset);
@@ -226,7 +336,9 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 		}
 	}
 
-	/* Placeholder - nvme_tcp_ofld_start_queue */
+	rc = nvme_tcp_ofld_start_queue(nctrl, 0);
+	if (rc)
+		goto out_cleanup_queue;
 
 	rc = nvme_enable_ctrl(nctrl);
 	if (rc)
@@ -243,19 +355,143 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 out_quiesce_queue:
 	blk_mq_quiesce_queue(nctrl->admin_q);
 	blk_sync_queue(nctrl->admin_q);
-
 out_stop_queue:
-	/* Placeholder - stop offload queue */
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
 	nvme_cancel_admin_tagset(nctrl);
-
+out_cleanup_queue:
+	if (new)
+		blk_cleanup_queue(nctrl->admin_q);
 out_cleanup_fabrics_q:
 	if (new)
 		blk_cleanup_queue(nctrl->fabrics_q);
 out_free_tagset:
 	if (new)
 		blk_mq_free_tag_set(nctrl->admin_tagset);
-out_destroy_queue:
-	/* Placeholder - free admin queue */
+out_free_queue:
+	nvme_tcp_ofld_free_queue(nctrl, 0);
+
+	return rc;
+}
+
+static unsigned int nvme_tcp_ofld_nr_io_queues(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+	u32 hw_vectors = dev->num_hw_vectors;
+	u32 nr_write_queues, nr_poll_queues;
+	u32 nr_io_queues, nr_total_queues;
+
+	nr_io_queues = min3(nctrl->opts->nr_io_queues, num_online_cpus(),
+			    hw_vectors);
+	nr_write_queues = min3(nctrl->opts->nr_write_queues, num_online_cpus(),
+			       hw_vectors);
+	nr_poll_queues = min3(nctrl->opts->nr_poll_queues, num_online_cpus(),
+			      hw_vectors);
+
+	nr_total_queues = nr_io_queues + nr_write_queues + nr_poll_queues;
+
+	return nr_total_queues;
+}
+
+static void
+nvme_tcp_ofld_set_io_queues(struct nvme_ctrl *nctrl, unsigned int nr_io_queues)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvmf_ctrl_options *opts = nctrl->opts;
+
+	if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) {
+		/*
+		 * separate read/write queues
+		 * hand out dedicated default queues only after we have
+		 * sufficient read queues.
+		 */
+		ctrl->io_queues[HCTX_TYPE_READ] = opts->nr_io_queues;
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_READ];
+		ctrl->io_queues[HCTX_TYPE_DEFAULT] =
+			min(opts->nr_write_queues, nr_io_queues);
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	} else {
+		/*
+		 * shared read/write queues
+		 * either no write queues were requested, or we don't have
+		 * sufficient queue count to have dedicated default queues.
+		 */
+		ctrl->io_queues[HCTX_TYPE_DEFAULT] =
+			min(opts->nr_io_queues, nr_io_queues);
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	}
+
+	if (opts->nr_poll_queues && nr_io_queues) {
+		/* map dedicated poll queues only if we have queues left */
+		ctrl->io_queues[HCTX_TYPE_POLL] =
+			min(opts->nr_poll_queues, nr_io_queues);
+	}
+}
+
+static int nvme_tcp_ofld_create_io_queues(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	int i, rc;
+
+	for (i = 1; i < nctrl->queue_count; i++) {
+		mutex_init(&ctrl->queues[i].queue_lock);
+
+		rc = ctrl->dev->ops->create_queue(&ctrl->queues[i],
+						  i, nctrl->sqsize + 1);
+		if (rc)
+			goto out_free_queues;
+
+		set_bit(NVME_TCP_OFLD_Q_ALLOCATED, &ctrl->queues[i].flags);
+	}
+
+	return 0;
+
+out_free_queues:
+	for (i--; i >= 1; i--)
+		nvme_tcp_ofld_free_queue(nctrl, i);
+
+	return rc;
+}
+
+static int nvme_tcp_ofld_alloc_io_queues(struct nvme_ctrl *nctrl)
+{
+	unsigned int nr_io_queues;
+	int rc;
+
+	nr_io_queues = nvme_tcp_ofld_nr_io_queues(nctrl);
+	rc = nvme_set_queue_count(nctrl, &nr_io_queues);
+	if (rc)
+		return rc;
+
+	nctrl->queue_count = nr_io_queues + 1;
+	if (nctrl->queue_count < 2) {
+		dev_err(nctrl->device,
+			"unable to set any I/O queues\n");
+
+		return -ENOMEM;
+	}
+
+	dev_info(nctrl->device, "creating %d I/O queues.\n", nr_io_queues);
+	nvme_tcp_ofld_set_io_queues(nctrl, nr_io_queues);
+
+	return nvme_tcp_ofld_create_io_queues(nctrl);
+}
+
+static int nvme_tcp_ofld_start_io_queues(struct nvme_ctrl *nctrl)
+{
+	int i, rc = 0;
+
+	for (i = 1; i < nctrl->queue_count; i++) {
+		rc = nvme_tcp_ofld_start_queue(nctrl, i);
+		if (rc)
+			goto out_stop_queues;
+	}
+
+	return 0;
+
+out_stop_queues:
+	for (i--; i >= 1; i--)
+		nvme_tcp_ofld_stop_queue(nctrl, i);
 
 	return rc;
 }
@@ -263,9 +499,10 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 static int
 nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 {
-	int rc;
+	int rc = nvme_tcp_ofld_alloc_io_queues(nctrl);
 
-	/* Placeholder - alloc_io_queues */
+	if (rc)
+		return rc;
 
 	if (new) {
 		nctrl->tagset = nvme_tcp_ofld_alloc_tagset(nctrl, false);
@@ -283,7 +520,9 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 		}
 	}
 
-	/* Placeholder - start_io_queues */
+	rc = nvme_tcp_ofld_start_io_queues(nctrl);
+	if (rc)
+		goto out_cleanup_connect_q;
 
 	if (!new) {
 		nvme_start_queues(nctrl);
@@ -305,16 +544,16 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 out_wait_freeze_timed_out:
 	nvme_stop_queues(nctrl);
 	nvme_sync_io_queues(nctrl);
-
-	/* Placeholder - Stop IO queues */
-
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+out_cleanup_connect_q:
+	nvme_cancel_tagset(nctrl);
 	if (new)
 		blk_cleanup_queue(nctrl->connect_q);
 out_free_tag_set:
 	if (new)
 		blk_mq_free_tag_set(nctrl->tagset);
 out_free_io_queues:
-	/* Placeholder - free_io_queues */
+	nvme_tcp_ofld_free_io_queues(nctrl);
 
 	return rc;
 }
@@ -341,6 +580,17 @@ static void nvme_tcp_ofld_reconnect_or_remove(struct nvme_ctrl *nctrl)
 	}
 }
 
+static int
+nvme_tcp_ofld_init_admin_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			      unsigned int hctx_idx)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = data;
+
+	hctx->driver_data = &ctrl->queues[0];
+
+	return 0;
+}
+
 static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 {
 	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
@@ -403,9 +653,19 @@ static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 	return 0;
 
 destroy_io:
-	/* Placeholder - stop and destroy io queues*/
+	if (nctrl->queue_count > 1) {
+		nvme_stop_queues(nctrl);
+		nvme_sync_io_queues(nctrl);
+		nvme_tcp_ofld_stop_io_queues(nctrl);
+		nvme_cancel_tagset(nctrl);
+		nvme_tcp_ofld_destroy_io_queues(nctrl, new);
+	}
 destroy_admin:
-	/* Placeholder - stop and destroy admin queue*/
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	nvme_cancel_admin_tagset(nctrl);
+	nvme_tcp_ofld_destroy_admin_queue(nctrl, new);
 out_release_ctrl:
 	ctrl->dev->ops->release_ctrl(ctrl);
 
@@ -454,15 +714,37 @@ static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
 }
 
 static void
-nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove)
+nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *nctrl, bool remove)
 {
-	/* Placeholder - teardown_admin_queue */
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	nvme_cancel_admin_tagset(nctrl);
+
+	if (remove)
+		blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	nvme_tcp_ofld_destroy_admin_queue(nctrl, remove);
 }
 
 static void
 nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
 {
-	/* Placeholder - teardown_io_queues */
+	if (nctrl->queue_count <= 1)
+		return;
+
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	nvme_start_freeze(nctrl);
+	nvme_stop_queues(nctrl);
+	nvme_sync_io_queues(nctrl);
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+	nvme_cancel_tagset(nctrl);
+
+	if (remove)
+		nvme_start_queues(nctrl);
+
+	nvme_tcp_ofld_destroy_io_queues(nctrl, remove);
 }
 
 static void nvme_tcp_ofld_reconnect_ctrl_work(struct work_struct *work)
@@ -577,6 +859,12 @@ nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 	return 0;
 }
 
+inline size_t nvme_tcp_ofld_inline_data_size(struct nvme_tcp_ofld_queue *queue)
+{
+	return queue->cmnd_capsule_len - sizeof(struct nvme_command);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_inline_data_size);
+
 static blk_status_t
 nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 		       const struct blk_mq_queue_data *bd)
@@ -588,22 +876,95 @@ nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 	return BLK_STS_OK;
 }
 
+static void
+nvme_tcp_ofld_exit_request(struct blk_mq_tag_set *set,
+			   struct request *rq, unsigned int hctx_idx)
+{
+	/*
+	 * Nothing is allocated in nvme_tcp_ofld_init_request,
+	 * hence empty.
+	 */
+}
+
+static int
+nvme_tcp_ofld_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			unsigned int hctx_idx)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = data;
+
+	hctx->driver_data = &ctrl->queues[hctx_idx + 1];
+
+	return 0;
+}
+
+static int nvme_tcp_ofld_map_queues(struct blk_mq_tag_set *set)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = set->driver_data;
+	struct nvmf_ctrl_options *opts = ctrl->nctrl.opts;
+
+	if (opts->nr_write_queues && ctrl->io_queues[HCTX_TYPE_READ]) {
+		/* separate read/write queues */
+		set->map[HCTX_TYPE_DEFAULT].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_DEFAULT].queue_offset = 0;
+		set->map[HCTX_TYPE_READ].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_READ];
+		set->map[HCTX_TYPE_READ].queue_offset =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	} else {
+		/* shared read/write queues */
+		set->map[HCTX_TYPE_DEFAULT].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_DEFAULT].queue_offset = 0;
+		set->map[HCTX_TYPE_READ].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_READ].queue_offset = 0;
+	}
+	blk_mq_map_queues(&set->map[HCTX_TYPE_DEFAULT]);
+	blk_mq_map_queues(&set->map[HCTX_TYPE_READ]);
+
+	if (opts->nr_poll_queues && ctrl->io_queues[HCTX_TYPE_POLL]) {
+		/* map dedicated poll queues only if we have queues left */
+		set->map[HCTX_TYPE_POLL].nr_queues =
+				ctrl->io_queues[HCTX_TYPE_POLL];
+		set->map[HCTX_TYPE_POLL].queue_offset =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT] +
+			ctrl->io_queues[HCTX_TYPE_READ];
+		blk_mq_map_queues(&set->map[HCTX_TYPE_POLL]);
+	}
+
+	dev_info(ctrl->nctrl.device,
+		 "mapped %d/%d/%d default/read/poll queues.\n",
+		 ctrl->io_queues[HCTX_TYPE_DEFAULT],
+		 ctrl->io_queues[HCTX_TYPE_READ],
+		 ctrl->io_queues[HCTX_TYPE_POLL]);
+
+	return 0;
+}
+
+static int nvme_tcp_ofld_poll(struct blk_mq_hw_ctx *hctx)
+{
+	/* Placeholder - Implement polling mechanism */
+
+	return 0;
+}
+
 static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
 	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.complete	= nvme_complete_rq,
 	.init_request	= nvme_tcp_ofld_init_request,
-	/*
-	 * All additional ops will be also implemented and registered similar to
-	 * tcp.c
-	 */
+	.exit_request	= nvme_tcp_ofld_exit_request,
+	.init_hctx	= nvme_tcp_ofld_init_hctx,
+	.map_queues	= nvme_tcp_ofld_map_queues,
+	.poll		= nvme_tcp_ofld_poll,
 };
 
 static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
 	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.complete	= nvme_complete_rq,
 	.init_request	= nvme_tcp_ofld_init_request,
-	/*
-	 * All additional ops will be also implemented and registered similar to
-	 * tcp.c
-	 */
+	.exit_request	= nvme_tcp_ofld_exit_request,
+	.init_hctx	= nvme_tcp_ofld_init_admin_hctx,
 };
 
 static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index b583e1824ef7..0f88f34a1a8e 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -65,6 +65,9 @@ struct nvme_tcp_ofld_queue {
 	unsigned long flags;
 	size_t cmnd_capsule_len;
 
+	/* mutex used during stop_queue */
+	struct mutex queue_lock;
+
 	u8 hdr_digest;
 	u8 data_digest;
 	u8 tos;
@@ -199,3 +202,4 @@ struct nvme_tcp_ofld_ops {
 int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl);
+inline size_t nvme_tcp_ofld_inline_data_size(struct nvme_tcp_ofld_queue *queue);
-- 
2.22.0



From hch at lst.de  Sun Jun 20 22:31:18 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 21 Jun 2021 07:31:18 +0200
Subject: [PATCH] remove all blanks in front of goto label
In-Reply-To: <BYAPR04MB49650E07DD0D902EADE35995860A9@BYAPR04MB4965.namprd04.prod.outlook.com>
References: <tencent_B28F4F86583B12ADE15402FA0EBA8CE87806@qq.com>
 <BYAPR04MB49650E07DD0D902EADE35995860A9@BYAPR04MB4965.namprd04.prod.outlook.com>
Message-ID: <20210621053118.GA4242@lst.de>

On Mon, Jun 21, 2021 at 03:23:34AM +0000, Chaitanya Kulkarni wrote:
> While reviewing this patch I found that on the host side there
> are many places where we've spaces before goto labeldefinition,
> perhaps following cleanup is useful ?

This is a pretty common style in Linux.  Please don't touch it for
existing code.


From hch at lst.de  Sun Jun 20 22:32:51 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 21 Jun 2021 07:32:51 +0200
Subject: [PATCH] nvme: add comment to not use kobj_to_dev()
In-Reply-To: <20210621030940.22286-1-chaitanya.kulkarni@wdc.com>
References: <20210621030940.22286-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210621053251.GA4360@lst.de>

On Sun, Jun 20, 2021 at 08:09:40PM -0700, Chaitanya Kulkarni wrote:
> Add a comment in nvme_ns_id_attrs_are_visible() and
> nvme_dev_attrs_are_visible() not to replace container_of() with
> kobj_to_dev(). Since use of kobj_to_dev() makes code harder to read.
> 
> Suggested-by: Keith Busch <kbusch at kernel.org>
> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

Err, no.  This code is completely obvious as is, and if people keep
sending these stupid patches we need to start blacklisting them.


From hch at lst.de  Sun Jun 20 23:38:20 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 21 Jun 2021 08:38:20 +0200
Subject: [PATCHv6] nvme: allow to re-attach namespaces after all paths
 are down
In-Reply-To: <20210609150118.130650-1-hare@suse.de>
References: <20210609150118.130650-1-hare@suse.de>
Message-ID: <20210621063820.GA5618@lst.de>

On Wed, Jun 09, 2021 at 05:01:18PM +0200, Hannes Reinecke wrote:
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 177cae44b612..6d7c2958b3e2 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -566,6 +566,9 @@ static void nvme_free_ns_head(struct kref *ref)
>  	struct nvme_ns_head *head =
>  		container_of(ref, struct nvme_ns_head, ref);
>  
> +	mutex_lock(&head->subsys->lock);
> +	list_del_init(&head->entry);
> +	mutex_unlock(&head->subsys->lock);
>  	nvme_mpath_remove_disk(head);
>  	ida_simple_remove(&head->subsys->ns_ida, head->instance);
>  	cleanup_srcu_struct(&head->srcu);
> @@ -3806,8 +3809,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid,
>   out_unlink_ns:
>  	mutex_lock(&ctrl->subsys->lock);
>  	list_del_rcu(&ns->siblings);
> -	if (list_empty(&ns->head->list))
> -		list_del_init(&ns->head->entry);
>  	mutex_unlock(&ctrl->subsys->lock);
>  	nvme_put_ns_head(ns->head);
>   out_free_queue:
> @@ -3828,8 +3829,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>  
>  	mutex_lock(&ns->ctrl->subsys->lock);
>  	list_del_rcu(&ns->siblings);
> -	if (list_empty(&ns->head->list))
> -		list_del_init(&ns->head->entry);
>  	mutex_unlock(&ns->ctrl->subsys->lock);
>  
>  	synchronize_rcu(); /* guarantee not available in head->list */
> @@ -3849,7 +3848,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>  	list_del_init(&ns->list);
>  	up_write(&ns->ctrl->namespaces_rwsem);
>  
> -	nvme_mpath_check_last_path(ns);
> +	nvme_mpath_check_last_path(ns->head);
>  	nvme_put_ns(ns);
>  }
>  
> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
> index 23573fe3fc7d..31153f6ec582 100644
> --- a/drivers/nvme/host/multipath.c
> +++ b/drivers/nvme/host/multipath.c
> @@ -266,6 +266,8 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
>  	int node = numa_node_id();
>  	struct nvme_ns *ns;
>  
> +	if (!(head->disk->flags & GENHD_FL_UP))
> +		return NULL;
>  	ns = srcu_dereference(head->current_path[node], &head->srcu);
>  	if (unlikely(!ns))
>  		return __nvme_find_path(head, node);
> @@ -281,6 +283,8 @@ static bool nvme_available_path(struct nvme_ns_head *head)
>  {
>  	struct nvme_ns *ns;
>  
> +	if (!(head->disk->flags & GENHD_FL_UP))
> +		return false;
>  	list_for_each_entry_rcu(ns, &head->list, siblings) {
>  		if (test_bit(NVME_CTRL_FAILFAST_EXPIRED, &ns->ctrl->flags))
>  			continue;
> @@ -771,20 +775,36 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id)
>  #endif
>  }
>  
> -void nvme_mpath_remove_disk(struct nvme_ns_head *head)
> +void nvme_mpath_check_last_path(struct nvme_ns_head *head)
>  {
> +	bool last_path = false;
>  	if (!head->disk)
>  		return;
> +
> +	/* Synchronize with nvme_init_ns_head() */
> +	mutex_lock(&head->subsys->lock);
> +	if (list_empty(&head->list))
> +		last_path = true;
> +	mutex_unlock(&head->subsys->lock);
> +	if (last_path) {
> +		kblockd_schedule_work(&head->requeue_work);
> +		if (head->disk->flags & GENHD_FL_UP) {
> +			nvme_cdev_del(&head->cdev, &head->cdev_device);
> +			del_gendisk(head->disk);
> +		}
>  	}
> +}
> +
> +void nvme_mpath_remove_disk(struct nvme_ns_head *head)
> +{
> +	if (!head->disk)
> +		return;
>  	blk_set_queue_dying(head->disk->queue);
>  	/* make sure all pending bios are cleaned up */
>  	kblockd_schedule_work(&head->requeue_work);
>  	flush_work(&head->requeue_work);
>  	blk_cleanup_queue(head->disk->queue);
> -	if (!test_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
> +	if (!test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
>  		/*
>  		 * if device_add_disk wasn't called, prevent
>  		 * disk release to put a bogus reference on the

So if a nvme_mpath_set_live comes in between nvme_mpath_check_last_path
and nvme_mpath_remove_disk we'll end up without a gendisk still, don't we?


From hch at lst.de  Sun Jun 20 23:39:12 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 21 Jun 2021 08:39:12 +0200
Subject: [PATCH] nvmet: use NVMET_MAX_NAMESPACES to set nn value
In-Reply-To: <20210621030109.20695-1-chaitanya.kulkarni@wdc.com>
References: <20210621030109.20695-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210621063912.GA5817@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Sun Jun 20 23:45:29 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 21 Jun 2021 08:45:29 +0200
Subject: [PATCH v2 0/8] NVMeTCP Offload ULP
In-Reply-To: <20210621051046.28783-1-smalin@marvell.com>
References: <20210621051046.28783-1-smalin@marvell.com>
Message-ID: <20210621064528.GA5906@lst.de>

This seems to lack drivers actually using the infrastructure.  That is not
only completly not acceptable for merging code into the kernel, but also
make the code impossible to review.


From hch at lst.de  Sun Jun 20 23:49:56 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 21 Jun 2021 08:49:56 +0200
Subject: [PATCH v2 3/8] nvme-fabrics: Expose nvmf_check_required_opts()
 globally
In-Reply-To: <20210621051046.28783-4-smalin@marvell.com>
References: <20210621051046.28783-1-smalin@marvell.com>
 <20210621051046.28783-4-smalin@marvell.com>
Message-ID: <20210621064956.GA6019@lst.de>

On Mon, Jun 21, 2021 at 08:10:41AM +0300, Shai Malin wrote:
> From: Prabhakar Kushwaha <pkushwaha at marvell.com>
> 
> nvmf_check_required_opts() is used to check if user provided opts has
> the required_opts or not. if not, it will log which options are not
> provided.
> 
> It can be leveraged by nvme-tcp-offload to check if provided opts are
> supported by this specific vendor driver or not.

WTF is a "vendor" driver?  Vendors are not really a concept we use anywhere
in Linux.


From hch at lst.de  Sun Jun 20 23:50:24 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 21 Jun 2021 08:50:24 +0200
Subject: [PATCH v2 4/8] nvme-tcp-offload: Add device scan implementation
In-Reply-To: <20210621051046.28783-5-smalin@marvell.com>
References: <20210621051046.28783-1-smalin@marvell.com>
 <20210621051046.28783-5-smalin@marvell.com>
Message-ID: <20210621065024.GB6019@lst.de>

On Mon, Jun 21, 2021 at 08:10:42AM +0300, Shai Malin wrote:
> +static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nctrl)

Please fix all these overly long lines in the whole series.


From hch at lst.de  Mon Jun 21 00:20:05 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 21 Jun 2021 09:20:05 +0200
Subject: [PATCH 13/16] block: switch polling to be bio based
In-Reply-To: <YMytSIBm7k2pqFlc@T590>
References: <20210615131034.752623-1-hch@lst.de>
 <20210615131034.752623-14-hch@lst.de> <YMliP6sFVuPhMbOB@T590>
 <20210618140147.GA16258@lst.de> <YMytSIBm7k2pqFlc@T590>
Message-ID: <20210621072005.GA6651@lst.de>

On Fri, Jun 18, 2021 at 10:27:20PM +0800, Ming Lei wrote:
> > How?  On a block device the caller needs to hold the block device open
> > to read/write from it.  On a file systems the file systems needs to
> > be mounted, which also holds a bdev reference.
> 
> +       rcu_read_lock();
> +       bio = READ_ONCE(kiocb->private);
> +       if (bio && bio->bi_bdev)
> 
> The bio may be ended now from another polling job, then the disk is
> closed & deleted, and released. Then request queue & hctxs are released.
> 
> +               ret = bio_poll(bio, flags);
> 
> But disk & request queue & hctx can still be referred in above bio_poll().

I don't see how this can happen.  A bio stashed into kiocb->private needs
to belong to the correct device initially.  For it to point to the "wrong"
device it needs to have been completed on the correct one, and then be
reused for a different device.  At the point it is reused that device
must obviously have been alive, and for it to be freed a RCU grace
period must have been passed.  And that grace period can't have started
earlier than when iocb_bio_iopoll was called.


From hare at suse.de  Mon Jun 21 00:33:24 2021
From: hare at suse.de (Hannes Reinecke)
Date: Mon, 21 Jun 2021 09:33:24 +0200
Subject: [PATCHv6] nvme: allow to re-attach namespaces after all paths are
 down
In-Reply-To: <20210621063820.GA5618@lst.de>
References: <20210609150118.130650-1-hare@suse.de>
 <20210621063820.GA5618@lst.de>
Message-ID: <9c02c4bb-992d-3be1-8e66-1c5aa90de515@suse.de>

On 6/21/21 8:38 AM, Christoph Hellwig wrote:
> On Wed, Jun 09, 2021 at 05:01:18PM +0200, Hannes Reinecke wrote:
>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>> index 177cae44b612..6d7c2958b3e2 100644
>> --- a/drivers/nvme/host/core.c
>> +++ b/drivers/nvme/host/core.c
>> @@ -566,6 +566,9 @@ static void nvme_free_ns_head(struct kref *ref)
>>   	struct nvme_ns_head *head =
>>   		container_of(ref, struct nvme_ns_head, ref);
>>   
>> +	mutex_lock(&head->subsys->lock);
>> +	list_del_init(&head->entry);
>> +	mutex_unlock(&head->subsys->lock);
>>   	nvme_mpath_remove_disk(head);
>>   	ida_simple_remove(&head->subsys->ns_ida, head->instance);
>>   	cleanup_srcu_struct(&head->srcu);
>> @@ -3806,8 +3809,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid,
>>    out_unlink_ns:
>>   	mutex_lock(&ctrl->subsys->lock);
>>   	list_del_rcu(&ns->siblings);
>> -	if (list_empty(&ns->head->list))
>> -		list_del_init(&ns->head->entry);
>>   	mutex_unlock(&ctrl->subsys->lock);
>>   	nvme_put_ns_head(ns->head);
>>    out_free_queue:
>> @@ -3828,8 +3829,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>>   
>>   	mutex_lock(&ns->ctrl->subsys->lock);
>>   	list_del_rcu(&ns->siblings);
>> -	if (list_empty(&ns->head->list))
>> -		list_del_init(&ns->head->entry);
>>   	mutex_unlock(&ns->ctrl->subsys->lock);
>>   
>>   	synchronize_rcu(); /* guarantee not available in head->list */
>> @@ -3849,7 +3848,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>>   	list_del_init(&ns->list);
>>   	up_write(&ns->ctrl->namespaces_rwsem);
>>   
>> -	nvme_mpath_check_last_path(ns);
>> +	nvme_mpath_check_last_path(ns->head);
>>   	nvme_put_ns(ns);
>>   }
>>   
>> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
>> index 23573fe3fc7d..31153f6ec582 100644
>> --- a/drivers/nvme/host/multipath.c
>> +++ b/drivers/nvme/host/multipath.c
>> @@ -266,6 +266,8 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
>>   	int node = numa_node_id();
>>   	struct nvme_ns *ns;
>>   
>> +	if (!(head->disk->flags & GENHD_FL_UP))
>> +		return NULL;
>>   	ns = srcu_dereference(head->current_path[node], &head->srcu);
>>   	if (unlikely(!ns))
>>   		return __nvme_find_path(head, node);
>> @@ -281,6 +283,8 @@ static bool nvme_available_path(struct nvme_ns_head *head)
>>   {
>>   	struct nvme_ns *ns;
>>   
>> +	if (!(head->disk->flags & GENHD_FL_UP))
>> +		return false;
>>   	list_for_each_entry_rcu(ns, &head->list, siblings) {
>>   		if (test_bit(NVME_CTRL_FAILFAST_EXPIRED, &ns->ctrl->flags))
>>   			continue;
>> @@ -771,20 +775,36 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id)
>>   #endif
>>   }
>>   
>> -void nvme_mpath_remove_disk(struct nvme_ns_head *head)
>> +void nvme_mpath_check_last_path(struct nvme_ns_head *head)
>>   {
>> +	bool last_path = false;
>>   	if (!head->disk)
>>   		return;
>> +
>> +	/* Synchronize with nvme_init_ns_head() */
>> +	mutex_lock(&head->subsys->lock);
>> +	if (list_empty(&head->list))
>> +		last_path = true;
>> +	mutex_unlock(&head->subsys->lock);
>> +	if (last_path) {
>> +		kblockd_schedule_work(&head->requeue_work);
>> +		if (head->disk->flags & GENHD_FL_UP) {
>> +			nvme_cdev_del(&head->cdev, &head->cdev_device);
>> +			del_gendisk(head->disk);
>> +		}
>>   	}
>> +}
>> +
>> +void nvme_mpath_remove_disk(struct nvme_ns_head *head)
>> +{
>> +	if (!head->disk)
>> +		return;
>>   	blk_set_queue_dying(head->disk->queue);
>>   	/* make sure all pending bios are cleaned up */
>>   	kblockd_schedule_work(&head->requeue_work);
>>   	flush_work(&head->requeue_work);
>>   	blk_cleanup_queue(head->disk->queue);
>> -	if (!test_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
>> +	if (!test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
>>   		/*
>>   		 * if device_add_disk wasn't called, prevent
>>   		 * disk release to put a bogus reference on the
> 
> So if a nvme_mpath_set_live comes in between nvme_mpath_check_last_path
> and nvme_mpath_remove_disk we'll end up without a gendisk still, don't we?
> 
I can't see how we can end up there.
If we call nvme_mpath_set_live() we will have to have a reference on 
ns_head, consequently nvme_mpath_remove_disk() will not be called for 
that ns_head as that function is called from the kref callback once all 
references are dropped.

Am I missing something?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From ming.lei at redhat.com  Mon Jun 21 01:35:29 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Mon, 21 Jun 2021 16:35:29 +0800
Subject: [PATCH 13/16] block: switch polling to be bio based
In-Reply-To: <20210621072005.GA6651@lst.de>
References: <20210615131034.752623-1-hch@lst.de>
 <20210615131034.752623-14-hch@lst.de> <YMliP6sFVuPhMbOB@T590>
 <20210618140147.GA16258@lst.de> <YMytSIBm7k2pqFlc@T590>
 <20210621072005.GA6651@lst.de>
Message-ID: <YNBPUUVtxSb6/gv1@T590>

On Mon, Jun 21, 2021 at 09:20:05AM +0200, Christoph Hellwig wrote:
> On Fri, Jun 18, 2021 at 10:27:20PM +0800, Ming Lei wrote:
> > > How?  On a block device the caller needs to hold the block device open
> > > to read/write from it.  On a file systems the file systems needs to
> > > be mounted, which also holds a bdev reference.
> > 
> > +       rcu_read_lock();
> > +       bio = READ_ONCE(kiocb->private);
> > +       if (bio && bio->bi_bdev)
> > 
> > The bio may be ended now from another polling job, then the disk is
> > closed & deleted, and released. Then request queue & hctxs are released.
> > 
> > +               ret = bio_poll(bio, flags);
> > 
> > But disk & request queue & hctx can still be referred in above bio_poll().
> 
> I don't see how this can happen.  A bio stashed into kiocb->private needs
> to belong to the correct device initially.  For it to point to the "wrong"
> device it needs to have been completed on the correct one, and then be
> reused for a different device.  At the point it is reused that device
> must obviously have been alive, and for it to be freed a RCU grace
> period must have been passed.  And that grace period can't have started
> earlier than when iocb_bio_iopoll was called.

gendisk isn't freed after a RCU grace period, so even though bio->bi_bdev
may not be freed really, but the gendisk may have been freed already.

+       rcu_read_lock();
+       bio = READ_ONCE(kiocb->private);
+       if (bio && bio->bi_bdev)

The bio may be ended now from another polling job, and it is freed
and re-allocated & freed, then bio->bi_bdev->bd_disk is freed too, which
will be observed in the following bio_poll().

+               ret = bio_poll(bio, flags);


Thanks,
Ming



From lkml at metux.net  Mon Jun 21 03:18:15 2021
From: lkml at metux.net (Enrico Weigelt, metux IT consult)
Date: Mon, 21 Jun 2021 12:18:15 +0200
Subject: [PATCH] drivers/nvme/host: Use kobj_to_dev() API
In-Reply-To: <20210618141817.GA18781@lst.de>
References: <1624025458-56465-1-git-send-email-zhouchuangao@vivo.com>
 <20210618141817.GA18781@lst.de>
Message-ID: <fd15a968-3d75-ba22-b0b0-4ed4d2a66372@metux.net>

On 18.06.21 16:18, Christoph Hellwig wrote:
> On Fri, Jun 18, 2021 at 07:10:58AM -0700, zhouchuangao wrote:
>> Use kobj_to_dev() API instead of container_of().
> 
> Why?  That just makes the code harder to read.

Why so ? Why to we have kobj_to_dev() at all ?
Do you suggest, we shall get rid of it (and all similar ones) ?

--mtx

-- 
---
Hinweis: unverschl?sselte E-Mails k?nnen leicht abgeh?rt und manipuliert
werden ! F?r eine vertrauliche Kommunikation senden Sie bitte ihren
GPG/PGP-Schl?ssel zu.
---
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info at metux.net -- +49-151-27565287


From lkp at intel.com  Mon Jun 21 07:49:33 2021
From: lkp at intel.com (kernel test robot)
Date: Mon, 21 Jun 2021 22:49:33 +0800
Subject: [linux-nvme:nvme-5.14 38/38] make[2]: *** No rule to make target
 '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-sharp-ls037v7dw01.ko.gz',
 needed by '__modinst'.
Message-ID: <202106212228.T0arIj5F-lkp@intel.com>

Hi Chaitanya,

First bad commit (maybe != root cause):

tree:   git://git.infradead.org/nvme.git nvme-5.14
head:   3c3ee16532c1be92350a2a88bd19283b7bdf32e9
commit: 3c3ee16532c1be92350a2a88bd19283b7bdf32e9 [38/38] nvmet: use NVMET_MAX_NAMESPACES to set nn value
config: arm-randconfig-c023-20210621 (attached as .config)
compiler: arm-linux-gnueabi-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        git remote add linux-nvme git://git.infradead.org/nvme.git
        git fetch --no-tags linux-nvme nvme-5.14
        git checkout 3c3ee16532c1be92350a2a88bd19283b7bdf32e9
        # save the attached .config to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=arm 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp at intel.com>

All errors (new ones prefixed by >>):

   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_iblock.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_mod.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_pscsi.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/tee/tee.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/bcm2711_thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/bcm2835_thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/brcmstb_thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/da9062-thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/dove_thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/hisi_thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/imx8mm_thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/imx_thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/k3_bandgap.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/khadas_mcu_fan.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/mtk_thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/rcar_gen3_thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/sprd_thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/sun8i_thermal.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/tty/serdev/serdev.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/uio/uio_dmem_genirq.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns-usb-common.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3-imx.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3-ti.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_imx.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_tegra.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/usbmisc_imx.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/dwc2/dwc2.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_fs.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_hid.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_printer.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_tcm.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_ffs.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_hid.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_printer.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/gadgetfs.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/raw_gadget.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/libcomposite.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/aspeed-vhub/aspeed-vhub.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/fotg210-udc.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/gr_udc.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/lpc32xx_udc.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/mv_udc.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/net2272.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/renesas_usb3.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/snps_udc_core.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/snps_udc_plat.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/udc-core.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/udc-xilinx.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/isp1760/isp1760.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/mtu3/mtu3.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/jz4740.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/musb_dsps.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/musb_hdrc.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/tusb6010.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-am335x-control.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-am335x.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-generic.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-isp1301.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-tahvo.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/mdev/mdev.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/vfio.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/vfio_iommu_type1.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/adp8860_bl.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/adp8870_bl.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/arcxcnn_bl.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/backlight.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/da903x_bl.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/led_bl.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lm3533_bl.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lm3630a_bl.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lp855x_bl.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lv5207lp.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/pwm_bl.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/qcom-wled.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/sky81452-backlight.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/tps65217_bl.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/wm831x_bl.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/arcfb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/atmel_lcdfb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/broadsheetfb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/core/fb_sys_fops.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/goldfishfb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/imxfb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/mmp/fb/mmpfb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/mmp/mmp_disp.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/connector-dvi.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/connector-hdmi.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/encoder-opa362.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-dpi.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-dsi-cm.ko.gz', needed by '__modinst'.
>> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-sharp-ls037v7dw01.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pvr2fb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pxa168fb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pxa3xx-gcu.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/s3c-fb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/sh_mobile_lcdcfb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/ssd1307fb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/vfb.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/ds1wm.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/ds2482.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/mxc_w1.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/w1-gpio.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds2408.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds2438.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds28e17.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_smem.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/adfs/adfs.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/cramfs/cramfs.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/exfat/exfat.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ext2/ext2.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ext4/ext4-inode-test.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/fat/vfat.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/freevxfs/freevxfs.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/gfs2/gfs2.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/hfs/hfs.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/hfsplus/hfsplus.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-centeuro.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-croatian.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-cyrillic.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-gaelic.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-roman.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_ascii.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp1250.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp1255.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp437.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp850.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp857.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp860.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp869.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp932.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp936.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp950.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_euc-jp.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_iso8859-1.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_iso8859-14.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_iso8859-15.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_iso8859-3.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_utf8.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ntfs/ntfs.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/overlayfs/overlay.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/qnx6/qnx6.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/quota/quota_v1.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/squashfs/squashfs.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/unicode/utf8-selftest.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/configs.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/locking/test-ww_mutex.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/resource_kunit.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/trace/ring_buffer_benchmark.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/bitfield_kunit.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crc8.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crypto/libchacha20poly1305.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crypto/libcurve25519-generic.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/globtest.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/kunit/kunit.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/lz4/lz4_compress.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/lz4/lz4_decompress.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/math/prime_numbers.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/math/test_div64.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/notifier-error-inject.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/parman.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/percpu_test.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/rbtree_test.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_bitmap.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_bits.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_free_pages.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_linear_ranges.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_meminit.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_module.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_parman.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_sort.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_stackinit.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_ubsan.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_uuid.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/z3fold.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/zbud.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/zpool.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/configfs/configfs_sample.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/bytestream-example.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/dma-example.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/inttype-example.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/record-example.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/rpmsg/rpmsg_client_sample.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/trace_events/trace-events-sample.ko.gz', needed by '__modinst'.
   make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/vfio-mdev/mdpy-fb.ko.gz', needed by '__modinst'.
   make[2]: Target '__modinst' not remade because of errors.
   make[1]: *** [Makefile:1765: modules_install] Error 2
   make: *** [Makefile:215: __sub-make] Error 2
   make: Target 'modules_install' not remade because of errors.

Kconfig warnings: (for reference only)
   WARNING: unmet direct dependencies detected for QCOM_SCM
   Depends on (ARM || ARM64) && HAVE_ARM_SMCCC
   Selected by
   - ARM_QCOM_SPM_CPUIDLE && CPU_IDLE && (ARM || ARM64) && (ARCH_QCOM || COMPILE_TEST && !ARM64 && MMU

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 28503 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20210621/0062881a/attachment-0001.gz>

From philip.li at intel.com  Mon Jun 21 08:22:09 2021
From: philip.li at intel.com (Philip Li)
Date: Mon, 21 Jun 2021 23:22:09 +0800
Subject: [kbuild-all] [linux-nvme:nvme-5.14 38/38] make[2]: *** No rule
 to make target
 '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-sharp-ls037v7dw01.ko.gz',
 needed by '__modinst'.
In-Reply-To: <202106212228.T0arIj5F-lkp@intel.com>
References: <202106212228.T0arIj5F-lkp@intel.com>
Message-ID: <20210621152209.GB158568@pl-dbox>

On Mon, Jun 21, 2021 at 10:49:33PM +0800, kernel test robot wrote:
> Hi Chaitanya,
> 
> First bad commit (maybe != root cause):
> 
> tree:   git://git.infradead.org/nvme.git nvme-5.14
> head:   3c3ee16532c1be92350a2a88bd19283b7bdf32e9
> commit: 3c3ee16532c1be92350a2a88bd19283b7bdf32e9 [38/38] nvmet: use NVMET_MAX_NAMESPACES to set nn value
> config: arm-randconfig-c023-20210621 (attached as .config)
> compiler: arm-linux-gnueabi-gcc (GCC) 9.3.0
> reproduce (this is a W=1 build):
>         wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>         chmod +x ~/bin/make.cross
>         git remote add linux-nvme git://git.infradead.org/nvme.git
>         git fetch --no-tags linux-nvme nvme-5.14
>         git checkout 3c3ee16532c1be92350a2a88bd19283b7bdf32e9
>         # save the attached .config to linux build tree
>         COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=arm 
> 
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp at intel.com>
> 
> All errors (new ones prefixed by >>):
Sorry for the broken report, kindly ignore this, we will fix
this asap.

> 
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_iblock.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_mod.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_pscsi.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/tee/tee.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/bcm2711_thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/bcm2835_thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/brcmstb_thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/da9062-thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/dove_thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/hisi_thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/imx8mm_thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/imx_thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/k3_bandgap.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/khadas_mcu_fan.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/mtk_thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/rcar_gen3_thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/sprd_thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/sun8i_thermal.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/tty/serdev/serdev.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/uio/uio_dmem_genirq.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns-usb-common.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3-imx.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3-ti.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_imx.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_tegra.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/usbmisc_imx.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/dwc2/dwc2.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_fs.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_hid.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_printer.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_tcm.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_ffs.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_hid.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_printer.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/gadgetfs.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/raw_gadget.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/libcomposite.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/aspeed-vhub/aspeed-vhub.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/fotg210-udc.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/gr_udc.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/lpc32xx_udc.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/mv_udc.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/net2272.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/renesas_usb3.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/snps_udc_core.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/snps_udc_plat.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/udc-core.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/udc-xilinx.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/isp1760/isp1760.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/mtu3/mtu3.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/jz4740.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/musb_dsps.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/musb_hdrc.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/tusb6010.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-am335x-control.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-am335x.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-generic.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-isp1301.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-tahvo.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/mdev/mdev.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/vfio.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/vfio_iommu_type1.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/adp8860_bl.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/adp8870_bl.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/arcxcnn_bl.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/backlight.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/da903x_bl.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/led_bl.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lm3533_bl.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lm3630a_bl.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lp855x_bl.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lv5207lp.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/pwm_bl.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/qcom-wled.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/sky81452-backlight.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/tps65217_bl.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/wm831x_bl.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/arcfb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/atmel_lcdfb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/broadsheetfb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/core/fb_sys_fops.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/goldfishfb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/imxfb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/mmp/fb/mmpfb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/mmp/mmp_disp.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/connector-dvi.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/connector-hdmi.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/encoder-opa362.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-dpi.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-dsi-cm.ko.gz', needed by '__modinst'.
> >> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-sharp-ls037v7dw01.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pvr2fb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pxa168fb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pxa3xx-gcu.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/s3c-fb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/sh_mobile_lcdcfb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/ssd1307fb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/vfb.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/ds1wm.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/ds2482.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/mxc_w1.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/w1-gpio.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds2408.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds2438.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds28e17.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_smem.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/adfs/adfs.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/cramfs/cramfs.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/exfat/exfat.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ext2/ext2.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ext4/ext4-inode-test.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/fat/vfat.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/freevxfs/freevxfs.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/gfs2/gfs2.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/hfs/hfs.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/hfsplus/hfsplus.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-centeuro.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-croatian.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-cyrillic.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-gaelic.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-roman.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_ascii.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp1250.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp1255.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp437.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp850.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp857.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp860.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp869.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp932.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp936.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp950.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_euc-jp.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_iso8859-1.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_iso8859-14.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_iso8859-15.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_iso8859-3.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_utf8.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ntfs/ntfs.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/overlayfs/overlay.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/qnx6/qnx6.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/quota/quota_v1.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/squashfs/squashfs.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/unicode/utf8-selftest.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/configs.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/locking/test-ww_mutex.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/resource_kunit.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/trace/ring_buffer_benchmark.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/bitfield_kunit.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crc8.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crypto/libchacha20poly1305.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crypto/libcurve25519-generic.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/globtest.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/kunit/kunit.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/lz4/lz4_compress.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/lz4/lz4_decompress.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/math/prime_numbers.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/math/test_div64.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/notifier-error-inject.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/parman.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/percpu_test.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/rbtree_test.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_bitmap.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_bits.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_free_pages.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_linear_ranges.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_meminit.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_module.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_parman.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_sort.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_stackinit.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_ubsan.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_uuid.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/z3fold.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/zbud.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/zpool.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/configfs/configfs_sample.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/bytestream-example.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/dma-example.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/inttype-example.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/record-example.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/rpmsg/rpmsg_client_sample.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/trace_events/trace-events-sample.ko.gz', needed by '__modinst'.
>    make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/vfio-mdev/mdpy-fb.ko.gz', needed by '__modinst'.
>    make[2]: Target '__modinst' not remade because of errors.
>    make[1]: *** [Makefile:1765: modules_install] Error 2
>    make: *** [Makefile:215: __sub-make] Error 2
>    make: Target 'modules_install' not remade because of errors.
> 
> Kconfig warnings: (for reference only)
>    WARNING: unmet direct dependencies detected for QCOM_SCM
>    Depends on (ARM || ARM64) && HAVE_ARM_SMCCC
>    Selected by
>    - ARM_QCOM_SPM_CPUIDLE && CPU_IDLE && (ARM || ARM64) && (ARCH_QCOM || COMPILE_TEST && !ARM64 && MMU
> 
> ---
> 0-DAY CI Kernel Test Service, Intel Corporation
> https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org


> _______________________________________________
> kbuild-all mailing list -- kbuild-all at lists.01.org
> To unsubscribe send an email to kbuild-all-leave at lists.01.org



From jonathan at knownhost.com  Mon Jun 21 09:04:18 2021
From: jonathan at knownhost.com (Jonathan Wright)
Date: Mon, 21 Jun 2021 11:04:18 -0500
Subject: nvmeof Issues with Zen 3/Ryzen 5000 Initiator
In-Reply-To: <69f0c4b6-c890-ef25-ddae-6d15d38881e0@knownhost.com>
References: <ffae89d5-05d8-e0a4-ca71-b3db18f70286@knownhost.com>
 <39fabb5d-537c-febf-a538-ff6215777da5@grimberg.me>
 <69f0c4b6-c890-ef25-ddae-6d15d38881e0@knownhost.com>
Message-ID: <06ac1cd6-a53d-a976-5779-07647c043e8b@knownhost.com>


On 6/3/21 11:57 AM, Jonathan Wright wrote:
>
>>> I've been testing NVMe over Fabrics for the past few weeks and the 
>>> performance has been nothing short of incredible, though I'm running 
>>> into some major issues that seems to be specifically related to AMD 
>>> Zen 3 Ryzen chips (in my case I'm testing with 5900x).
>>>
>>> Target:
>>> Supermicro X10 board
>>> Xeon E5-2620v4
>>> Intel E810 NIC
>>>
>>> Problematic Client/initiator:
>>> ASRock X570 board
>>> Ryzen 9 5900x
>>> Intel E810 NIC
>>>
>>> Stable Client/initiator:
>>> Supermicro X10 board
>>> Xeon E5-2620v4
>>> Intel E810 NIC
>>>
>>> I'm using the same 2 E810 NICs and pair of 25G DACs in both cases.? 
>>> The NICs are directly connected with the DACs and there is no switch 
>>> in the equation.? To trigger the issue I'm simply using FIO similar 
>>> to this:
>>>
>>> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 
>>> --name=test --filename=/dev/nvme0n1 --bs=4k --iodepth=64 --size=10G 
>>> --readwrite=randread --time_based --runtime=1200
>>>
>>> I'm primarily using RDMA/iWARP right now but I've also tested RoCE2 
>>> which presents the same issues/symptoms.? Primary testing has been 
>>> done with Ubuntu 20.04.2 with CentOS 8 in the mix as well just to 
>>> try and rule out a weird distro-specific issue. All tests used the 
>>> latest ice/irdma drivers from Intel (1.5.8 and 1.5.2 respectively)
>>
>> CCing Shiraz Saleem who maintains irdma.
>
> Thanks.? I've done some testing now with Mellanox ConnectX-4 cards 
> with Zen 3 and the issue does not exist.? This seems to point the 
> finger to something specific between irdma and Zen 2/3 since 
> irdma/E810 works fine on all-Intel hardware.? I tested the Mellanox on 
> both Ubuntu 20.04 stock kernel (5.4) and CentOS 8.3 (stock kernel 
> 4.18) as I tested the E810 on these combinations.
>
> Further I tested an AMD target with an Intel initiator and the issue 
> still exists so it doesn't seem to matter which end the Zen 3 (and/or 
> Zen2) chip is on when paired with an E810/irdma.
>
> The issue also exists with Zen 2 (Ryzen 3600).
>
> @Shriaz since I guess this isn't a common setup right now let me know 
> if I can be of any assistance with getting to the bottom of this 
> seeming incompatibility.

@Intel is there any desire to fix this currently?? I'll be sending this 
current lab equipment with the Zen3 systems to other tasks soon and will 
no longer be able to help troubleshoot or test.? Without a fix we'll 
have to look into other vendors for fabric setups.



From jonathan at knownhost.com  Mon Jun 21 09:06:26 2021
From: jonathan at knownhost.com (Jonathan Wright)
Date: Mon, 21 Jun 2021 11:06:26 -0500
Subject: nvmeof Issues with Zen 3/Ryzen 5000 Initiator
In-Reply-To: <06ac1cd6-a53d-a976-5779-07647c043e8b@knownhost.com>
References: <ffae89d5-05d8-e0a4-ca71-b3db18f70286@knownhost.com>
 <39fabb5d-537c-febf-a538-ff6215777da5@grimberg.me>
 <69f0c4b6-c890-ef25-ddae-6d15d38881e0@knownhost.com>
 <06ac1cd6-a53d-a976-5779-07647c043e8b@knownhost.com>
Message-ID: <f2d12959-3d03-83a6-4c4c-2603e4a7b612@knownhost.com>

Adding Nikolova, Tatyana E and Ismail, Mustafa to CC per auto-responder 
from @Shiraz.

On 6/21/21 11:04 AM, Jonathan Wright wrote:
>
> On 6/3/21 11:57 AM, Jonathan Wright wrote:
>>
>>>> I've been testing NVMe over Fabrics for the past few weeks and the 
>>>> performance has been nothing short of incredible, though I'm 
>>>> running into some major issues that seems to be specifically 
>>>> related to AMD Zen 3 Ryzen chips (in my case I'm testing with 5900x).
>>>>
>>>> Target:
>>>> Supermicro X10 board
>>>> Xeon E5-2620v4
>>>> Intel E810 NIC
>>>>
>>>> Problematic Client/initiator:
>>>> ASRock X570 board
>>>> Ryzen 9 5900x
>>>> Intel E810 NIC
>>>>
>>>> Stable Client/initiator:
>>>> Supermicro X10 board
>>>> Xeon E5-2620v4
>>>> Intel E810 NIC
>>>>
>>>> I'm using the same 2 E810 NICs and pair of 25G DACs in both cases.? 
>>>> The NICs are directly connected with the DACs and there is no 
>>>> switch in the equation.? To trigger the issue I'm simply using FIO 
>>>> similar to this:
>>>>
>>>> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 
>>>> --name=test --filename=/dev/nvme0n1 --bs=4k --iodepth=64 --size=10G 
>>>> --readwrite=randread --time_based --runtime=1200
>>>>
>>>> I'm primarily using RDMA/iWARP right now but I've also tested RoCE2 
>>>> which presents the same issues/symptoms. Primary testing has been 
>>>> done with Ubuntu 20.04.2 with CentOS 8 in the mix as well just to 
>>>> try and rule out a weird distro-specific issue. All tests used the 
>>>> latest ice/irdma drivers from Intel (1.5.8 and 1.5.2 respectively)
>>>
>>> CCing Shiraz Saleem who maintains irdma.
>>
>> Thanks.? I've done some testing now with Mellanox ConnectX-4 cards 
>> with Zen 3 and the issue does not exist.? This seems to point the 
>> finger to something specific between irdma and Zen 2/3 since 
>> irdma/E810 works fine on all-Intel hardware.? I tested the Mellanox 
>> on both Ubuntu 20.04 stock kernel (5.4) and CentOS 8.3 (stock kernel 
>> 4.18) as I tested the E810 on these combinations.
>>
>> Further I tested an AMD target with an Intel initiator and the issue 
>> still exists so it doesn't seem to matter which end the Zen 3 (and/or 
>> Zen2) chip is on when paired with an E810/irdma.
>>
>> The issue also exists with Zen 2 (Ryzen 3600).
>>
>> @Shriaz since I guess this isn't a common setup right now let me know 
>> if I can be of any assistance with getting to the bottom of this 
>> seeming incompatibility.
>
> @Intel is there any desire to fix this currently?? I'll be sending 
> this current lab equipment with the Zen3 systems to other tasks soon 
> and will no longer be able to help troubleshoot or test. Without a fix 
> we'll have to look into other vendors for fabric setups.
>
-- 
Jonathan Wright
KnownHost, LLC
https://www.knownhost.com



From mgurtovoy at nvidia.com  Mon Jun 21 09:11:12 2021
From: mgurtovoy at nvidia.com (Max Gurtovoy)
Date: Mon, 21 Jun 2021 19:11:12 +0300
Subject: [PATCH 1/3] mm,memory_hotplug: export mhp min alignment
In-Reply-To: <46524325-8e2a-9395-e0d0-7d559c753c67@nvidia.com>
References: <20210602111055.10480-1-mgurtovoy@nvidia.com>
 <20210602111055.10480-2-mgurtovoy@nvidia.com>
 <283740c3-db3f-3c9a-2954-f1c037a13e86@redhat.com>
 <46524325-8e2a-9395-e0d0-7d559c753c67@nvidia.com>
Message-ID: <7a4acae5-b7f8-9312-8191-aece7270104f@nvidia.com>

hi David,

do we have a conclusion for this series ?

Is the below suggestion accepted by the maintainers ?

I would like to send a new series before closing 5.14 merge window.

On 6/3/2021 1:52 PM, Max Gurtovoy wrote:
>
> On 6/2/2021 3:14 PM, David Hildenbrand wrote:
>> On 02.06.21 13:10, Max Gurtovoy wrote:
>>> Hotplugged memory has alignmet restrictions. E.g, it disallows all
>>> operations smaller than a sub-section and only allow operations smaller
>>> than a section for SPARSEMEM_VMEMMAP. Export the alignment restrictions
>>> for mhp users.
>>>
>>> Signed-off-by: Max Gurtovoy <mgurtovoy at nvidia.com>
>>> ---
>>> ? include/linux/memory_hotplug.h |? 5 +++++
>>> ? mm/memory_hotplug.c??????????? | 33 +++++++++++++++++++--------------
>>> ? 2 files changed, 24 insertions(+), 14 deletions(-)
>>>
>>> diff --git a/include/linux/memory_hotplug.h 
>>> b/include/linux/memory_hotplug.h
>>> index 28f32fd00fe9..c55a9049b11e 100644
>>> --- a/include/linux/memory_hotplug.h
>>> +++ b/include/linux/memory_hotplug.h
>>> @@ -76,6 +76,7 @@ struct mhp_params {
>>> ? ? bool mhp_range_allowed(u64 start, u64 size, bool need_mapping);
>>> ? struct range mhp_get_pluggable_range(bool need_mapping);
>>> +unsigned long mhp_get_min_align(void);
>>> ? ? /*
>>> ?? * Zone resizing functions
>>> @@ -248,6 +249,10 @@ void mem_hotplug_done(void);
>>> ????? ___page;??????????????? \
>>> ?? })
>>> ? +static inline unsigned long mhp_get_min_align(void)
>>> +{
>>> +??? return 0;
>>> +}
>>> ? static inline unsigned zone_span_seqbegin(struct zone *zone)
>>> ? {
>>> ????? return 0;
>>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>>> index 9e86e9ee0a10..161bb6704a9b 100644
>>> --- a/mm/memory_hotplug.c
>>> +++ b/mm/memory_hotplug.c
>>> @@ -270,24 +270,29 @@ void __init 
>>> register_page_bootmem_info_node(struct pglist_data *pgdat)
>>> ? }
>>> ? #endif /* CONFIG_HAVE_BOOTMEM_INFO_NODE */
>>> ? +/*
>>> + * Disallow all operations smaller than a sub-section and only
>>> + * allow operations smaller than a section for
>>> + * SPARSEMEM_VMEMMAP. Note that check_hotplug_memory_range()
>>> + * enforces a larger memory_block_size_bytes() granularity for
>>> + * memory that will be marked online, so this check should only
>>> + * fire for direct arch_{add,remove}_memory() users outside of
>>> + * add_memory_resource().
>>> + */
>>> +unsigned long mhp_get_min_align(void)
>>> +{
>>> +??? if (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP))
>>> +??????? return PAGES_PER_SUBSECTION;
>>> +??? return PAGES_PER_SECTION;
>>> +}
>>> +EXPORT_SYMBOL_GPL(mhp_get_min_align);
>>
>> We have to main interfaces to "hotplug" memory:
>>
>> a) add_memory() and friends for System RAM, which have memory block 
>> alignment requirements.
>>
>> b) memremap_pages(), which has the alignemnt requirements you mention 
>> here.
>>
>> I feel like what you need would better be exposed in mm/memremap.c, 
>> for example, via "memremap_min_alignment" so it matches the 
>> "memremap_pages" semantics.
>>
>> And then, memremap_pages() is only available with CONFIG_ZONE_DEVICE, 
>> which depends on SPARSEMEM_VMEMMAP. So you'll always have 
>> PAGES_PER_SUBSECTION.
>>
>> I can already spot "memremap_compat_align", maybe you can reuse that 
>> or handle it accordingly in there?
>
> Yes I think that since subsection is aligned to PAGE_SIZE I can do:
>
> size_t pci_p2pdma_align_size(size_t size)
> {
> ??????? unsigned long min_align;
>
> ??????? min_align = memremap_compat_align();
> ??????? if (!IS_ALIGNED(size, min_align))
> ??????????????? return ALIGN_DOWN(size, min_align);
>
> ??????? return size;
> }
>
>
> thoughts ?
>
>>
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme


From kbusch at kernel.org  Mon Jun 21 09:17:43 2021
From: kbusch at kernel.org (Keith Busch)
Date: Mon, 21 Jun 2021 09:17:43 -0700
Subject: [PATCH] nvme: add comment to not use kobj_to_dev()
In-Reply-To: <20210621053251.GA4360@lst.de>
References: <20210621030940.22286-1-chaitanya.kulkarni@wdc.com>
 <20210621053251.GA4360@lst.de>
Message-ID: <20210621161743.GB1268033@dhcp-10-100-145-180.wdc.com>

On Mon, Jun 21, 2021 at 07:32:51AM +0200, Christoph Hellwig wrote:
> On Sun, Jun 20, 2021 at 08:09:40PM -0700, Chaitanya Kulkarni wrote:
> > Add a comment in nvme_ns_id_attrs_are_visible() and
> > nvme_dev_attrs_are_visible() not to replace container_of() with
> > kobj_to_dev(). Since use of kobj_to_dev() makes code harder to read.
> > 
> > Suggested-by: Keith Busch <kbusch at kernel.org>
> > Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> 
> Err, no.  This code is completely obvious as is, and if people keep
> sending these stupid patches we need to start blacklisting them.

People will continue to send these patches as long as
scripts/coccinelle/api/kobj_to_dev.cocci keeps warning about it.


From axboe at kernel.dk  Mon Jun 21 09:19:20 2021
From: axboe at kernel.dk (Jens Axboe)
Date: Mon, 21 Jun 2021 10:19:20 -0600
Subject: [PATCH] nvme: add comment to not use kobj_to_dev()
In-Reply-To: <20210621161743.GB1268033@dhcp-10-100-145-180.wdc.com>
References: <20210621030940.22286-1-chaitanya.kulkarni@wdc.com>
 <20210621053251.GA4360@lst.de>
 <20210621161743.GB1268033@dhcp-10-100-145-180.wdc.com>
Message-ID: <9ca39a1d-2b66-9a4c-fee0-960ab858b9c2@kernel.dk>

On 6/21/21 10:17 AM, Keith Busch wrote:
> On Mon, Jun 21, 2021 at 07:32:51AM +0200, Christoph Hellwig wrote:
>> On Sun, Jun 20, 2021 at 08:09:40PM -0700, Chaitanya Kulkarni wrote:
>>> Add a comment in nvme_ns_id_attrs_are_visible() and
>>> nvme_dev_attrs_are_visible() not to replace container_of() with
>>> kobj_to_dev(). Since use of kobj_to_dev() makes code harder to read.
>>>
>>> Suggested-by: Keith Busch <kbusch at kernel.org>
>>> Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
>>
>> Err, no.  This code is completely obvious as is, and if people keep
>> sending these stupid patches we need to start blacklisting them.
> 
> People will continue to send these patches as long as
> scripts/coccinelle/api/kobj_to_dev.cocci keeps warning about it.

How about we remove that rule then? FWIW, I totally agree with
conversions like this being harmful, container_of() is perfectly
readable, and the kobj_to_dev() is one of those things you always have
to cscope in to see wtf it is.

-- 
Jens Axboe



From hch at lst.de  Mon Jun 21 09:20:19 2021
From: hch at lst.de (Christoph Hellwig)
Date: Mon, 21 Jun 2021 18:20:19 +0200
Subject: [PATCH] nvme: add comment to not use kobj_to_dev()
In-Reply-To: <20210621161743.GB1268033@dhcp-10-100-145-180.wdc.com>
References: <20210621030940.22286-1-chaitanya.kulkarni@wdc.com>
 <20210621053251.GA4360@lst.de>
 <20210621161743.GB1268033@dhcp-10-100-145-180.wdc.com>
Message-ID: <20210621162019.GA7655@lst.de>

On Mon, Jun 21, 2021 at 09:17:43AM -0700, Keith Busch wrote:
> On Mon, Jun 21, 2021 at 07:32:51AM +0200, Christoph Hellwig wrote:
> > On Sun, Jun 20, 2021 at 08:09:40PM -0700, Chaitanya Kulkarni wrote:
> > > Add a comment in nvme_ns_id_attrs_are_visible() and
> > > nvme_dev_attrs_are_visible() not to replace container_of() with
> > > kobj_to_dev(). Since use of kobj_to_dev() makes code harder to read.
> > > 
> > > Suggested-by: Keith Busch <kbusch at kernel.org>
> > > Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
> > 
> > Err, no.  This code is completely obvious as is, and if people keep
> > sending these stupid patches we need to start blacklisting them.
> 
> People will continue to send these patches as long as
> scripts/coccinelle/api/kobj_to_dev.cocci keeps warning about it.

Maybe we need to fix that instead?


From kbusch at kernel.org  Mon Jun 21 10:26:53 2021
From: kbusch at kernel.org (Keith Busch)
Date: Mon, 21 Jun 2021 10:26:53 -0700
Subject: [PATCHv6] nvme: allow to re-attach namespaces after all paths
 are down
In-Reply-To: <20210609150118.130650-1-hare@suse.de>
References: <20210609150118.130650-1-hare@suse.de>
Message-ID: <20210621172653.GC1268033@dhcp-10-100-145-180.wdc.com>

On Wed, Jun 09, 2021 at 05:01:18PM +0200, Hannes Reinecke wrote:
> -void nvme_mpath_remove_disk(struct nvme_ns_head *head)
> +void nvme_mpath_check_last_path(struct nvme_ns_head *head)
>  {
> +	bool last_path = false;
>  	if (!head->disk)
>  		return;
> -	if (head->disk->flags & GENHD_FL_UP) {
> -		nvme_cdev_del(&head->cdev, &head->cdev_device);
> -		del_gendisk(head->disk);
> +
> +	/* Synchronize with nvme_init_ns_head() */
> +	mutex_lock(&head->subsys->lock);
> +	if (list_empty(&head->list))
> +		last_path = true;
> +	mutex_unlock(&head->subsys->lock);
> +	if (last_path) {
> +		kblockd_schedule_work(&head->requeue_work);
> +		if (head->disk->flags & GENHD_FL_UP) {
> +			nvme_cdev_del(&head->cdev, &head->cdev_device);
> +			del_gendisk(head->disk);
> +		}
>  	}
> +}

This is being called when a path is removed. If it is the last path, the
head's block device is being deleted. If a live head reference is held,
and the a path reconnects, the code will find the previous head, but the
disk is gone and it's not coming back, so the new connection can't be
used.


From sagi at grimberg.me  Mon Jun 21 11:13:51 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Mon, 21 Jun 2021 11:13:51 -0700
Subject: [PATCHv6] nvme: allow to re-attach namespaces after all paths are
 down
In-Reply-To: <20210609150118.130650-1-hare@suse.de>
References: <20210609150118.130650-1-hare@suse.de>
Message-ID: <4903ef70-ed16-4b81-3570-60e9fcc5ecb0@grimberg.me>



On 6/9/21 8:01 AM, Hannes Reinecke wrote:
> We should only remove the ns head from the list of heads per
> subsystem if the reference count drops to zero. That cleans up
> reference counting, and allows us to call del_gendisk() once the last
> path is removed (as then the ns_head should be removed anyway).
> As this introduces a (theoretical) race condition where I/O might have
> been requeued before the last path went down we also should be checking
> if the gendisk is still present in nvme_ns_head_submit_bio(),
> and failing I/O if so.
> 
> Changes to v5:
> - Synchronize between nvme_init_ns_head() and nvme_mpath_check_last_path()
> - Check for removed gendisk in nvme_ns_head_submit_bio()
> Changes to v4:
> - Call del_gendisk() in nvme_mpath_check_last_path() to avoid deadlock
> Changes to v3:
> - Simplify if() clause to detect duplicate namespaces
> Changes to v2:
> - Drop memcpy() statement
> Changes to v1:
> - Always check NSIDs after reattach
> 
> Signed-off-by: Hannes Reinecke <hare at suse.de>
> ---
>   drivers/nvme/host/core.c      |  9 ++++-----
>   drivers/nvme/host/multipath.c | 30 +++++++++++++++++++++++++-----
>   drivers/nvme/host/nvme.h      | 11 ++---------
>   3 files changed, 31 insertions(+), 19 deletions(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 177cae44b612..6d7c2958b3e2 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -566,6 +566,9 @@ static void nvme_free_ns_head(struct kref *ref)
>   	struct nvme_ns_head *head =
>   		container_of(ref, struct nvme_ns_head, ref);
>   
> +	mutex_lock(&head->subsys->lock);
> +	list_del_init(&head->entry);
> +	mutex_unlock(&head->subsys->lock);
>   	nvme_mpath_remove_disk(head);
>   	ida_simple_remove(&head->subsys->ns_ida, head->instance);
>   	cleanup_srcu_struct(&head->srcu);
> @@ -3806,8 +3809,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid,
>    out_unlink_ns:
>   	mutex_lock(&ctrl->subsys->lock);
>   	list_del_rcu(&ns->siblings);
> -	if (list_empty(&ns->head->list))
> -		list_del_init(&ns->head->entry);
>   	mutex_unlock(&ctrl->subsys->lock);
>   	nvme_put_ns_head(ns->head);
>    out_free_queue:
> @@ -3828,8 +3829,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>   
>   	mutex_lock(&ns->ctrl->subsys->lock);
>   	list_del_rcu(&ns->siblings);
> -	if (list_empty(&ns->head->list))
> -		list_del_init(&ns->head->entry);
>   	mutex_unlock(&ns->ctrl->subsys->lock);
>   
>   	synchronize_rcu(); /* guarantee not available in head->list */
> @@ -3849,7 +3848,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>   	list_del_init(&ns->list);
>   	up_write(&ns->ctrl->namespaces_rwsem);
>   
> -	nvme_mpath_check_last_path(ns);
> +	nvme_mpath_check_last_path(ns->head);
>   	nvme_put_ns(ns);
>   }
>   
> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
> index 23573fe3fc7d..31153f6ec582 100644
> --- a/drivers/nvme/host/multipath.c
> +++ b/drivers/nvme/host/multipath.c
> @@ -266,6 +266,8 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
>   	int node = numa_node_id();
>   	struct nvme_ns *ns;
>   
> +	if (!(head->disk->flags & GENHD_FL_UP))
> +		return NULL;
>   	ns = srcu_dereference(head->current_path[node], &head->srcu);
>   	if (unlikely(!ns))
>   		return __nvme_find_path(head, node);
> @@ -281,6 +283,8 @@ static bool nvme_available_path(struct nvme_ns_head *head)
>   {
>   	struct nvme_ns *ns;
>   
> +	if (!(head->disk->flags & GENHD_FL_UP))
> +		return false;

nvme_available_path should have no business looking at the head gendisk,
it should just understand if a PATH (a.k.a a controller) exists.

IMO, the fact that it does should tell that we should take a step back
and think about this. We are trying to keep an zombie nshead around
just for the possibility the host will reconnect (not as part of
error recovery, but as a brand new connect). Why shouldn't we just
remove it and restore it as a brand new nshead when the host attaches
again?

And IMO for queue_if_no_path, we should add a special handling such that
nvme_ns_head_submit_bio would do something explicit that is a special
case like the following:
--
         ns = nvme_find_path(head);
         if (likely(ns)) {
		/* submit */
         } else if (nvme_available_path(head)) {
		/* requeue */
	} else if (nvme_queue_if_no_path(head) &&
		   nvme_zombie_head(head))
		/* requeue */
         } else {
                 /* fail */
         }
--

And it should hopefully make it easier to debug.


From lkp at intel.com  Mon Jun 21 13:54:33 2021
From: lkp at intel.com (kernel test robot)
Date: Tue, 22 Jun 2021 04:54:33 +0800
Subject: [linux-nvme:nvme-5.14] BUILD REGRESSION
 3c3ee16532c1be92350a2a88bd19283b7bdf32e9
Message-ID: <60d0fc89.N1OWFZvjefcSg/hL%lkp@intel.com>

tree/branch: git://git.infradead.org/nvme.git nvme-5.14
branch HEAD: 3c3ee16532c1be92350a2a88bd19283b7bdf32e9  nvmet: use NVMET_MAX_NAMESPACES to set nn value

Error/Warning in current branch:

make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/block/bfq.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/block/mq-deadline.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/authenc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/authencesn.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/blake2b_generic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/blowfish_common.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/blowfish_generic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/cast6_generic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/cmac.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/cts.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/curve25519-generic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/ecdsa_generic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/essiv.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/fcrypt.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/lz4.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/pcbc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/rmd160.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/seqiv.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/sha3_generic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/sha512_generic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/auxdisplay/hd44780_common.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/base/regmap/regmap-slimbus.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/block/cryptoloop.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/block/loop.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/block/virtio_blk.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/cdrom/cdrom.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/dtlk.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/hw_random/cctrng.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/hw_random/exynos-trng.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/hw_random/imx-rngc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/hw_random/stm32-rng.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/bt-bmc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/ipmi_msghandler.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/ipmi_poweroff.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/ipmi_si.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/ipmi_ssif.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/ipmi_watchdog.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/tpm/st33zp24/tpm_st33zp24.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/tpm/st33zp24/tpm_st33zp24_i2c.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/tpm/tpm_infineon.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/bcm/clk-raspberrypi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-cdce706.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-cdce925.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-cs2000-cp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-s2mps11.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-scmi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-scpi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/a7-pll.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/apss-ipq-pll.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/camcc-sc7180.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/camcc-sdm845.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-ipq6018.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-ipq8074.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-msm8996.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-sc7280.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-sc8180x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-sdm660.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-sdm845.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-sdx55.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gpucc-sc7180.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gpucc-sdm660.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gpucc-sdm845.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/hfpll.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/lcc-msm8960.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/lpasscc-sdm845.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/turingcc-qcs404.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/videocc-sm8150.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/samsung/clk-exynos-audss.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/ti/adpll.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/comedi/comedi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/comedi/drivers/8255.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/comedi/drivers/comedi_8255.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/counter/counter.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/counter/ftm-quaddec.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/counter/ti-eqep.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dax/dax.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/devfreq/event/exynos-nocp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/devfreq/event/rockchip-dfi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/devfreq/exynos-bus.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/devfreq/governor_passive.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/devfreq/governor_userspace.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma-buf/dmabuf_selftests.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/dma-jz4780.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/dmatest.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/fsl-qdma.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/iop-adma.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/mediatek/mtk-hsdma.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/milbeaut-hdmac.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/milbeaut-xdmac.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/nbpfaxi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/qcom/bam_dma.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/qcom/qcom_adm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/sh/rcar-dmac.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/sun6i-dma.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/tegra20-apb-dma.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/ti/omap-dma.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/edac/al_mc_edac.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/edac/edac_core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/edac/xgene_edac.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/extcon/extcon-ptn5150.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/extcon/extcon-qcom-spmi-misc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/extcon/extcon-usb-gpio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/firmware/arm_scpi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/firmware/broadcom/tee_bnxt_fw.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/firmware/turris-mox-rwtm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/fsi/fsi-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/fsi/fsi-master-aspeed.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/fsi/fsi-master-ast-cf.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/fsi/fsi-master-gpio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/fsi/fsi-scom.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gnss/gnss-sirf.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gnss/gnss.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-74xx-mmio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-adnp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-adp5588.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-amd-fch.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-bd70528.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-bd71828.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-clps711x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-dwapb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-eic-sprd.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-hisi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-kempld.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-logicvc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-lpc18xx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-madera.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-max732x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-max77620.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-menz127.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-mxc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-pca953x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-pca9570.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-pmic-eic-sprd.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-rcar.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-siox.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-tegra186.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-wcd934x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-wm831x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-xgs-iproc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hsi/controllers/omap_ssi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ad7414.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adc128d818.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adm1029.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ads7828.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adt7410.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adt7462.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adt7475.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adt7x10.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/bt1-pvt.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/da9055-hwmon.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/dme1737.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ds1621.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/f71805f.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/f71882fg.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/f75375s.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/gl520sm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/hih6130.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ibmaem.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ibmpex.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ina209.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ina2xx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/it87.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/lm63.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/lm83.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/lm90.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ltc2947-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ltc2947-i2c.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ltc2992.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ltc4245.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ltc4260.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/max16065.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/max1619.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/max6639.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/mlxreg-fan.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/mr75203.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/nct6683.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/nct6775.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pcf8591.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/adm1266.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/adm1275.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/bel-pfe.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/fsp-3y.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/inspur-ipsps.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/ir36021.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/ir38064.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/lm25066.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/ltc2978.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/max15301.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/max16064.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/max20751.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/max34440.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/max8688.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/mp2975.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/pm6764tr.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/pmbus_core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/pxe1610.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/stpddc60.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/tps40422.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/tps53679.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/ucd9200.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/xdpe12284.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/zl6100.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/raspberrypi-hwmon.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/sht21.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/sht3x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/sl28cpld-hwmon.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/smm665.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/stts751.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/tmp103.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/tmp421.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/vt1211.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/w83795.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/wm8350-hwmon.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/algos/i2c-algo-pcf.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-at91.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-bcm-iproc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-bcm2835.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-davinci.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-designware-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-designware-platform.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-elektor.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-exynos5.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-jz4780.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-kempld.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-lpc2k.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-mlxcpld.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-mt7621.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-mv64xxx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-mxs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-ocores.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-omap.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-pmcmsp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-pxa.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-qcom-cci.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-tegra-bpmp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-uniphier.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-xlp9xx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/i2c-dev.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/i2c-mux.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/muxes/i2c-mux-mlxcpld.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/muxes/i2c-mux-pinctrl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/muxes/i2c-mux-reg.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i3c/i3c.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i3c/master/dw-i3c-master.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i3c/master/i3c-master-cdns.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i3c/master/mipi-i3c-hci/mipi-i3c-hci.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i3c/master/svc-i3c-master.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/ide/ide-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/ide/ide_platform.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/altera_ps2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/apbps2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/arc_ps2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/olpc_apsp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/ps2mult.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/sa1111ps2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/serio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/serio_raw.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/ipack/ipack.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/irqchip/irq-madera.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/blink/leds-lgm-sso.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/led-class-multicolor.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/led-class.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-acer-a500.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-ariel.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-aw2013.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-bcm6328.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-bd2802.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-cobalt-qube.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-gpio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-ip30.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-is31fl32xx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lm3532.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lm3533.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lm355x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lm3642.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lp3952.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lp8860.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lt3593.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-max77650.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-mlxreg.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-mt6323.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-netxbig.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-ns2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-ot200.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-pca963x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-pwm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-tca6507.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-tlc591xx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-tps6105x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-turris-omnia.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-wm831x-status.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/trigger/ledtrig-activity.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/trigger/ledtrig-oneshot.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/trigger/ledtrig-pattern.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/uleds.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mailbox/bcm-pdc-mailbox.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mailbox/mailbox-test.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mailbox/mtk-cmdq-mailbox.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mailbox/platform_mhu.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mailbox/qcom-apcs-ipc-mailbox.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mcb/mcb-lpc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-bio-prison.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-bufio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-cache.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-clone.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-crypt.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-delay.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-dust.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-ebs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-era.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-historical-service-time.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-log.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-mirror.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-mod.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-multipath.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-queue-length.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-region-hash.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-round-robin.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-service-time.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-snapshot.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-thin-pool.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-verity.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-writecache.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/persistent-data/dm-persistent-data.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/memory/mtk-smi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/bcm590xx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/da9062-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/da9063.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/gateworks-gsc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/hi6421-pmic-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/iqs62x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/khadas-mcu.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/lp3943.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/madera-i2c.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/madera.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/max14577.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/max77650.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/max77693.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/max8907.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/mc13xxx-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/mc13xxx-i2c.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/mt6397.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/ntxec.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/qcom-pm8xxx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/rave-sp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/retu-mfd.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/rohm-bd70528.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/rohm-bd71828.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/sec-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/sec-irq.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/si476x-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/sky81452.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/ssbi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/stm32-lptimer.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/stmfx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/ti_am335x_tscadc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/tps65010.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/tps6507x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/tps65217.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/tps65912-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/tps65912-i2c.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/wcd934x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/ad525x_dpot.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/bh1770glc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/c2port/core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/dummy-irq.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/eeprom/at24.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/eeprom/idt_89hpesx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/eeprom/max6875.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/hmc6352.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/isl29003.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/qcom-coincell.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/core/mmc_core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/core/pwrseq_simple.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/cqhci.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/meson-gx-mmc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/meson-mx-sdhc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/meson-mx-sdio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/moxart-mmc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/mvsdio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/omap_hsmmc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/owl-mmc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/pxamci.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/s3cmci.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/sdhci.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/sunxi-mmc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mux/mux-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mux/mux-gpio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mux/mux-mmio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem-imx-ocotp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem-rave-sp-eeprom.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem-rmem.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem-rockchip-otp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem_brcm_nvram.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem_jz4780_efuse.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem_rockchip_efuse.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem_sprd_efuse.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/perf/arm_dmc620_pmu.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/allwinner/phy-sun50i-usb3.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/amlogic/phy-meson-axg-mipi-dphy.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/amlogic/phy-meson-gxl-usb2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/amlogic/phy-meson8b-usb2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/broadcom/phy-bcm-sr-pcie.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/cadence/cdns-dphy.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/cadence/phy-cadence-sierra.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/cadence/phy-cadence-torrent.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/freescale/phy-fsl-imx8mq-usb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/hisilicon/phy-hi6220-usb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/hisilicon/phy-histb-combphy.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/ingenic/phy-ingenic-usb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/lantiq/phy-lantiq-rcu-usb2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/lantiq/phy-lantiq-vrx200-pcie.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/marvell/phy-armada38x-comphy.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/marvell/phy-berlin-usb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/marvell/phy-mmp3-hsic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/marvell/phy-pxa-usb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/mediatek/phy-mtk-hdmi-drv.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/mediatek/phy-mtk-ufs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/mediatek/phy-mtk-xsphy.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/microchip/sparx5_serdes.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/motorola/phy-mapphone-mdm6600.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/mscc/phy-ocelot-serdes.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/phy-xgene.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/qualcomm/phy-ath79-usb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/qualcomm/phy-qcom-ipq806x-usb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/qualcomm/phy-qcom-usb-hs-28nm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/ralink/phy-ralink-usb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/rockchip/phy-rockchip-dphy-rx0.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/rockchip/phy-rockchip-inno-dsidphy.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/samsung/phy-exynos-dp-video.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/samsung/phy-samsung-ufs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/socionext/phy-uniphier-usb3hs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/socionext/phy-uniphier-usb3ss.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/tegra/phy-tegra194-p2u.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/ti/phy-am654-serdes.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/ti/phy-j721e-wiz.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/cirrus/pinctrl-lochnagar.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/cirrus/pinctrl-madera.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/mediatek/pinctrl-mt6779.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-da850-pupd.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-equilibrium.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-mcp23s08.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-mcp23s08_i2c.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-rockchip.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-single.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-stmfx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/platform/mellanox/mlxreg-io.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/acer_a500_battery.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/adp5061.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bd70528-charger.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bd99954-charger.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bq24257_charger.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bq2515x_charger.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bq25890_charger.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bq25980_charger.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bq27xxx_battery_i2c.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/isp1704_charger.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/lp8727_charger.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/max77650-charger.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/max77693_charger.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/rt5033_battery.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/tps65217_charger.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/ucs1002_power.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/wm831x_power.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/z2_battery.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-atmel-tcb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-hibvt.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-iqs620a.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-lpc18xx-sct.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-ntxec.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-omap-dmtimer.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-pca9685.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-pxa.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-raspberrypi-poe.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-rcar.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-samsung.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-sl28cpld.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-stm32.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-tegra.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/88pg86x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/aat2870-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/act8945a-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/ad5398.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/anatop-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/bcm590xx-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/bd70528-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/bd71828-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/da9063-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/da9211-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/gpio-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/hi6421-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/isl9305.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/lp3971.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/lp3972.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/ltc3676.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max14577-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max77650-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max77686-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max77802-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max77826-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max8649.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/mc13783-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/mc13xxx-regulator-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/mp5416.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/mt6311-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/mt6397-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/pf8x00-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/pv88080-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/pwm-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/qcom-labibb-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/qcom-rpmh-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/rc5t583-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/rohm-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/rpi-panel-attiny-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/rt4801-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/s2mpa01.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/sky81452-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/slg51000-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/stpmic1_regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/tps6507x-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/tps65132-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/tps65217-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/uniphier-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/vctrl-regulator.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/reset/hisilicon/hi6220_reset.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/reset/reset-meson.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/reset/reset-qcom-pdc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/reset/reset-ti-syscon.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-ab-eoz9.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-aspeed.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-brcmstb-waketimer.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-cadence.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-da9063.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-ds1553.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-ds1685.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-ds2404.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-em3027.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-fm3130.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-fsl-ftm-alarm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-generic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-goldfish.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-hym8563.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-isl1208.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-lpc24xx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-mc13xxx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-msm6242.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-mt2712.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-mt7622.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-mxc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-ntxec.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-pcf85363.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-pcf8583.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-pxa.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-rx8025.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-rx8581.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-snvs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-vt8500.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-wm8350.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/aha152x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ch.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/fdomain.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/fdomain_isa.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/g_NCR5380.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/iscsi_boot_sysfs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/libsas/libsas.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/qlogicfas.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/qlogicfas408.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/raid_class.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/scsi_debug.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/scsi_mod.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/scsi_transport_sas.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/scsi_transport_spi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/sd_mod.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ses.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/sg.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/sr_mod.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ufs/ufs-exynos.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ufs/ufs-hisi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ufs/ufshcd-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ufs/ufshcd-pltfrm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/siox/siox-bus-gpio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/siox/siox-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/slimbus/slimbus.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/amlogic/meson-canvas.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/amlogic/meson-clk-measure.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/fsl/dpaa2-console.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/litex/litex_soc_ctrl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/mediatek/mtk-cmdq-helper.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/cmd-db.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/qcom-geni-se.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/qcom_rpmh.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/rpmhpd.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/smd-rpm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/wcnss_ctrl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/loopback/tcm_loop.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/sbp/sbp_target.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_iblock.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_mod.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_pscsi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/tee/tee.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/bcm2711_thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/bcm2835_thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/brcmstb_thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/da9062-thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/dove_thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/hisi_thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/imx8mm_thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/imx_thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/k3_bandgap.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/khadas_mcu_fan.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/mtk_thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/rcar_gen3_thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/sprd_thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/sun8i_thermal.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/tty/serdev/serdev.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/uio/uio_dmem_genirq.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns-usb-common.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3-imx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3-ti.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_imx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_tegra.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/usbmisc_imx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/dwc2/dwc2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_fs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_hid.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_printer.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_tcm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_ffs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_hid.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_printer.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/gadgetfs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/raw_gadget.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/libcomposite.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/aspeed-vhub/aspeed-vhub.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/fotg210-udc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/gr_udc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/lpc32xx_udc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/mv_udc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/net2272.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/renesas_usb3.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/snps_udc_core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/snps_udc_plat.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/udc-core.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/udc-xilinx.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/isp1760/isp1760.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/mtu3/mtu3.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/jz4740.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/musb_dsps.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/musb_hdrc.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/tusb6010.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-am335x-control.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-am335x.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-generic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-isp1301.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-tahvo.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/mdev/mdev.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/vfio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/vfio_iommu_type1.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/adp8860_bl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/adp8870_bl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/arcxcnn_bl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/backlight.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/da903x_bl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/led_bl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lm3533_bl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lm3630a_bl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lp855x_bl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lv5207lp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/pwm_bl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/qcom-wled.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/sky81452-backlight.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/tps65217_bl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/wm831x_bl.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/arcfb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/atmel_lcdfb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/broadsheetfb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/core/fb_sys_fops.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/goldfishfb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/imxfb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/mmp/fb/mmpfb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/mmp/mmp_disp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/connector-dvi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/connector-hdmi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/encoder-opa362.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-dpi.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-dsi-cm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-sharp-ls037v7dw01.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pvr2fb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pxa168fb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pxa3xx-gcu.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/s3c-fb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/sh_mobile_lcdcfb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/ssd1307fb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/vfb.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/ds1wm.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/ds2482.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/mxc_w1.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/w1-gpio.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds2408.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds2438.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds28e17.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_smem.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/adfs/adfs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/cramfs/cramfs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/exfat/exfat.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ext2/ext2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ext4/ext4-inode-test.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/fat/vfat.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/freevxfs/freevxfs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/gfs2/gfs2.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/hfs/hfs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/hfsplus/hfsplus.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-centeuro.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-croatian.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-cyrillic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-gaelic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-roman.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_ascii.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp1250.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp1255.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp437.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp850.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp857.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp860.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp869.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp932.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp936.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp950.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_euc-jp.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_iso8859-1.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_utf8.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ntfs/ntfs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/overlayfs/overlay.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/qnx6/qnx6.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/quota/quota_v1.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/squashfs/squashfs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/unicode/utf8-selftest.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/configs.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/locking/test-ww_mutex.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/resource_kunit.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/trace/ring_buffer_benchmark.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/bitfield_kunit.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crc8.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crypto/libchacha20poly1305.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crypto/libcurve25519-generic.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/globtest.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/kunit/kunit.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/lz4/lz4_compress.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/lz4/lz4_decompress.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/math/prime_numbers.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/math/test_div64.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/notifier-error-inject.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/parman.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/percpu_test.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/rbtree_test.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_bitmap.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_bits.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_free_pages.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_linear_ranges.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_meminit.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_module.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_parman.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_sort.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_stackinit.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_ubsan.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_uuid.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/z3fold.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/zbud.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/zpool.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/configfs/configfs_sample.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/bytestream-example.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/dma-example.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/inttype-example.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/record-example.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/rpmsg/rpmsg_client_sample.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/trace_events/trace-events-sample.ko.gz', needed by '__modinst'.
make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/vfio-mdev/mdpy-fb.ko.gz', needed by '__modinst'.

Error/Warning ids grouped by kconfigs:

gcc_recent_errors
`-- arm-randconfig-c023-20210621
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-block-bfq.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-block-mq-deadline.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-authenc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-authencesn.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-blake2b_generic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-blowfish_common.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-blowfish_generic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-cast6_generic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-cmac.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-cts.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-curve25519-generic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-ecdsa_generic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-essiv.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-fcrypt.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-lz4.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-pcbc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-rmd160.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-seqiv.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-sha3_generic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-sha512_generic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-auxdisplay-hd44780_common.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-base-regmap-regmap-slimbus.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-block-cryptoloop.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-block-loop.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-block-virtio_blk.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-cdrom-cdrom.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-dtlk.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-hw_random-cctrng.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-hw_random-exynos-trng.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-hw_random-imx-rngc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-hw_random-stm32-rng.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-bt-bmc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-ipmi_msghandler.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-ipmi_poweroff.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-ipmi_si.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-ipmi_ssif.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-ipmi_watchdog.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-tpm-st33zp24-tpm_st33zp24.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-tpm-st33zp24-tpm_st33zp24_i2c.ko.gz-needed-by-__modins
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-tpm-tpm_infineon.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-bcm-clk-raspberrypi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-cdce706.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-cdce925.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-cs2000-cp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-s2mps11.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-scmi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-scpi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-a7-pll.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-apss-ipq-pll.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-camcc-sc7180.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-camcc-sdm845.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-ipq6018.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-ipq8074.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-msm8996.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-sc7280.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-sc818.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-sdm660.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-sdm845.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-sdx55.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gpucc-sc7180.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gpucc-sdm660.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gpucc-sdm845.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-hfpll.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-lcc-msm8960.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-lpasscc-sdm845.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-turingcc-qcs404.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-videocc-sm8150.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-samsung-clk-exynos-audss.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-ti-adpll.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-comedi-comedi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-comedi-drivers-.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-comedi-drivers-comedi_8255.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-counter-counter.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-counter-ftm-quaddec.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-counter-ti-eqep.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dax-dax.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-devfreq-event-exynos-nocp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-devfreq-event-rockchip-dfi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-devfreq-exynos-bus.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-devfreq-governor_passive.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-devfreq-governor_userspace.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-buf-dmabuf_selftests.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-dma-jz4780.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-dmatest.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-fsl-qdma.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-iop-adma.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-mediatek-mtk-hsdma.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-milbeaut-hdmac.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-milbeaut-xdmac.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-nbpfaxi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-qcom-bam_dma.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-qcom-qcom_adm.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-sh-rcar-dmac.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-sun6i-dma.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-tegra20-apb-dma.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-ti-omap-dma.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-edac-al_mc_edac.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-edac-edac_core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-edac-xgene_edac.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-extcon-extcon-ptn5150.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-extcon-extcon-qcom-spmi-misc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-extcon-extcon-usb-gpio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-firmware-arm_scpi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-firmware-broadcom-tee_bnxt_fw.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-firmware-turris-mox-rwtm.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-fsi-fsi-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-fsi-fsi-master-aspeed.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-fsi-fsi-master-ast-cf.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-fsi-fsi-master-gpio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-fsi-fsi-scom.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gnss-gnss-sirf.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gnss-gnss.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-74xx-mmio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-adnp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-adp5588.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-amd-fch.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-bd70528.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-bd71828.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-clps711x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-dwapb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-eic-sprd.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-hisi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-kempld.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-logicvc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-lpc18xx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-madera.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-max732x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-max77620.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-menz127.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-mxc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-pca953x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-pca9570.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-pmic-eic-sprd.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-rcar.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-siox.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-tegra186.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-wcd934x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-wm831x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-xgs-iproc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hsi-controllers-omap_ssi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ad7414.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adc128d818.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adm1029.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ads7828.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adt7410.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adt7462.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adt7475.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adt7x10.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-bt1-pvt.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-da9055-hwmon.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-dme1737.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ds1621.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-f71805f.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-f71882fg.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-f75375s.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-gl520sm.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-hih6130.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ibmaem.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ibmpex.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ina209.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ina2xx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-it87.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-lm63.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-lm83.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-lm90.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ltc2947-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ltc2947-i2c.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ltc2992.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ltc4245.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ltc4260.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-max16065.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-max1619.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-max6639.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-mlxreg-fan.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-mr75203.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-nct6683.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-nct6775.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pcf8591.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-adm1266.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-adm1275.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-bel-pfe.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-fsp-3y.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-inspur-ipsps.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-ir36021.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-ir38064.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-lm25066.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-ltc2978.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-max15301.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-max16064.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-max20751.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-max34440.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-max8688.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-mp2975.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-pm6764tr.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-pmbus_core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-pxe1610.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-stpddc60.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-tps40422.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-tps53679.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-ucd9200.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-xdpe12284.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-zl6100.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-raspberrypi-hwmon.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-sht21.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-sht3x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-sl28cpld-hwmon.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-smm665.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-stts751.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-tmp103.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-tmp421.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-vt1211.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-w83795.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-wm8350-hwmon.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-algos-i2c-algo-pcf.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-at91.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-bcm-iproc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-bcm2835.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-davinci.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-designware-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-designware-platform.ko.gz-needed-by-__modins
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-elektor.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-exynos5.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-jz4780.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-kempld.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-lpc2k.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-mlxcpld.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-mt7621.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-mv64xxx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-mxs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-ocores.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-omap.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-pmcmsp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-pxa.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-qcom-cci.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-tegra-bpmp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-uniphier.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-xlp9xx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-i2c-dev.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-i2c-mux.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-muxes-i2c-mux-mlxcpld.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-muxes-i2c-mux-pca9541.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-muxes-i2c-mux-pinctrl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-muxes-i2c-mux-reg.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i3c-i3c.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i3c-master-dw-i3c-master.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i3c-master-i3c-master-cdns.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i3c-master-mipi-i3c-hci-mipi-i3c-hci.ko.gz-needed-by-__modi
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i3c-master-svc-i3c-master.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-ide-ide-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-ide-ide_platform.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-altera_ps2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-apbps2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-arc_ps2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-olpc_apsp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-ps2mult.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-sa1111ps2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-serio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-serio_raw.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-ipack-ipack.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-irqchip-irq-madera.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-blink-leds-lgm-sso.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-led-class-multicolor.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-led-class.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-acer-a500.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-ariel.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-aw2013.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-bcm6328.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-bd2802.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-cobalt-qube.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-gpio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-ip30.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-is31fl32xx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lm3532.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lm3533.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lm355x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lm3642.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lp3952.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lp8860.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lt3593.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-max77650.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-mlxreg.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-mt6323.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-netxbig.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-ns2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-ot200.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-pca963x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-pwm.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-tca6507.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-tlc591xx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-tps6105x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-turris-omnia.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-wm831x-status.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-trigger-ledtrig-activity.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-trigger-ledtrig-oneshot.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-trigger-ledtrig-pattern.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-uleds.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mailbox-bcm-pdc-mailbox.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mailbox-mailbox-test.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mailbox-mtk-cmdq-mailbox.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mailbox-platform_mhu.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mailbox-qcom-apcs-ipc-mailbox.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mcb-mcb-lpc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-bio-prison.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-bufio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-cache.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-clone.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-crypt.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-delay.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-dust.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-ebs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-era.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-historical-service-time.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-log.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-mirror.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-mod.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-multipath.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-queue-length.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-region-hash.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-round-robin.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-service-time.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-snapshot.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-thin-pool.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-verity.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-writecache.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-persistent-data-dm-persistent-data.ko.gz-needed-by-__mod
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-memory-mtk-smi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-bcm59x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-da9062-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-da9063.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-gateworks-gsc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-hi6421-pmic-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-iqs62x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-khadas-mcu.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-lp3943.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-madera-i2c.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-madera.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-max14577.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-max77650.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-max77693.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-max8907.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-mc13xxx-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-mc13xxx-i2c.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-mt6397.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-ntxec.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-qcom-pm8xxx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-rave-sp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-retu-mfd.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-rohm-bd70528.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-rohm-bd71828.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-sec-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-sec-irq.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-si476x-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-sky81452.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-ssbi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-stm32-lptimer.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-stmfx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-ti_am335x_tscadc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-tps65010.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-tps6507x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-tps65217.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-tps65912-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-tps65912-i2c.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-wcd934x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-ad525x_dpot.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-bh1770glc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-c2port-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-dummy-irq.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-eeprom-at24.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-eeprom-idt_89hpesx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-eeprom-max6875.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-hmc6352.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-isl29003.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-qcom-coincell.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-core-mmc_core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-core-pwrseq_simple.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-cqhci.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-meson-gx-mmc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-meson-mx-sdhc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-meson-mx-sdio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-moxart-mmc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-mvsdio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-omap_hsmmc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-owl-mmc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-pxamci.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-s3cmci.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-sdhci.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-sunxi-mmc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mux-mux-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mux-mux-gpio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mux-mux-mmio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem-imx-ocotp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem-rave-sp-eeprom.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem-rmem.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem-rockchip-otp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem_brcm_nvram.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem_jz4780_efuse.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem_rockchip_efuse.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem_sprd_efuse.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-perf-arm_dmc620_pmu.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-allwinner-phy-sun50i-usb3.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-amlogic-phy-meson-axg-mipi-dphy.ko.gz-needed-by-__modin
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-amlogic-phy-meson-gxl-usb2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-amlogic-phy-meson8b-usb2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-broadcom-phy-bcm-sr-pcie.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-cadence-cdns-dphy.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-cadence-phy-cadence-sierra.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-cadence-phy-cadence-torrent.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-freescale-phy-fsl-imx8mq-usb.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-hisilicon-phy-hi6220-usb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-hisilicon-phy-histb-combphy.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-ingenic-phy-ingenic-usb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-lantiq-phy-lantiq-rcu-usb2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-lantiq-phy-lantiq-vrx200-pcie.ko.gz-needed-by-__modinst
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-marvell-phy-armada38x-comphy.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-marvell-phy-berlin-usb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-marvell-phy-mmp3-hsic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-marvell-phy-pxa-usb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-mediatek-phy-mtk-hdmi-drv.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-mediatek-phy-mtk-ufs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-mediatek-phy-mtk-xsphy.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-microchip-sparx5_serdes.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-motorola-phy-mapphone-mdm6600.ko.gz-needed-by-__modinst
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-mscc-phy-ocelot-serdes.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-phy-xgene.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-qualcomm-phy-ath79-usb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-qualcomm-phy-qcom-ipq806x-usb.ko.gz-needed-by-__modinst
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-qualcomm-phy-qcom-usb-hs-28nm.ko.gz-needed-by-__modinst
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-ralink-phy-ralink-usb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-rockchip-phy-rockchip-dphy-rx0.ko.gz-needed-by-__modins
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-rockchip-phy-rockchip-inno-dsidphy.ko.gz-needed-by-__mo
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-samsung-phy-exynos-dp-video.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-samsung-phy-samsung-ufs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-socionext-phy-uniphier-usb3hs.ko.gz-needed-by-__modinst
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-socionext-phy-uniphier-usb3ss.ko.gz-needed-by-__modinst
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-tegra-phy-tegra194-p2u.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-ti-phy-am654-serdes.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-ti-phy-j721e-wiz.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-cirrus-pinctrl-lochnagar.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-cirrus-pinctrl-madera.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-mediatek-pinctrl-mt6779.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-da850-pupd.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-equilibrium.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-mcp23s08.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-mcp23s08_i2c.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-rockchip.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-single.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-stmfx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-platform-mellanox-mlxreg-io.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-acer_a500_battery.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-adp5061.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bd70528-charger.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bd99954-charger.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bq24257_charger.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bq2515x_charger.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bq25890_charger.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bq25980_charger.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bq27xxx_battery_i2c.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-isp1704_charger.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-lp8727_charger.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-max77650-charger.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-max77693_charger.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-rt5033_battery.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-tps65217_charger.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-ucs1002_power.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-wm831x_power.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-z2_battery.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-atmel-tcb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-hibvt.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-iqs620a.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-lpc18xx-sct.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-ntxec.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-omap-dmtimer.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-pca9685.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-pxa.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-raspberrypi-poe.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-rcar.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-samsung.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-sl28cpld.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-stm32.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-tegra.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-88pg86x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-aat2870-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-act8945a-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-ad5398.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-anatop-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-bcm59x-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-bd70528-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-bd71828-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-da9063-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-da9211-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-gpio-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-hi6421-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-isl9305.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-lp3971.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-lp3972.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-ltc3676.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max14577-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max77650-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max77686-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max77802-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max77826-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max8649.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-mc13783-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-mc13xxx-regulator-core.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-mp5416.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-mt6311-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-mt6397-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-pf8x00-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-pv88080-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-pwm-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-qcom-labibb-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-qcom-rpmh-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-rc5t583-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-rohm-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-rpi-panel-attiny-regulator.ko.gz-needed-by-__modi
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-rt4801-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-s2mpa01.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-sky81452-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-slg51000-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-stpmic1_regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-tps6507x-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-tps65132-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-tps65217-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-uniphier-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-vctrl-regulator.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-reset-hisilicon-hi6220_reset.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-reset-reset-meson.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-reset-reset-qcom-pdc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-reset-reset-ti-syscon.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-ab-eoz9.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-aspeed.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-brcmstb-waketimer.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-cadence.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-da9063.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-ds1553.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-ds1685.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-ds2404.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-em3027.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-fm3130.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-fsl-ftm-alarm.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-generic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-goldfish.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-hym8563.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-isl1208.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-lpc24xx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-mc13xxx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-msm6242.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-mt2712.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-mt7622.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-mxc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-ntxec.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-pcf85363.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-pcf8583.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-pxa.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-rx8025.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-rx8581.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-snvs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-vt8500.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-wm8350.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-aha152x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ch.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-device_handler-scsi_dh_alua.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-device_handler-scsi_dh_emc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-device_handler-scsi_dh_hp_sw.ko.gz-needed-by-__modinst
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-device_handler-scsi_dh_rdac.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-fdomain.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-fdomain_isa.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-g_NCR5380.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-iscsi_boot_sysfs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-libsas-libsas.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-qlogicfas.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-qlogicfas408.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-raid_class.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-scsi_debug.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-scsi_mod.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-scsi_transport_sas.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-scsi_transport_spi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-sd_mod.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ses.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-sg.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-sr_mod.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ufs-ufs-exynos.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ufs-ufs-hisi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ufs-ufshcd-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ufs-ufshcd-pltfrm.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-siox-siox-bus-gpio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-siox-siox-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-slimbus-slimbus.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-amlogic-meson-canvas.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-amlogic-meson-clk-measure.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-fsl-dpaa2-console.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-litex-litex_soc_ctrl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-mediatek-mtk-cmdq-helper.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-cmd-db.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-qcom-geni-se.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-qcom_rpmh.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-rpmhpd.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-smd-rpm.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-wcnss_ctrl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-target-loopback-tcm_loop.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-target-sbp-sbp_target.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-target-target_core_iblock.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-target-target_core_mod.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-target-target_core_pscsi.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-tee-tee.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-broadcom-bcm2711_thermal.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-broadcom-bcm2835_thermal.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-broadcom-brcmstb_thermal.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-da9062-thermal.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-dove_thermal.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-hisi_thermal.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-imx8mm_thermal.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-imx_thermal.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-k3_bandgap.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-khadas_mcu_fan.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-mtk_thermal.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-rcar_gen3_thermal.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-sprd_thermal.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-sun8i_thermal.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-tty-serdev-serdev.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-uio-uio_dmem_genirq.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-cdns3-cdns-usb-common.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-cdns3-cdns3-imx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-cdns3-cdns3-ti.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-cdns3-cdns3.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-ci_hdrc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-ci_hdrc_imx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-ci_hdrc_msm.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-ci_hdrc_tegra.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-ci_hdrc_usb2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-usbmisc_imx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-dwc2-dwc2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-function-usb_f_fs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-function-usb_f_hid.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-function-usb_f_mass_storage.ko.gz-needed-by-__mo
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-function-usb_f_printer.ko.gz-needed-by-__modinst
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-function-usb_f_tcm.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-g_ffs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-g_hid.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-g_mass_storage.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-g_printer.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-gadgetfs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-raw_gadget.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-tcm_usb_gadget.ko.gz-needed-by-__modinst-
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-libcomposite.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-aspeed-vhub-aspeed-vhub.ko.gz-needed-by-__mo
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-fotg210-udc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-gr_udc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-lpc32xx_udc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-mv_u3d_core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-mv_udc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-net2272.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-pxa27x_udc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-r8a66597-udc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-renesas_usb3.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-snps_udc_core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-snps_udc_plat.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-udc-core.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-udc-xilinx.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-isp1760-isp1760.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-mtu3-mtu3.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-musb-jz4740.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-musb-musb_dsps.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-musb-musb_hdrc.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-musb-tusb6010.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-phy-phy-am335x-control.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-phy-phy-am335x.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-phy-phy-generic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-phy-phy-isp1301.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-phy-phy-tahvo.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-vfio-mdev-mdev.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-vfio-vfio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-vfio-vfio_iommu_type1.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-adp8860_bl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-adp8870_bl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-arcxcnn_bl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-backlight.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-da903x_bl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-led_bl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-lm3533_bl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-lm3630a_bl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-lp855x_bl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-lv5207lp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-pwm_bl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-qcom-wled.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-sky81452-backlight.ko.gz-needed-by-__modins
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-tps65217_bl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-wm831x_bl.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-arcfb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-atmel_lcdfb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-broadsheetfb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-core-fb_sys_fops.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-goldfishfb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-imxfb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-mmp-fb-mmpfb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-mmp-mmp_disp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-connector-dvi.ko.gz-neede
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-connector-hdmi.ko.gz-need
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-encoder-opa362.ko.gz-need
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-panel-dpi.ko.gz-needed-by
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-panel-dsi-cm.ko.gz-needed
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-panel-sharp-ls037v7dw01.k
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-pvr2fb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-pxa168fb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-pxa3xx-gcu.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-s3c-fb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-sh_mobile_lcdcfb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-ssd1307fb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-vfb.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-masters-ds1wm.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-masters-ds2482.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-masters-mxc_w1.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-masters-w1-gpio.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-slaves-w1_ds2408.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-slaves-w1_ds2438.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-slaves-w1_ds28e17.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-slaves-w1_smem.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-adfs-adfs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-cramfs-cramfs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-exfat-exfat.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-ext2-ext2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-ext4-ext4-inode-test.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-fat-vfat.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-freevxfs-freevxfs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-gfs2-gfs2.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-hfs-hfs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-hfsplus-hfsplus.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-mac-centeuro.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-mac-croatian.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-mac-cyrillic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-mac-gaelic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-mac-roman.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_ascii.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp1250.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp1255.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp437.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp850.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp857.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp860.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp869.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp932.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp936.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp950.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_euc-jp.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_iso8859-.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_utf8.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-ntfs-ntfs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-overlayfs-overlay.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-qnx6-qnx6.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-quota-quota_v1.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-squashfs-squashfs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-unicode-utf8-selftest.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-kernel-configs.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-kernel-locking-test-ww_mutex.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-kernel-resource_kunit.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-kernel-trace-ring_buffer_benchmark.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-bitfield_kunit.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-crc8.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-crypto-libchacha20poly1305.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-crypto-libcurve25519-generic.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-globtest.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-kunit-kunit.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-lz4-lz4_compress.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-lz4-lz4_decompress.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-math-prime_numbers.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-math-test_div64.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-notifier-error-inject.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-parman.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-percpu_test.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-rbtree_test.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_bitmap.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_bits.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_free_pages.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_linear_ranges.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_meminit.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_module.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_parman.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_sort.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_stackinit.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_ubsan.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_uuid.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-mm-z3fold.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-mm-zbud.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-mm-zpool.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-configfs-configfs_sample.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-kfifo-bytestream-example.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-kfifo-dma-example.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-kfifo-inttype-example.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-kfifo-record-example.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-rpmsg-rpmsg_client_sample.ko.gz-needed-by-__modinst-.
    |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-trace_events-trace-events-sample.ko.gz-needed-by-__modinst-
    `-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-vfio-mdev-mdpy-fb.ko.gz-needed-by-__modinst-.

elapsed time: 723m

configs tested: 110
configs skipped: 3

gcc tested configs:
arm                                 defconfig
arm64                            allyesconfig
arm64                               defconfig
arm                              allyesconfig
arm                              allmodconfig
arm                            zeus_defconfig
arm                  colibri_pxa270_defconfig
sh                           se7780_defconfig
powerpc                  mpc866_ads_defconfig
arc                     haps_hs_smp_defconfig
sh                          landisk_defconfig
powerpc                      ep88xc_defconfig
mips                  maltasmvp_eva_defconfig
s390                             alldefconfig
m68k                          multi_defconfig
powerpc                     tqm5200_defconfig
m68k                        mvme16x_defconfig
h8300                    h8300h-sim_defconfig
mips                       lemote2f_defconfig
powerpc                      katmai_defconfig
mips                  decstation_64_defconfig
m68k                        m5272c3_defconfig
sh                         ecovec24_defconfig
arm                          collie_defconfig
powerpc                    sam440ep_defconfig
powerpc                       maple_defconfig
ia64                          tiger_defconfig
mips                        qi_lb60_defconfig
powerpc                     tqm8560_defconfig
microblaze                      mmu_defconfig
arm                         cm_x300_defconfig
m68k                         apollo_defconfig
mips                     decstation_defconfig
arm                   milbeaut_m10v_defconfig
powerpc                        warp_defconfig
arm                        multi_v5_defconfig
x86_64                            allnoconfig
ia64                             allmodconfig
ia64                                defconfig
ia64                             allyesconfig
m68k                             allmodconfig
m68k                                defconfig
m68k                             allyesconfig
nios2                               defconfig
arc                              allyesconfig
nds32                             allnoconfig
nds32                               defconfig
nios2                            allyesconfig
csky                                defconfig
alpha                               defconfig
alpha                            allyesconfig
xtensa                           allyesconfig
h8300                            allyesconfig
arc                                 defconfig
sh                               allmodconfig
parisc                              defconfig
s390                             allyesconfig
s390                             allmodconfig
parisc                           allyesconfig
s390                                defconfig
i386                             allyesconfig
sparc                            allyesconfig
sparc                               defconfig
i386                                defconfig
mips                             allyesconfig
mips                             allmodconfig
powerpc                          allyesconfig
powerpc                          allmodconfig
powerpc                           allnoconfig
x86_64               randconfig-a002-20210621
x86_64               randconfig-a001-20210621
x86_64               randconfig-a005-20210621
x86_64               randconfig-a003-20210621
x86_64               randconfig-a004-20210621
x86_64               randconfig-a006-20210621
i386                 randconfig-a002-20210621
i386                 randconfig-a003-20210621
i386                 randconfig-a006-20210621
i386                 randconfig-a005-20210621
i386                 randconfig-a004-20210621
i386                 randconfig-a001-20210621
i386                 randconfig-a011-20210621
i386                 randconfig-a014-20210621
i386                 randconfig-a013-20210621
i386                 randconfig-a015-20210621
i386                 randconfig-a012-20210621
i386                 randconfig-a016-20210621
riscv                    nommu_k210_defconfig
riscv                            allyesconfig
riscv                    nommu_virt_defconfig
riscv                             allnoconfig
riscv                               defconfig
riscv                          rv32_defconfig
riscv                            allmodconfig
x86_64                    rhel-8.3-kselftests
um                           x86_64_defconfig
um                             i386_defconfig
um                            kunit_defconfig
x86_64                           allyesconfig
x86_64                              defconfig
x86_64                               rhel-8.3
x86_64                      rhel-8.3-kbuiltin
x86_64                                  kexec

clang tested configs:
x86_64               randconfig-b001-20210621
x86_64               randconfig-a012-20210621
x86_64               randconfig-a015-20210621
x86_64               randconfig-a014-20210621
x86_64               randconfig-a013-20210621
x86_64               randconfig-a016-20210621
x86_64               randconfig-a011-20210621

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org


From philip.li at intel.com  Mon Jun 21 15:43:54 2021
From: philip.li at intel.com (Philip Li)
Date: Tue, 22 Jun 2021 06:43:54 +0800
Subject: [linux-nvme:nvme-5.14] BUILD REGRESSION
 3c3ee16532c1be92350a2a88bd19283b7bdf32e9
In-Reply-To: <60d0fc89.N1OWFZvjefcSg/hL%lkp@intel.com>
References: <60d0fc89.N1OWFZvjefcSg/hL%lkp@intel.com>
Message-ID: <20210621224354.GB165241@pl-dbox>

On Tue, Jun 22, 2021 at 04:54:33AM +0800, kernel test robot wrote:
> tree/branch: git://git.infradead.org/nvme.git nvme-5.14
> branch HEAD: 3c3ee16532c1be92350a2a88bd19283b7bdf32e9  nvmet: use NVMET_MAX_NAMESPACES to set nn value
> 
> Error/Warning in current branch:
Sorry for broken report, kindly ignore below warnings, we
will fix this asap.

> 
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/block/bfq.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/block/mq-deadline.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/authenc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/authencesn.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/blake2b_generic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/blowfish_common.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/blowfish_generic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/cast6_generic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/cmac.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/cts.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/curve25519-generic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/ecdsa_generic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/essiv.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/fcrypt.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/lz4.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/pcbc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/rmd160.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/seqiv.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/sha3_generic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/crypto/sha512_generic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/auxdisplay/hd44780_common.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/base/regmap/regmap-slimbus.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/block/cryptoloop.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/block/loop.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/block/virtio_blk.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/cdrom/cdrom.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/dtlk.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/hw_random/cctrng.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/hw_random/exynos-trng.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/hw_random/imx-rngc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/hw_random/stm32-rng.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/bt-bmc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/ipmi_msghandler.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/ipmi_poweroff.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/ipmi_si.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/ipmi_ssif.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/ipmi/ipmi_watchdog.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/tpm/st33zp24/tpm_st33zp24.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/tpm/st33zp24/tpm_st33zp24_i2c.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/char/tpm/tpm_infineon.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/bcm/clk-raspberrypi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-cdce706.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-cdce925.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-cs2000-cp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-s2mps11.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-scmi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/clk-scpi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/a7-pll.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/apss-ipq-pll.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/camcc-sc7180.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/camcc-sdm845.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-ipq6018.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-ipq8074.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-msm8996.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-sc7280.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-sc8180x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-sdm660.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-sdm845.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gcc-sdx55.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gpucc-sc7180.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gpucc-sdm660.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/gpucc-sdm845.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/hfpll.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/lcc-msm8960.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/lpasscc-sdm845.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/turingcc-qcs404.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/qcom/videocc-sm8150.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/samsung/clk-exynos-audss.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/clk/ti/adpll.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/comedi/comedi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/comedi/drivers/8255.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/comedi/drivers/comedi_8255.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/counter/counter.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/counter/ftm-quaddec.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/counter/ti-eqep.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dax/dax.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/devfreq/event/exynos-nocp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/devfreq/event/rockchip-dfi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/devfreq/exynos-bus.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/devfreq/governor_passive.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/devfreq/governor_userspace.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma-buf/dmabuf_selftests.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/dma-jz4780.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/dmatest.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/fsl-qdma.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/iop-adma.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/mediatek/mtk-hsdma.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/milbeaut-hdmac.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/milbeaut-xdmac.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/nbpfaxi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/qcom/bam_dma.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/qcom/qcom_adm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/sh/rcar-dmac.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/sun6i-dma.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/tegra20-apb-dma.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/dma/ti/omap-dma.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/edac/al_mc_edac.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/edac/edac_core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/edac/xgene_edac.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/extcon/extcon-ptn5150.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/extcon/extcon-qcom-spmi-misc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/extcon/extcon-usb-gpio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/firmware/arm_scpi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/firmware/broadcom/tee_bnxt_fw.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/firmware/turris-mox-rwtm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/fsi/fsi-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/fsi/fsi-master-aspeed.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/fsi/fsi-master-ast-cf.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/fsi/fsi-master-gpio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/fsi/fsi-scom.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gnss/gnss-sirf.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gnss/gnss.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-74xx-mmio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-adnp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-adp5588.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-amd-fch.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-bd70528.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-bd71828.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-clps711x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-dwapb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-eic-sprd.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-hisi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-kempld.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-logicvc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-lpc18xx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-madera.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-max732x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-max77620.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-menz127.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-mxc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-pca953x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-pca9570.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-pmic-eic-sprd.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-rcar.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-siox.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-tegra186.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-wcd934x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-wm831x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/gpio/gpio-xgs-iproc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hsi/controllers/omap_ssi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ad7414.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adc128d818.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adm1029.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ads7828.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adt7410.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adt7462.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adt7475.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/adt7x10.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/bt1-pvt.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/da9055-hwmon.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/dme1737.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ds1621.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/f71805f.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/f71882fg.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/f75375s.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/gl520sm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/hih6130.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ibmaem.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ibmpex.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ina209.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ina2xx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/it87.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/lm63.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/lm83.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/lm90.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ltc2947-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ltc2947-i2c.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ltc2992.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ltc4245.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/ltc4260.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/max16065.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/max1619.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/max6639.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/mlxreg-fan.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/mr75203.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/nct6683.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/nct6775.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pcf8591.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/adm1266.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/adm1275.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/bel-pfe.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/fsp-3y.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/inspur-ipsps.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/ir36021.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/ir38064.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/lm25066.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/ltc2978.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/max15301.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/max16064.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/max20751.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/max34440.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/max8688.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/mp2975.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/pm6764tr.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/pmbus_core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/pxe1610.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/stpddc60.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/tps40422.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/tps53679.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/ucd9200.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/xdpe12284.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/pmbus/zl6100.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/raspberrypi-hwmon.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/sht21.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/sht3x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/sl28cpld-hwmon.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/smm665.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/stts751.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/tmp103.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/tmp421.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/vt1211.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/w83795.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/hwmon/wm8350-hwmon.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/algos/i2c-algo-pcf.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-at91.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-bcm-iproc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-bcm2835.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-davinci.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-designware-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-designware-platform.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-elektor.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-exynos5.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-jz4780.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-kempld.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-lpc2k.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-mlxcpld.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-mt7621.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-mv64xxx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-mxs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-ocores.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-omap.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-pmcmsp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-pxa.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-qcom-cci.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-tegra-bpmp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-uniphier.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/busses/i2c-xlp9xx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/i2c-dev.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/i2c-mux.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/muxes/i2c-mux-mlxcpld.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/muxes/i2c-mux-pinctrl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i2c/muxes/i2c-mux-reg.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i3c/i3c.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i3c/master/dw-i3c-master.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i3c/master/i3c-master-cdns.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i3c/master/mipi-i3c-hci/mipi-i3c-hci.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/i3c/master/svc-i3c-master.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/ide/ide-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/ide/ide_platform.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/altera_ps2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/apbps2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/arc_ps2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/olpc_apsp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/ps2mult.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/sa1111ps2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/serio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/input/serio/serio_raw.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/ipack/ipack.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/irqchip/irq-madera.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/blink/leds-lgm-sso.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/led-class-multicolor.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/led-class.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-acer-a500.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-ariel.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-aw2013.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-bcm6328.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-bd2802.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-cobalt-qube.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-gpio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-ip30.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-is31fl32xx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lm3532.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lm3533.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lm355x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lm3642.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lp3952.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lp8860.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-lt3593.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-max77650.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-mlxreg.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-mt6323.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-netxbig.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-ns2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-ot200.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-pca963x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-pwm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-tca6507.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-tlc591xx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-tps6105x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-turris-omnia.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/leds-wm831x-status.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/trigger/ledtrig-activity.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/trigger/ledtrig-oneshot.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/trigger/ledtrig-pattern.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/leds/uleds.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mailbox/bcm-pdc-mailbox.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mailbox/mailbox-test.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mailbox/mtk-cmdq-mailbox.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mailbox/platform_mhu.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mailbox/qcom-apcs-ipc-mailbox.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mcb/mcb-lpc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-bio-prison.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-bufio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-cache.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-clone.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-crypt.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-delay.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-dust.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-ebs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-era.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-historical-service-time.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-log.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-mirror.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-mod.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-multipath.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-queue-length.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-region-hash.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-round-robin.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-service-time.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-snapshot.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-thin-pool.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-verity.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/dm-writecache.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/md/persistent-data/dm-persistent-data.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/memory/mtk-smi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/bcm590xx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/da9062-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/da9063.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/gateworks-gsc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/hi6421-pmic-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/iqs62x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/khadas-mcu.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/lp3943.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/madera-i2c.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/madera.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/max14577.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/max77650.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/max77693.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/max8907.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/mc13xxx-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/mc13xxx-i2c.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/mt6397.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/ntxec.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/qcom-pm8xxx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/rave-sp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/retu-mfd.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/rohm-bd70528.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/rohm-bd71828.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/sec-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/sec-irq.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/si476x-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/sky81452.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/ssbi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/stm32-lptimer.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/stmfx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/ti_am335x_tscadc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/tps65010.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/tps6507x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/tps65217.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/tps65912-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/tps65912-i2c.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mfd/wcd934x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/ad525x_dpot.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/bh1770glc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/c2port/core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/dummy-irq.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/eeprom/at24.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/eeprom/idt_89hpesx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/eeprom/max6875.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/hmc6352.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/isl29003.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/misc/qcom-coincell.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/core/mmc_core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/core/pwrseq_simple.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/cqhci.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/meson-gx-mmc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/meson-mx-sdhc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/meson-mx-sdio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/moxart-mmc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/mvsdio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/omap_hsmmc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/owl-mmc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/pxamci.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/s3cmci.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/sdhci.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mmc/host/sunxi-mmc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mux/mux-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mux/mux-gpio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/mux/mux-mmio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem-imx-ocotp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem-rave-sp-eeprom.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem-rmem.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem-rockchip-otp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem_brcm_nvram.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem_jz4780_efuse.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem_rockchip_efuse.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/nvmem/nvmem_sprd_efuse.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/perf/arm_dmc620_pmu.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/allwinner/phy-sun50i-usb3.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/amlogic/phy-meson-axg-mipi-dphy.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/amlogic/phy-meson-gxl-usb2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/amlogic/phy-meson8b-usb2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/broadcom/phy-bcm-sr-pcie.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/cadence/cdns-dphy.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/cadence/phy-cadence-sierra.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/cadence/phy-cadence-torrent.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/freescale/phy-fsl-imx8mq-usb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/hisilicon/phy-hi6220-usb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/hisilicon/phy-histb-combphy.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/ingenic/phy-ingenic-usb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/lantiq/phy-lantiq-rcu-usb2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/lantiq/phy-lantiq-vrx200-pcie.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/marvell/phy-armada38x-comphy.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/marvell/phy-berlin-usb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/marvell/phy-mmp3-hsic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/marvell/phy-pxa-usb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/mediatek/phy-mtk-hdmi-drv.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/mediatek/phy-mtk-ufs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/mediatek/phy-mtk-xsphy.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/microchip/sparx5_serdes.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/motorola/phy-mapphone-mdm6600.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/mscc/phy-ocelot-serdes.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/phy-xgene.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/qualcomm/phy-ath79-usb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/qualcomm/phy-qcom-ipq806x-usb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/qualcomm/phy-qcom-usb-hs-28nm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/ralink/phy-ralink-usb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/rockchip/phy-rockchip-dphy-rx0.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/rockchip/phy-rockchip-inno-dsidphy.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/samsung/phy-exynos-dp-video.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/samsung/phy-samsung-ufs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/socionext/phy-uniphier-usb3hs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/socionext/phy-uniphier-usb3ss.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/tegra/phy-tegra194-p2u.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/ti/phy-am654-serdes.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/phy/ti/phy-j721e-wiz.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/cirrus/pinctrl-lochnagar.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/cirrus/pinctrl-madera.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/mediatek/pinctrl-mt6779.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-da850-pupd.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-equilibrium.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-mcp23s08.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-mcp23s08_i2c.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-rockchip.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-single.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pinctrl/pinctrl-stmfx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/platform/mellanox/mlxreg-io.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/acer_a500_battery.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/adp5061.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bd70528-charger.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bd99954-charger.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bq24257_charger.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bq2515x_charger.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bq25890_charger.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bq25980_charger.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/bq27xxx_battery_i2c.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/isp1704_charger.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/lp8727_charger.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/max77650-charger.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/max77693_charger.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/rt5033_battery.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/tps65217_charger.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/ucs1002_power.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/wm831x_power.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/power/supply/z2_battery.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-atmel-tcb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-hibvt.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-iqs620a.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-lpc18xx-sct.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-ntxec.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-omap-dmtimer.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-pca9685.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-pxa.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-raspberrypi-poe.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-rcar.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-samsung.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-sl28cpld.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-stm32.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/pwm/pwm-tegra.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/88pg86x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/aat2870-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/act8945a-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/ad5398.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/anatop-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/bcm590xx-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/bd70528-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/bd71828-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/da9063-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/da9211-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/gpio-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/hi6421-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/isl9305.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/lp3971.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/lp3972.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/ltc3676.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max14577-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max77650-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max77686-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max77802-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max77826-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/max8649.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/mc13783-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/mc13xxx-regulator-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/mp5416.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/mt6311-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/mt6397-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/pf8x00-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/pv88080-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/pwm-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/qcom-labibb-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/qcom-rpmh-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/rc5t583-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/rohm-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/rpi-panel-attiny-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/rt4801-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/s2mpa01.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/sky81452-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/slg51000-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/stpmic1_regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/tps6507x-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/tps65132-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/tps65217-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/uniphier-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/regulator/vctrl-regulator.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/reset/hisilicon/hi6220_reset.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/reset/reset-meson.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/reset/reset-qcom-pdc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/reset/reset-ti-syscon.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-ab-eoz9.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-aspeed.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-brcmstb-waketimer.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-cadence.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-da9063.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-ds1553.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-ds1685.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-ds2404.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-em3027.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-fm3130.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-fsl-ftm-alarm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-generic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-goldfish.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-hym8563.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-isl1208.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-lpc24xx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-mc13xxx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-msm6242.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-mt2712.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-mt7622.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-mxc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-ntxec.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-pcf85363.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-pcf8583.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-pxa.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-rx8025.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-rx8581.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-snvs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-vt8500.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/rtc/rtc-wm8350.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/aha152x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ch.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/fdomain.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/fdomain_isa.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/g_NCR5380.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/iscsi_boot_sysfs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/libsas/libsas.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/qlogicfas.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/qlogicfas408.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/raid_class.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/scsi_debug.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/scsi_mod.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/scsi_transport_sas.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/scsi_transport_spi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/sd_mod.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ses.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/sg.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/sr_mod.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ufs/ufs-exynos.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ufs/ufs-hisi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ufs/ufshcd-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/scsi/ufs/ufshcd-pltfrm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/siox/siox-bus-gpio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/siox/siox-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/slimbus/slimbus.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/amlogic/meson-canvas.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/amlogic/meson-clk-measure.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/fsl/dpaa2-console.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/litex/litex_soc_ctrl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/mediatek/mtk-cmdq-helper.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/cmd-db.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/qcom-geni-se.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/qcom_rpmh.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/rpmhpd.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/smd-rpm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/soc/qcom/wcnss_ctrl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/loopback/tcm_loop.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/sbp/sbp_target.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_iblock.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_mod.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/target/target_core_pscsi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/tee/tee.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/bcm2711_thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/bcm2835_thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/broadcom/brcmstb_thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/da9062-thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/dove_thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/hisi_thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/imx8mm_thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/imx_thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/k3_bandgap.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/khadas_mcu_fan.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/mtk_thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/rcar_gen3_thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/sprd_thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/thermal/sun8i_thermal.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/tty/serdev/serdev.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/uio/uio_dmem_genirq.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns-usb-common.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3-imx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3-ti.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/cdns3/cdns3.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_imx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_tegra.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/chipidea/usbmisc_imx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/dwc2/dwc2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_fs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_hid.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_printer.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/function/usb_f_tcm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_ffs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_hid.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/g_printer.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/gadgetfs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/raw_gadget.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/libcomposite.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/aspeed-vhub/aspeed-vhub.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/fotg210-udc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/gr_udc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/lpc32xx_udc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/mv_udc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/net2272.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/renesas_usb3.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/snps_udc_core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/snps_udc_plat.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/udc-core.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/gadget/udc/udc-xilinx.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/isp1760/isp1760.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/mtu3/mtu3.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/jz4740.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/musb_dsps.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/musb_hdrc.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/musb/tusb6010.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-am335x-control.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-am335x.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-generic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-isp1301.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/usb/phy/phy-tahvo.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/mdev/mdev.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/vfio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/vfio/vfio_iommu_type1.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/adp8860_bl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/adp8870_bl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/arcxcnn_bl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/backlight.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/da903x_bl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/led_bl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lm3533_bl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lm3630a_bl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lp855x_bl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/lv5207lp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/pwm_bl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/qcom-wled.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/sky81452-backlight.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/tps65217_bl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/backlight/wm831x_bl.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/arcfb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/atmel_lcdfb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/broadsheetfb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/core/fb_sys_fops.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/goldfishfb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/imxfb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/mmp/fb/mmpfb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/mmp/mmp_disp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/connector-dvi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/connector-hdmi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/encoder-opa362.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-dpi.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-dsi-cm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/omap2/omapfb/displays/panel-sharp-ls037v7dw01.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pvr2fb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pxa168fb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/pxa3xx-gcu.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/s3c-fb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/sh_mobile_lcdcfb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/ssd1307fb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/video/fbdev/vfb.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/ds1wm.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/ds2482.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/mxc_w1.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/masters/w1-gpio.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds2408.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds2438.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_ds28e17.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/drivers/w1/slaves/w1_smem.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/adfs/adfs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/cramfs/cramfs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/exfat/exfat.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ext2/ext2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ext4/ext4-inode-test.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/fat/vfat.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/freevxfs/freevxfs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/gfs2/gfs2.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/hfs/hfs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/hfsplus/hfsplus.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-centeuro.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-croatian.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-cyrillic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-gaelic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/mac-roman.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_ascii.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp1250.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp1255.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp437.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp850.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp857.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp860.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp869.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp932.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp936.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_cp950.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_euc-jp.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_iso8859-1.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/nls/nls_utf8.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/ntfs/ntfs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/overlayfs/overlay.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/qnx6/qnx6.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/quota/quota_v1.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/squashfs/squashfs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/fs/unicode/utf8-selftest.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/configs.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/locking/test-ww_mutex.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/resource_kunit.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/kernel/trace/ring_buffer_benchmark.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/bitfield_kunit.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crc8.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crypto/libchacha20poly1305.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/crypto/libcurve25519-generic.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/globtest.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/kunit/kunit.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/lz4/lz4_compress.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/lz4/lz4_decompress.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/math/prime_numbers.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/math/test_div64.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/notifier-error-inject.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/parman.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/percpu_test.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/rbtree_test.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_bitmap.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_bits.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_free_pages.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_linear_ranges.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_meminit.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_module.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_parman.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_sort.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_stackinit.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_ubsan.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/lib/test_uuid.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/z3fold.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/zbud.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/mm/zpool.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/configfs/configfs_sample.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/bytestream-example.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/dma-example.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/inttype-example.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/kfifo/record-example.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/rpmsg/rpmsg_client_sample.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/trace_events/trace-events-sample.ko.gz', needed by '__modinst'.
> make[2]: *** No rule to make target '/tmp/kernel/arm-randconfig-c023-20210621/gcc-9.3.0/3c3ee16532c1be92350a2a88bd19283b7bdf32e9/lib/modules/5.13.0-rc3+/kernel/samples/vfio-mdev/mdpy-fb.ko.gz', needed by '__modinst'.
> 
> Error/Warning ids grouped by kconfigs:
> 
> gcc_recent_errors
> `-- arm-randconfig-c023-20210621
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-block-bfq.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-block-mq-deadline.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-authenc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-authencesn.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-blake2b_generic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-blowfish_common.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-blowfish_generic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-cast6_generic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-cmac.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-cts.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-curve25519-generic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-ecdsa_generic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-essiv.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-fcrypt.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-lz4.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-pcbc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-rmd160.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-seqiv.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-sha3_generic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-crypto-sha512_generic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-auxdisplay-hd44780_common.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-base-regmap-regmap-slimbus.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-block-cryptoloop.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-block-loop.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-block-virtio_blk.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-cdrom-cdrom.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-dtlk.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-hw_random-cctrng.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-hw_random-exynos-trng.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-hw_random-imx-rngc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-hw_random-stm32-rng.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-bt-bmc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-ipmi_msghandler.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-ipmi_poweroff.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-ipmi_si.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-ipmi_ssif.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-ipmi-ipmi_watchdog.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-tpm-st33zp24-tpm_st33zp24.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-tpm-st33zp24-tpm_st33zp24_i2c.ko.gz-needed-by-__modins
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-char-tpm-tpm_infineon.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-bcm-clk-raspberrypi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-cdce706.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-cdce925.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-cs2000-cp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-s2mps11.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-scmi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-clk-scpi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-a7-pll.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-apss-ipq-pll.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-camcc-sc7180.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-camcc-sdm845.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-ipq6018.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-ipq8074.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-msm8996.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-sc7280.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-sc818.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-sdm660.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-sdm845.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gcc-sdx55.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gpucc-sc7180.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gpucc-sdm660.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-gpucc-sdm845.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-hfpll.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-lcc-msm8960.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-lpasscc-sdm845.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-turingcc-qcs404.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-qcom-videocc-sm8150.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-samsung-clk-exynos-audss.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-clk-ti-adpll.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-comedi-comedi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-comedi-drivers-.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-comedi-drivers-comedi_8255.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-counter-counter.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-counter-ftm-quaddec.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-counter-ti-eqep.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dax-dax.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-devfreq-event-exynos-nocp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-devfreq-event-rockchip-dfi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-devfreq-exynos-bus.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-devfreq-governor_passive.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-devfreq-governor_userspace.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-buf-dmabuf_selftests.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-dma-jz4780.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-dmatest.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-fsl-qdma.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-iop-adma.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-mediatek-mtk-hsdma.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-milbeaut-hdmac.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-milbeaut-xdmac.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-nbpfaxi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-qcom-bam_dma.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-qcom-qcom_adm.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-sh-rcar-dmac.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-sun6i-dma.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-tegra20-apb-dma.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-dma-ti-omap-dma.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-edac-al_mc_edac.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-edac-edac_core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-edac-xgene_edac.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-extcon-extcon-ptn5150.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-extcon-extcon-qcom-spmi-misc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-extcon-extcon-usb-gpio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-firmware-arm_scpi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-firmware-broadcom-tee_bnxt_fw.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-firmware-turris-mox-rwtm.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-fsi-fsi-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-fsi-fsi-master-aspeed.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-fsi-fsi-master-ast-cf.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-fsi-fsi-master-gpio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-fsi-fsi-scom.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gnss-gnss-sirf.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gnss-gnss.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-74xx-mmio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-adnp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-adp5588.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-amd-fch.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-bd70528.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-bd71828.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-clps711x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-dwapb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-eic-sprd.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-hisi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-kempld.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-logicvc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-lpc18xx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-madera.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-max732x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-max77620.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-menz127.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-mxc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-pca953x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-pca9570.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-pmic-eic-sprd.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-rcar.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-siox.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-tegra186.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-wcd934x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-wm831x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-gpio-gpio-xgs-iproc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hsi-controllers-omap_ssi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ad7414.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adc128d818.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adm1029.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ads7828.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adt7410.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adt7462.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adt7475.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-adt7x10.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-bt1-pvt.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-da9055-hwmon.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-dme1737.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ds1621.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-f71805f.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-f71882fg.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-f75375s.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-gl520sm.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-hih6130.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ibmaem.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ibmpex.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ina209.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ina2xx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-it87.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-lm63.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-lm83.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-lm90.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ltc2947-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ltc2947-i2c.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ltc2992.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ltc4245.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-ltc4260.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-max16065.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-max1619.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-max6639.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-mlxreg-fan.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-mr75203.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-nct6683.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-nct6775.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pcf8591.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-adm1266.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-adm1275.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-bel-pfe.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-fsp-3y.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-inspur-ipsps.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-ir36021.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-ir38064.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-lm25066.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-ltc2978.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-max15301.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-max16064.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-max20751.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-max34440.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-max8688.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-mp2975.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-pm6764tr.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-pmbus_core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-pxe1610.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-stpddc60.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-tps40422.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-tps53679.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-ucd9200.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-xdpe12284.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-pmbus-zl6100.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-raspberrypi-hwmon.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-sht21.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-sht3x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-sl28cpld-hwmon.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-smm665.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-stts751.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-tmp103.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-tmp421.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-vt1211.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-w83795.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-hwmon-wm8350-hwmon.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-algos-i2c-algo-pcf.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-at91.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-bcm-iproc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-bcm2835.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-davinci.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-designware-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-designware-platform.ko.gz-needed-by-__modins
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-elektor.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-exynos5.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-jz4780.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-kempld.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-lpc2k.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-mlxcpld.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-mt7621.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-mv64xxx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-mxs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-ocores.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-omap.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-pmcmsp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-pxa.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-qcom-cci.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-tegra-bpmp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-uniphier.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-busses-i2c-xlp9xx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-i2c-dev.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-i2c-mux.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-muxes-i2c-mux-mlxcpld.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-muxes-i2c-mux-pca9541.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-muxes-i2c-mux-pinctrl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i2c-muxes-i2c-mux-reg.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i3c-i3c.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i3c-master-dw-i3c-master.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i3c-master-i3c-master-cdns.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i3c-master-mipi-i3c-hci-mipi-i3c-hci.ko.gz-needed-by-__modi
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-i3c-master-svc-i3c-master.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-ide-ide-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-ide-ide_platform.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-altera_ps2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-apbps2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-arc_ps2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-olpc_apsp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-ps2mult.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-sa1111ps2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-serio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-input-serio-serio_raw.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-ipack-ipack.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-irqchip-irq-madera.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-blink-leds-lgm-sso.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-led-class-multicolor.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-led-class.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-acer-a500.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-ariel.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-aw2013.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-bcm6328.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-bd2802.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-cobalt-qube.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-gpio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-ip30.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-is31fl32xx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lm3532.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lm3533.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lm355x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lm3642.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lp3952.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lp8860.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-lt3593.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-max77650.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-mlxreg.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-mt6323.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-netxbig.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-ns2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-ot200.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-pca963x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-pwm.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-tca6507.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-tlc591xx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-tps6105x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-turris-omnia.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-leds-wm831x-status.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-trigger-ledtrig-activity.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-trigger-ledtrig-oneshot.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-trigger-ledtrig-pattern.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-leds-uleds.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mailbox-bcm-pdc-mailbox.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mailbox-mailbox-test.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mailbox-mtk-cmdq-mailbox.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mailbox-platform_mhu.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mailbox-qcom-apcs-ipc-mailbox.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mcb-mcb-lpc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-bio-prison.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-bufio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-cache.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-clone.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-crypt.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-delay.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-dust.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-ebs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-era.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-historical-service-time.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-log.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-mirror.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-mod.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-multipath.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-queue-length.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-region-hash.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-round-robin.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-service-time.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-snapshot.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-thin-pool.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-verity.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-dm-writecache.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-md-persistent-data-dm-persistent-data.ko.gz-needed-by-__mod
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-memory-mtk-smi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-bcm59x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-da9062-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-da9063.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-gateworks-gsc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-hi6421-pmic-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-iqs62x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-khadas-mcu.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-lp3943.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-madera-i2c.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-madera.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-max14577.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-max77650.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-max77693.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-max8907.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-mc13xxx-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-mc13xxx-i2c.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-mt6397.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-ntxec.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-qcom-pm8xxx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-rave-sp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-retu-mfd.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-rohm-bd70528.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-rohm-bd71828.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-sec-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-sec-irq.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-si476x-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-sky81452.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-ssbi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-stm32-lptimer.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-stmfx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-ti_am335x_tscadc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-tps65010.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-tps6507x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-tps65217.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-tps65912-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-tps65912-i2c.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mfd-wcd934x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-ad525x_dpot.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-bh1770glc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-c2port-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-dummy-irq.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-eeprom-at24.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-eeprom-idt_89hpesx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-eeprom-max6875.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-hmc6352.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-isl29003.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-misc-qcom-coincell.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-core-mmc_core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-core-pwrseq_simple.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-cqhci.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-meson-gx-mmc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-meson-mx-sdhc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-meson-mx-sdio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-moxart-mmc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-mvsdio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-omap_hsmmc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-owl-mmc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-pxamci.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-s3cmci.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-sdhci.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mmc-host-sunxi-mmc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mux-mux-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mux-mux-gpio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-mux-mux-mmio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem-imx-ocotp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem-rave-sp-eeprom.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem-rmem.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem-rockchip-otp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem_brcm_nvram.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem_jz4780_efuse.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem_rockchip_efuse.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-nvmem-nvmem_sprd_efuse.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-perf-arm_dmc620_pmu.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-allwinner-phy-sun50i-usb3.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-amlogic-phy-meson-axg-mipi-dphy.ko.gz-needed-by-__modin
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-amlogic-phy-meson-gxl-usb2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-amlogic-phy-meson8b-usb2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-broadcom-phy-bcm-sr-pcie.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-cadence-cdns-dphy.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-cadence-phy-cadence-sierra.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-cadence-phy-cadence-torrent.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-freescale-phy-fsl-imx8mq-usb.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-hisilicon-phy-hi6220-usb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-hisilicon-phy-histb-combphy.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-ingenic-phy-ingenic-usb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-lantiq-phy-lantiq-rcu-usb2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-lantiq-phy-lantiq-vrx200-pcie.ko.gz-needed-by-__modinst
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-marvell-phy-armada38x-comphy.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-marvell-phy-berlin-usb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-marvell-phy-mmp3-hsic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-marvell-phy-pxa-usb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-mediatek-phy-mtk-hdmi-drv.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-mediatek-phy-mtk-ufs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-mediatek-phy-mtk-xsphy.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-microchip-sparx5_serdes.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-motorola-phy-mapphone-mdm6600.ko.gz-needed-by-__modinst
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-mscc-phy-ocelot-serdes.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-phy-xgene.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-qualcomm-phy-ath79-usb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-qualcomm-phy-qcom-ipq806x-usb.ko.gz-needed-by-__modinst
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-qualcomm-phy-qcom-usb-hs-28nm.ko.gz-needed-by-__modinst
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-ralink-phy-ralink-usb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-rockchip-phy-rockchip-dphy-rx0.ko.gz-needed-by-__modins
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-rockchip-phy-rockchip-inno-dsidphy.ko.gz-needed-by-__mo
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-samsung-phy-exynos-dp-video.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-samsung-phy-samsung-ufs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-socionext-phy-uniphier-usb3hs.ko.gz-needed-by-__modinst
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-socionext-phy-uniphier-usb3ss.ko.gz-needed-by-__modinst
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-tegra-phy-tegra194-p2u.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-ti-phy-am654-serdes.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-phy-ti-phy-j721e-wiz.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-cirrus-pinctrl-lochnagar.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-cirrus-pinctrl-madera.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-mediatek-pinctrl-mt6779.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-da850-pupd.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-equilibrium.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-mcp23s08.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-mcp23s08_i2c.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-rockchip.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-single.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pinctrl-pinctrl-stmfx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-platform-mellanox-mlxreg-io.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-acer_a500_battery.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-adp5061.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bd70528-charger.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bd99954-charger.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bq24257_charger.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bq2515x_charger.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bq25890_charger.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bq25980_charger.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-bq27xxx_battery_i2c.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-isp1704_charger.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-lp8727_charger.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-max77650-charger.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-max77693_charger.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-rt5033_battery.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-tps65217_charger.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-ucs1002_power.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-wm831x_power.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-power-supply-z2_battery.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-atmel-tcb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-hibvt.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-iqs620a.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-lpc18xx-sct.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-ntxec.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-omap-dmtimer.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-pca9685.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-pxa.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-raspberrypi-poe.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-rcar.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-samsung.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-sl28cpld.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-stm32.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-pwm-pwm-tegra.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-88pg86x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-aat2870-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-act8945a-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-ad5398.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-anatop-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-bcm59x-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-bd70528-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-bd71828-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-da9063-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-da9211-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-gpio-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-hi6421-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-isl9305.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-lp3971.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-lp3972.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-ltc3676.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max14577-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max77650-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max77686-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max77802-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max77826-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-max8649.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-mc13783-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-mc13xxx-regulator-core.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-mp5416.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-mt6311-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-mt6397-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-pf8x00-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-pv88080-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-pwm-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-qcom-labibb-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-qcom-rpmh-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-rc5t583-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-rohm-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-rpi-panel-attiny-regulator.ko.gz-needed-by-__modi
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-rt4801-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-s2mpa01.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-sky81452-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-slg51000-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-stpmic1_regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-tps6507x-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-tps65132-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-tps65217-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-uniphier-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-regulator-vctrl-regulator.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-reset-hisilicon-hi6220_reset.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-reset-reset-meson.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-reset-reset-qcom-pdc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-reset-reset-ti-syscon.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-ab-eoz9.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-aspeed.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-brcmstb-waketimer.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-cadence.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-da9063.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-ds1553.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-ds1685.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-ds2404.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-em3027.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-fm3130.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-fsl-ftm-alarm.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-generic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-goldfish.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-hym8563.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-isl1208.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-lpc24xx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-mc13xxx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-msm6242.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-mt2712.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-mt7622.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-mxc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-ntxec.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-pcf85363.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-pcf8583.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-pxa.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-rx8025.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-rx8581.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-snvs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-vt8500.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-rtc-rtc-wm8350.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-aha152x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ch.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-device_handler-scsi_dh_alua.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-device_handler-scsi_dh_emc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-device_handler-scsi_dh_hp_sw.ko.gz-needed-by-__modinst
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-device_handler-scsi_dh_rdac.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-fdomain.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-fdomain_isa.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-g_NCR5380.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-iscsi_boot_sysfs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-libsas-libsas.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-qlogicfas.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-qlogicfas408.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-raid_class.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-scsi_debug.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-scsi_mod.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-scsi_transport_sas.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-scsi_transport_spi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-sd_mod.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ses.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-sg.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-sr_mod.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ufs-ufs-exynos.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ufs-ufs-hisi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ufs-ufshcd-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-scsi-ufs-ufshcd-pltfrm.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-siox-siox-bus-gpio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-siox-siox-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-slimbus-slimbus.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-amlogic-meson-canvas.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-amlogic-meson-clk-measure.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-fsl-dpaa2-console.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-litex-litex_soc_ctrl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-mediatek-mtk-cmdq-helper.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-cmd-db.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-qcom-geni-se.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-qcom_rpmh.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-rpmhpd.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-smd-rpm.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-soc-qcom-wcnss_ctrl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-target-loopback-tcm_loop.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-target-sbp-sbp_target.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-target-target_core_iblock.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-target-target_core_mod.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-target-target_core_pscsi.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-tee-tee.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-broadcom-bcm2711_thermal.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-broadcom-bcm2835_thermal.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-broadcom-brcmstb_thermal.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-da9062-thermal.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-dove_thermal.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-hisi_thermal.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-imx8mm_thermal.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-imx_thermal.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-k3_bandgap.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-khadas_mcu_fan.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-mtk_thermal.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-rcar_gen3_thermal.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-sprd_thermal.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-thermal-sun8i_thermal.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-tty-serdev-serdev.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-uio-uio_dmem_genirq.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-cdns3-cdns-usb-common.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-cdns3-cdns3-imx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-cdns3-cdns3-ti.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-cdns3-cdns3.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-ci_hdrc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-ci_hdrc_imx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-ci_hdrc_msm.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-ci_hdrc_tegra.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-ci_hdrc_usb2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-chipidea-usbmisc_imx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-dwc2-dwc2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-function-usb_f_fs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-function-usb_f_hid.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-function-usb_f_mass_storage.ko.gz-needed-by-__mo
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-function-usb_f_printer.ko.gz-needed-by-__modinst
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-function-usb_f_tcm.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-g_ffs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-g_hid.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-g_mass_storage.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-g_printer.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-gadgetfs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-raw_gadget.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-legacy-tcm_usb_gadget.ko.gz-needed-by-__modinst-
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-libcomposite.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-aspeed-vhub-aspeed-vhub.ko.gz-needed-by-__mo
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-fotg210-udc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-gr_udc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-lpc32xx_udc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-mv_u3d_core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-mv_udc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-net2272.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-pxa27x_udc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-r8a66597-udc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-renesas_usb3.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-snps_udc_core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-snps_udc_plat.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-udc-core.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-gadget-udc-udc-xilinx.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-isp1760-isp1760.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-mtu3-mtu3.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-musb-jz4740.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-musb-musb_dsps.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-musb-musb_hdrc.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-musb-tusb6010.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-phy-phy-am335x-control.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-phy-phy-am335x.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-phy-phy-generic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-phy-phy-isp1301.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-usb-phy-phy-tahvo.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-vfio-mdev-mdev.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-vfio-vfio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-vfio-vfio_iommu_type1.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-adp8860_bl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-adp8870_bl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-arcxcnn_bl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-backlight.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-da903x_bl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-led_bl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-lm3533_bl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-lm3630a_bl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-lp855x_bl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-lv5207lp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-pwm_bl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-qcom-wled.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-sky81452-backlight.ko.gz-needed-by-__modins
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-tps65217_bl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-backlight-wm831x_bl.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-arcfb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-atmel_lcdfb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-broadsheetfb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-core-fb_sys_fops.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-goldfishfb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-imxfb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-mmp-fb-mmpfb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-mmp-mmp_disp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-connector-dvi.ko.gz-neede
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-connector-hdmi.ko.gz-need
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-encoder-opa362.ko.gz-need
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-panel-dpi.ko.gz-needed-by
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-panel-dsi-cm.ko.gz-needed
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-omap2-omapfb-displays-panel-sharp-ls037v7dw01.k
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-pvr2fb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-pxa168fb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-pxa3xx-gcu.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-s3c-fb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-sh_mobile_lcdcfb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-ssd1307fb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-video-fbdev-vfb.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-masters-ds1wm.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-masters-ds2482.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-masters-mxc_w1.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-masters-w1-gpio.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-slaves-w1_ds2408.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-slaves-w1_ds2438.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-slaves-w1_ds28e17.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-drivers-w1-slaves-w1_smem.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-adfs-adfs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-cramfs-cramfs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-exfat-exfat.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-ext2-ext2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-ext4-ext4-inode-test.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-fat-vfat.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-freevxfs-freevxfs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-gfs2-gfs2.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-hfs-hfs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-hfsplus-hfsplus.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-mac-centeuro.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-mac-croatian.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-mac-cyrillic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-mac-gaelic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-mac-roman.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_ascii.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp1250.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp1255.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp437.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp850.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp857.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp860.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp869.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp932.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp936.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_cp950.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_euc-jp.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_iso8859-.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-nls-nls_utf8.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-ntfs-ntfs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-overlayfs-overlay.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-qnx6-qnx6.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-quota-quota_v1.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-squashfs-squashfs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-fs-unicode-utf8-selftest.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-kernel-configs.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-kernel-locking-test-ww_mutex.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-kernel-resource_kunit.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-kernel-trace-ring_buffer_benchmark.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-bitfield_kunit.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-crc8.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-crypto-libchacha20poly1305.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-crypto-libcurve25519-generic.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-globtest.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-kunit-kunit.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-lz4-lz4_compress.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-lz4-lz4_decompress.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-math-prime_numbers.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-math-test_div64.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-notifier-error-inject.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-parman.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-percpu_test.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-rbtree_test.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_bitmap.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_bits.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_free_pages.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_linear_ranges.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_meminit.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_module.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_parman.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_sort.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_stackinit.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_ubsan.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-lib-test_uuid.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-mm-z3fold.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-mm-zbud.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-mm-zpool.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-configfs-configfs_sample.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-kfifo-bytestream-example.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-kfifo-dma-example.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-kfifo-inttype-example.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-kfifo-record-example.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-rpmsg-rpmsg_client_sample.ko.gz-needed-by-__modinst-.
>     |-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-trace_events-trace-events-sample.ko.gz-needed-by-__modinst-
>     `-- make:No-rule-to-make-target-tmp-kernel-arm-randconfig-c023-gcc-..-3c3ee16532c1be92350a2a88bd19283b7bdf32e9-lib-modules-..-rc3-kernel-samples-vfio-mdev-mdpy-fb.ko.gz-needed-by-__modinst-.
> 
> elapsed time: 723m
> 
> configs tested: 110
> configs skipped: 3
> 
> gcc tested configs:
> arm                                 defconfig
> arm64                            allyesconfig
> arm64                               defconfig
> arm                              allyesconfig
> arm                              allmodconfig
> arm                            zeus_defconfig
> arm                  colibri_pxa270_defconfig
> sh                           se7780_defconfig
> powerpc                  mpc866_ads_defconfig
> arc                     haps_hs_smp_defconfig
> sh                          landisk_defconfig
> powerpc                      ep88xc_defconfig
> mips                  maltasmvp_eva_defconfig
> s390                             alldefconfig
> m68k                          multi_defconfig
> powerpc                     tqm5200_defconfig
> m68k                        mvme16x_defconfig
> h8300                    h8300h-sim_defconfig
> mips                       lemote2f_defconfig
> powerpc                      katmai_defconfig
> mips                  decstation_64_defconfig
> m68k                        m5272c3_defconfig
> sh                         ecovec24_defconfig
> arm                          collie_defconfig
> powerpc                    sam440ep_defconfig
> powerpc                       maple_defconfig
> ia64                          tiger_defconfig
> mips                        qi_lb60_defconfig
> powerpc                     tqm8560_defconfig
> microblaze                      mmu_defconfig
> arm                         cm_x300_defconfig
> m68k                         apollo_defconfig
> mips                     decstation_defconfig
> arm                   milbeaut_m10v_defconfig
> powerpc                        warp_defconfig
> arm                        multi_v5_defconfig
> x86_64                            allnoconfig
> ia64                             allmodconfig
> ia64                                defconfig
> ia64                             allyesconfig
> m68k                             allmodconfig
> m68k                                defconfig
> m68k                             allyesconfig
> nios2                               defconfig
> arc                              allyesconfig
> nds32                             allnoconfig
> nds32                               defconfig
> nios2                            allyesconfig
> csky                                defconfig
> alpha                               defconfig
> alpha                            allyesconfig
> xtensa                           allyesconfig
> h8300                            allyesconfig
> arc                                 defconfig
> sh                               allmodconfig
> parisc                              defconfig
> s390                             allyesconfig
> s390                             allmodconfig
> parisc                           allyesconfig
> s390                                defconfig
> i386                             allyesconfig
> sparc                            allyesconfig
> sparc                               defconfig
> i386                                defconfig
> mips                             allyesconfig
> mips                             allmodconfig
> powerpc                          allyesconfig
> powerpc                          allmodconfig
> powerpc                           allnoconfig
> x86_64               randconfig-a002-20210621
> x86_64               randconfig-a001-20210621
> x86_64               randconfig-a005-20210621
> x86_64               randconfig-a003-20210621
> x86_64               randconfig-a004-20210621
> x86_64               randconfig-a006-20210621
> i386                 randconfig-a002-20210621
> i386                 randconfig-a003-20210621
> i386                 randconfig-a006-20210621
> i386                 randconfig-a005-20210621
> i386                 randconfig-a004-20210621
> i386                 randconfig-a001-20210621
> i386                 randconfig-a011-20210621
> i386                 randconfig-a014-20210621
> i386                 randconfig-a013-20210621
> i386                 randconfig-a015-20210621
> i386                 randconfig-a012-20210621
> i386                 randconfig-a016-20210621
> riscv                    nommu_k210_defconfig
> riscv                            allyesconfig
> riscv                    nommu_virt_defconfig
> riscv                             allnoconfig
> riscv                               defconfig
> riscv                          rv32_defconfig
> riscv                            allmodconfig
> x86_64                    rhel-8.3-kselftests
> um                           x86_64_defconfig
> um                             i386_defconfig
> um                            kunit_defconfig
> x86_64                           allyesconfig
> x86_64                              defconfig
> x86_64                               rhel-8.3
> x86_64                      rhel-8.3-kbuiltin
> x86_64                                  kexec
> 
> clang tested configs:
> x86_64               randconfig-b001-20210621
> x86_64               randconfig-a012-20210621
> x86_64               randconfig-a015-20210621
> x86_64               randconfig-a014-20210621
> x86_64               randconfig-a013-20210621
> x86_64               randconfig-a016-20210621
> x86_64               randconfig-a011-20210621
> 
> ---
> 0-DAY CI Kernel Test Service, Intel Corporation
> https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org


From cachen at purestorage.com  Mon Jun 21 17:27:08 2021
From: cachen at purestorage.com (Casey Chen)
Date: Mon, 21 Jun 2021 17:27:08 -0700
Subject: rr: nvme-pci: fix races in nvme_setup_io_queues() and UAF introduced
 by nvme_dev_remove_admin()
Message-ID: <20210622002710.26591-1-cachen@purestorage.com>

Found two bugs while power-cycling PCIe NVMe drive for hours:
- Races in nvme_setup_io_queues()
- UAF introduced by nvme_dev_remove_admin(), which is found
after fixing the races. Without fixing the races, the system
just crashes and cannot survive to reproduce the UAF.

The proposed fixes have been tested for several days for correctness.

0. Code baseline

Tag nvme-5.14-2021-06-08 of repo http://git.infradead.org/nvme.git

1. Testing method

while :; power off one drive;
    sleep $((RANDOM%3)).$((RANDOM%10));
    power on the same drive;
    sleep $((RANDOM%3)).$((RANDOM%10));
done

2. Sample crash trace due to races in nvme_setup_io_queues()
(task ID shown after the timestamps)

[11668.533431][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Card present
...
[11668.681298][T251231] nvme nvme12: pci function 0000:8c:00.0
[11668.681354][T26714] nvme 0000:8c:00.0: enabling device (0100 -> 0102)
[11669.046119][   C31] pcieport 0000:87:08.0: pciehp: pending interrupts 0x0108 from Slot Status
[11669.046142][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Link Down
[11669.046146][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Card not present
[11669.046149][  T716] pcieport 0000:87:08.0: pciehp: pciehp_unconfigure_device: domain:bus:dev = 0000:8c:00
[11669.077428][  T716] ------------[ cut here ]------------
[11669.077431][  T716] kernel BUG at drivers/pci/msi.c:348!
[11669.077555][  T716] invalid opcode: 0000 [#1] SMP KASAN
[11669.077658][  T716] CPU: 31 PID: 716 Comm: irq/127-pciehp Not tainted 5.13.0-rc3+
[11669.078022][  T716] RIP: 0010:free_msi_irqs+0x28a/0x2d0
...
[11669.093982][  T716] Call Trace:
[11669.096850][  T716]  pci_free_irq_vectors+0xe/0x20
[11669.099695][  T716]  nvme_dev_disable+0x140/0x760 [nvme]
[11669.102503][  T716]  ? _raw_spin_lock_irqsave+0x9c/0x100
[11669.105271][  T716]  ? trace_hardirqs_on+0x2c/0xe0
[11669.107994][  T716]  nvme_remove+0x191/0x1e0 [nvme]
[11669.110689][  T716]  pci_device_remove+0x6b/0x110
[11669.113316][  T716]  device_release_driver_internal+0x14f/0x280
[11669.115939][  T716]  pci_stop_bus_device+0xcb/0x100
[11669.118515][  T716]  pci_stop_and_remove_bus_device+0xe/0x20
[11669.121079][  T716]  pciehp_unconfigure_device+0xfa/0x200
[11669.123597][  T716]  ? pciehp_configure_device+0x1c0/0x1c0
[11669.126049][  T716]  ? trace_hardirqs_on+0x2c/0xe0
[11669.128444][  T716]  pciehp_disable_slot+0xc4/0x1a0
[11669.130771][  T716]  ? pciehp_runtime_suspend+0x40/0x40
[11669.133054][  T716]  ? __mutex_lock_slowpath+0x10/0x10
[11669.135289][  T716]  ? trace_hardirqs_on+0x2c/0xe0
[11669.137462][  T716]  pciehp_handle_presence_or_link_change+0x15c/0x4f0
[11669.139632][  T716]  ? down_read+0x11f/0x1a0
[11669.141731][  T716]  ? pciehp_handle_disable_request+0x80/0x80
[11669.143817][  T716]  ? rwsem_down_read_slowpath+0x600/0x600
[11669.145851][  T716]  ? __radix_tree_lookup+0xb2/0x130
[11669.147842][  T716]  pciehp_ist+0x19d/0x1a0
[11669.149790][  T716]  ? pciehp_set_indicators+0xe0/0xe0
[11669.151704][  T716]  ? irq_finalize_oneshot.part.46+0x1d0/0x1d0
[11669.153588][  T716]  irq_thread_fn+0x3f/0xa0
[11669.155407][  T716]  irq_thread+0x195/0x290
[11669.157147][  T716]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
[11669.158883][  T716]  ? _raw_read_lock_irq+0x50/0x50
[11669.160611][  T716]  ? _raw_read_lock_irq+0x50/0x50
[11669.162320][  T716]  ? irq_forced_thread_fn+0xf0/0xf0
[11669.164032][  T716]  ? trace_hardirqs_on+0x2c/0xe0
[11669.165731][  T716]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
[11669.167461][  T716]  kthread+0x1c8/0x1f0
[11669.169173][  T716]  ? kthread_parkme+0x40/0x40
[11669.170883][  T716]  ret_from_fork+0x22/0x30

3. KASAN report for the UAF introduced by nvme_dev_remove_admin()
(task ID is shown after the timestamp)

[18319.015748][T246989] nvme nvme13: pci function 0000:8c:00.0
[18319.015795][T215541] nvme 0000:8c:00.0: enabling device (0100 -> 0102)
[18319.369086][   C31] pcieport 0000:87:08.0: pciehp: pending interrupts 0x0108 from Slot Status
[18319.369107][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Link Down
[18319.369111][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Card not present
[18319.369116][  T716] pcieport 0000:87:08.0: pciehp: pciehp_unconfigure_device: domain:bus:dev = 0000:8c:00
[18320.452045][T215541] nvme nvme13: 88/0/0 default/read/poll queues
[18320.469475][T215541] nvme nvme13: failed to mark controller live state
[18320.469483][T215541] nvme nvme13: Removing after probe failure status: -19
[18320.551295][T215541] ==================================================================
[18320.551299][T215541] BUG: KASAN: use-after-free in __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551311][T215541] Read of size 4 at addr ffff888897904d04 by task kworker/u178:2/215541
[18320.551315][T215541] 
[18320.551318][T215541] CPU: 86 PID: 215541 Comm: kworker/u178:2 Not tainted 5.13.0-rc3+
[18320.551327][T215541] Workqueue: nvme-reset-wq nvme_reset_work [nvme]
[18320.551339][T215541] Call Trace:
[18320.551343][T215541]  dump_stack+0xa4/0xdb
[18320.551354][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551359][T215541]  print_address_description.constprop.10+0x3a/0x60
[18320.551366][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551372][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551377][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
[18320.551382][T215541]  kasan_report.cold.15+0x7c/0xd8
[18320.551390][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551395][T215541]  __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551401][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
[18320.551407][T215541]  ? bt_iter+0xf0/0xf0
[18320.551412][T215541]  ? __blk_mq_all_tag_iter+0x2c9/0x3f0
[18320.551417][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
[18320.551422][T215541]  ? bt_iter+0xf0/0xf0
[18320.551427][T215541]  ? dev_printk_emit+0x95/0xbb
[18320.551436][T215541]  blk_mq_tagset_busy_iter+0x75/0xa0
[18320.551441][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
[18320.551446][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
[18320.551451][T215541]  blk_mq_tagset_wait_completed_request+0x86/0xc0
[18320.551457][T215541]  ? blk_mq_tagset_busy_iter+0xa0/0xa0
[18320.551463][T215541]  ? blk_mq_tagset_busy_iter+0x80/0xa0
[18320.551469][T215541]  ? trace_event_raw_event_nvme_setup_cmd+0x2d0/0x2d0 [nvme_core]
[18320.551493][T215541]  nvme_dev_disable+0x4f6/0x760 [nvme]
[18320.551502][T215541]  ? trace_hardirqs_on+0x2c/0xe0
[18320.551510][T215541]  nvme_reset_work+0x226/0x2060 [nvme]
[18320.551520][T215541]  ? nvme_remove+0x1e0/0x1e0 [nvme]
[18320.551528][T215541]  ? __update_load_avg_cfs_rq+0x1d8/0x550
[18320.551537][T215541]  ? down_read+0x11f/0x1a0
[18320.551545][T215541]  ? newidle_balance+0x444/0x690
[18320.551552][T215541]  ? update_load_avg+0x626/0xbe0
[18320.551557][T215541]  ? update_cfs_group+0x1e/0x150
[18320.551562][T215541]  ? load_balance+0x11d0/0x11d0
[18320.551567][T215541]  ? dequeue_entity+0x150/0x730
[18320.551573][T215541]  ? nvme_irq_check+0x60/0x60 [nvme]
[18320.551581][T215541]  ? finish_task_switch+0x101/0x3d0
[18320.551588][T215541]  ? read_word_at_a_time+0xe/0x20
[18320.551594][T215541]  ? strscpy+0xc1/0x1d0
[18320.551598][T215541]  process_one_work+0x4b9/0x7b0
[18320.551604][T215541]  worker_thread+0x72/0x710
[18320.551610][T215541]  ? process_one_work+0x7b0/0x7b0
[18320.551614][T215541]  kthread+0x1c8/0x1f0
[18320.551618][T215541]  ? kthread_parkme+0x40/0x40
[18320.551622][T215541]  ret_from_fork+0x22/0x30
[18320.551630][T215541] 
[18320.551632][T215541] Allocated by task 215541:
[18320.551635][T215541]  kasan_save_stack+0x19/0x40
[18320.551639][T215541]  __kasan_kmalloc+0x7f/0xa0
[18320.551642][T215541]  kmem_cache_alloc_node_trace+0x187/0x2b0
[18320.551648][T215541]  blk_mq_init_tags+0x47/0x100
[18320.551651][T215541]  blk_mq_alloc_rq_map+0x44/0xf0
[18320.551656][T215541]  __blk_mq_alloc_map_and_request+0x7f/0x140
[18320.551661][T215541]  blk_mq_alloc_tag_set+0x25e/0x510
[18320.551666][T215541]  nvme_reset_work+0x14f9/0x2060 [nvme]
[18320.551674][T215541]  process_one_work+0x4b9/0x7b0
[18320.551678][T215541]  worker_thread+0x72/0x710
[18320.551682][T215541]  kthread+0x1c8/0x1f0
[18320.551685][T215541]  ret_from_fork+0x22/0x30
[18320.551689][T215541] 
[18320.551690][T215541] Freed by task 716:
[18320.551693][T215541]  kasan_save_stack+0x19/0x40
[18320.551696][T215541]  kasan_set_track+0x1c/0x30
[18320.551699][T215541]  kasan_set_free_info+0x20/0x30
[18320.551704][T215541]  __kasan_slab_free+0xec/0x130
[18320.551707][T215541]  kfree+0xa8/0x460
[18320.551712][T215541]  blk_mq_free_map_and_requests+0x8d/0xc0
[18320.551717][T215541]  blk_mq_free_tag_set+0x30/0xf0
[18320.551721][T215541]  nvme_remove+0x199/0x1e0 [nvme]
[18320.551729][T215541]  pci_device_remove+0x6b/0x110
[18320.551735][T215541]  device_release_driver_internal+0x14f/0x280
[18320.551744][T215541]  pci_stop_bus_device+0xcb/0x100
[18320.551750][T215541]  pci_stop_and_remove_bus_device+0xe/0x20
[18320.551754][T215541]  pciehp_unconfigure_device+0xfa/0x200
[18320.551761][T215541]  pciehp_disable_slot+0xc4/0x1a0
[18320.551765][T215541]  pciehp_handle_presence_or_link_change+0x15c/0x4f0
[18320.551770][T215541]  pciehp_ist+0x19d/0x1a0
[18320.551774][T215541]  irq_thread_fn+0x3f/0xa0
[18320.551780][T215541]  irq_thread+0x195/0x290
[18320.551783][T215541]  kthread+0x1c8/0x1f0
[18320.551786][T215541]  ret_from_fork+0x22/0x30
[18320.551791][T215541] 
[18320.551792][T215541] The buggy address belongs to the object at ffff888897904d00
[18320.551792][T215541]  which belongs to the cache kmalloc-192 of size 192
[18320.551795][T215541] The buggy address is located 4 bytes inside of
[18320.551795][T215541]  192-byte region [ffff888897904d00, ffff888897904dc0)
[18320.551800][T215541] The buggy address belongs to the page:
[18320.551802][T215541] page:000000002f3df664 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x897904
[18320.551807][T215541] head:000000002f3df664 order:1 compound_mapcount:0
[18320.551810][T215541] flags: 0x200000000010200(slab|head|node=0|zone=2)
[18320.551819][T215541] raw: 0200000000010200 dead000000000100 dead000000000122 ffff88810004ca00
[18320.551824][T215541] raw: 0000000000000000 0000000000200020 00000001ffffffff 0000000000000000
[18320.551826][T215541] page dumped because: kasan: bad access detected
[18320.551828][T215541] 
[18320.551829][T215541] Memory state around the buggy address:
[18320.551832][T215541]  ffff888897904c00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[18320.551835][T215541]  ffff888897904c80: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
[18320.551838][T215541] >ffff888897904d00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[18320.551841][T215541]                    ^
[18320.551843][T215541]  ffff888897904d80: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
[18320.551846][T215541]  ffff888897904e00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[18320.551848][T215541] ==================================================================
[18320.551850][T215541] Disabling lock debugging due to kernel taint




From cachen at purestorage.com  Mon Jun 21 17:27:09 2021
From: cachen at purestorage.com (Casey Chen)
Date: Mon, 21 Jun 2021 17:27:09 -0700
Subject: [PATCH 1/2] nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <20210622002710.26591-1-cachen@purestorage.com>
References: <20210622002710.26591-1-cachen@purestorage.com>
Message-ID: <20210622002710.26591-2-cachen@purestorage.com>

Below two paths could overlap each other if we power off a drive quickly
after power it on. There are multiple races in nvme_setup_io_queues()
because of shutdown_lock missing and improper use of NVMEQ_ENABLED bit.

nvme_reset_work()                                nvme_remove()
  nvme_setup_io_queues()                           nvme_dev_disable()
  ...                                              ...
A1  clear NVMEQ_ENABLED bit for admin queue          lock
    retry:                                       B1  nvme_suspend_io_queues()
A2    pci_free_irq() admin queue                 B2  nvme_suspend_queue() admin queue
A3    pci_free_irq_vectors()                         nvme_pci_disable()
A4    nvme_setup_irqs();                         B3    pci_free_irq_vectors()
      ...                                            unlock
A5    queue_request_irq() for admin queue
      set NVMEQ_ENABLED bit
      ...
      nvme_create_io_queues()
A6      result = queue_request_irq();
        set NVMEQ_ENABLED bit
      ...
      fail to allocate enough IO queues:
A7      nvme_suspend_io_queues()
        goto retry

1). If B3 runs in between A1 and A2, it will crash if irqaction haven't
  been freed by A2. B2 is supposed to free admin queue IRQ but it simply
  can't fulfill the job as A1 has cleared NVMEQ_ENABLED bit.

Fix: combine A1 A2 so IRQ get freed as soon as NVMEQ_ENABLED bit get cleared.

2). After solved #1, A2 could race with B3 if A2 is freeing IRQ while B3
  is checking irqaction. A3 also could race with B2 if B2 is freeing
  IRQ while A3 is checking irqaction.

Fix: A2 and A3 take lock for mutual exclusion.

3) A3 could race with B3 since they could run free_msi_irqs() in parallel.

Fix: A3 takes lock for mutual exclusion.

4). A4 could fail to allocate all needed IRQ vectors if A3 and A4 are
  interrupted by B3.

Fix: A4 takes lock for mutual exclusion.

5). If A5/A6 happened after B2/B1, B3 will crash since irqaction is not NULL.
  They are just allocated by A5/A6.

Fix: Lock queue_request_irq() and setting of NVMEQ_ENABLED bit.

6). A7 could get chance to pci_free_irq() for certain IO queue while B3
  is checking irqaction.

Fix: A7 takes lock.

7). nvme_dev->online_queues need to be protected by shutdown_lock. Since it
  is not atomic, both paths could modify it using its own copy.

Co-developed-by: Yuanyuan Zhong <yzhong at purestorage.com>
Signed-off-by: Casey Chen <cachen at purestorage.com>
---
 drivers/nvme/host/pci.c | 37 +++++++++++++++++++++++++++++++------
 1 file changed, 31 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..5abc4c3be454 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1590,8 +1590,13 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
 		goto release_cq;
 
 	nvmeq->cq_vector = vector;
-	nvme_init_queue(nvmeq, qid);
 
+	/*
+	 * Return early if locking fails as device is being disabled.
+	 */
+	if (!mutex_trylock(&dev->shutdown_lock))
+		return -ENODEV;
+	nvme_init_queue(nvmeq, qid);
 	if (!polled) {
 		result = queue_request_irq(nvmeq);
 		if (result < 0)
@@ -1599,10 +1604,12 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
 	}
 
 	set_bit(NVMEQ_ENABLED, &nvmeq->flags);
+	mutex_unlock(&dev->shutdown_lock);
 	return result;
 
 release_sq:
 	dev->online_queues--;
+	mutex_unlock(&dev->shutdown_lock);
 	adapter_delete_sq(dev, qid);
 release_cq:
 	adapter_delete_cq(dev, qid);
@@ -2176,7 +2183,16 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	if (nr_io_queues == 0)
 		return 0;
 
-	clear_bit(NVMEQ_ENABLED, &adminq->flags);
+	/*
+	 * Free IRQ resources as soon as NVMEQ_ENABLED bit transitions
+	 * from set to unset. If there is a window to it is truely freed,
+	 * pci_free_irq_vectors() jumping into this window will crash.
+	 * And take lock to avoid racing with pci_free_irq_vectors() in
+	 * nvme_dev_disable() path.
+	 */
+	mutex_lock(&dev->shutdown_lock);
+	if (test_and_clear_bit(NVMEQ_ENABLED, &adminq->flags))
+		pci_free_irq(pdev, 0, adminq);
 
 	if (dev->cmb_use_sqes) {
 		result = nvme_cmb_qdepth(dev, nr_io_queues,
@@ -2192,14 +2208,17 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 		result = nvme_remap_bar(dev, size);
 		if (!result)
 			break;
-		if (!--nr_io_queues)
+		if (!--nr_io_queues) {
+			mutex_unlock(&dev->shutdown_lock);
 			return -ENOMEM;
+		}
 	} while (1);
 	adminq->q_db = dev->dbs;
 
  retry:
 	/* Deregister the admin queue's interrupt */
-	pci_free_irq(pdev, 0, adminq);
+	if (test_and_clear_bit(NVMEQ_ENABLED, &adminq->flags))
+		pci_free_irq(pdev, 0, adminq);
 
 	/*
 	 * If we enable msix early due to not intx, disable it again before
@@ -2208,8 +2227,10 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	pci_free_irq_vectors(pdev);
 
 	result = nvme_setup_irqs(dev, nr_io_queues);
-	if (result <= 0)
+	if (result <= 0) {
+		mutex_unlock(&dev->shutdown_lock);
 		return -EIO;
+	}
 
 	dev->num_vecs = result;
 	result = max(result - 1, 1);
@@ -2222,9 +2243,12 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	 * number of interrupts.
 	 */
 	result = queue_request_irq(adminq);
-	if (result)
+	if (result) {
+		mutex_unlock(&dev->shutdown_lock);
 		return result;
+	}
 	set_bit(NVMEQ_ENABLED, &adminq->flags);
+	mutex_unlock(&dev->shutdown_lock);
 
 	result = nvme_create_io_queues(dev);
 	if (result || dev->online_queues < 2)
@@ -2233,6 +2257,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	if (dev->online_queues - 1 < dev->max_qid) {
 		nr_io_queues = dev->online_queues - 1;
 		nvme_disable_io_queues(dev);
+		mutex_lock(&dev->shutdown_lock);
 		nvme_suspend_io_queues(dev);
 		goto retry;
 	}
-- 
2.17.1



From cachen at purestorage.com  Mon Jun 21 17:27:10 2021
From: cachen at purestorage.com (Casey Chen)
Date: Mon, 21 Jun 2021 17:27:10 -0700
Subject: [PATCH 2/2] nvme-pci: Fix UAF introduced by nvme_dev_remove_admin()
In-Reply-To: <20210622002710.26591-1-cachen@purestorage.com>
References: <20210622002710.26591-1-cachen@purestorage.com>
Message-ID: <20210622002710.26591-3-cachen@purestorage.com>

nvme_dev_remove_admin() could free admin_q and admin_tagset while
they are being accessed by nvme_dev_disable(), which could come from
nvme_remove_dead_ctrl() by nvme_reset_work() during cleanup.

Commit cb4bfda62afa ("nvme-pci: fix hot removal during error handling") was
to avoid requests being stuck on a removed controller by killing admin queue.
But the later fix c8e9e9b7646e ("nvme-pci: unquiesce admin queue on shutdown"),
together with nvme_dev_disable(dev, true) right before nvme_dev_remove_admin()
could help dispatch requests and fail them early, so we don't need
nvme_dev_remove_admin() any more.

Fixes: cb4bfda62afa ("nvme-pci: fix hot removal during error handling")

upstream_status: upstream
---
 drivers/nvme/host/pci.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 5abc4c3be454..49ef2027bcb0 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -3023,7 +3023,6 @@ static void nvme_remove(struct pci_dev *pdev)
 	if (!pci_device_is_present(pdev)) {
 		nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DEAD);
 		nvme_dev_disable(dev, true);
-		nvme_dev_remove_admin(dev);
 	}
 
 	flush_work(&dev->ctrl.reset_work);
-- 
2.17.1



From hare at suse.de  Mon Jun 21 23:21:53 2021
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 22 Jun 2021 08:21:53 +0200
Subject: [PATCHv6] nvme: allow to re-attach namespaces after all paths are
 down
In-Reply-To: <20210621172653.GC1268033@dhcp-10-100-145-180.wdc.com>
References: <20210609150118.130650-1-hare@suse.de>
 <20210621172653.GC1268033@dhcp-10-100-145-180.wdc.com>
Message-ID: <b4637359-9846-4086-2fc2-a1a990933c7b@suse.de>

On 6/21/21 7:26 PM, Keith Busch wrote:
> On Wed, Jun 09, 2021 at 05:01:18PM +0200, Hannes Reinecke wrote:
>> -void nvme_mpath_remove_disk(struct nvme_ns_head *head)
>> +void nvme_mpath_check_last_path(struct nvme_ns_head *head)
>>   {
>> +	bool last_path = false;
>>   	if (!head->disk)
>>   		return;
>> -	if (head->disk->flags & GENHD_FL_UP) {
>> -		nvme_cdev_del(&head->cdev, &head->cdev_device);
>> -		del_gendisk(head->disk);
>> +
>> +	/* Synchronize with nvme_init_ns_head() */
>> +	mutex_lock(&head->subsys->lock);
>> +	if (list_empty(&head->list))
>> +		last_path = true;
>> +	mutex_unlock(&head->subsys->lock);
>> +	if (last_path) {
>> +		kblockd_schedule_work(&head->requeue_work);
>> +		if (head->disk->flags & GENHD_FL_UP) {
>> +			nvme_cdev_del(&head->cdev, &head->cdev_device);
>> +			del_gendisk(head->disk);
>> +		}
>>   	}
>> +}
> 
> This is being called when a path is removed. If it is the last path, the
> head's block device is being deleted. If a live head reference is held,
> and the a path reconnects, the code will find the previous head, but the
> disk is gone and it's not coming back, so the new connection can't be
> used.
> 
Good point. But that can be fixed by removing the ns_head from the 
subsystem list at this check rather than in nvme_free_ns_head().
Will be sending an updated version.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From hare at suse.de  Mon Jun 21 23:31:54 2021
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 22 Jun 2021 08:31:54 +0200
Subject: [PATCHv6] nvme: allow to re-attach namespaces after all paths are
 down
In-Reply-To: <4903ef70-ed16-4b81-3570-60e9fcc5ecb0@grimberg.me>
References: <20210609150118.130650-1-hare@suse.de>
 <4903ef70-ed16-4b81-3570-60e9fcc5ecb0@grimberg.me>
Message-ID: <4a05b0de-8639-0747-e9f4-c20400854b02@suse.de>

On 6/21/21 8:13 PM, Sagi Grimberg wrote:
> 
> 
> On 6/9/21 8:01 AM, Hannes Reinecke wrote:
>> We should only remove the ns head from the list of heads per
>> subsystem if the reference count drops to zero. That cleans up
>> reference counting, and allows us to call del_gendisk() once the last
>> path is removed (as then the ns_head should be removed anyway).
>> As this introduces a (theoretical) race condition where I/O might have
>> been requeued before the last path went down we also should be checking
>> if the gendisk is still present in nvme_ns_head_submit_bio(),
>> and failing I/O if so.
>>
>> Changes to v5:
>> - Synchronize between nvme_init_ns_head() and 
>> nvme_mpath_check_last_path()
>> - Check for removed gendisk in nvme_ns_head_submit_bio()
>> Changes to v4:
>> - Call del_gendisk() in nvme_mpath_check_last_path() to avoid deadlock
>> Changes to v3:
>> - Simplify if() clause to detect duplicate namespaces
>> Changes to v2:
>> - Drop memcpy() statement
>> Changes to v1:
>> - Always check NSIDs after reattach
>>
>> Signed-off-by: Hannes Reinecke <hare at suse.de>
>> ---
>> ? drivers/nvme/host/core.c????? |? 9 ++++-----
>> ? drivers/nvme/host/multipath.c | 30 +++++++++++++++++++++++++-----
>> ? drivers/nvme/host/nvme.h????? | 11 ++---------
>> ? 3 files changed, 31 insertions(+), 19 deletions(-)
>>
>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>> index 177cae44b612..6d7c2958b3e2 100644
>> --- a/drivers/nvme/host/core.c
>> +++ b/drivers/nvme/host/core.c
>> @@ -566,6 +566,9 @@ static void nvme_free_ns_head(struct kref *ref)
>> ????? struct nvme_ns_head *head =
>> ????????? container_of(ref, struct nvme_ns_head, ref);
>> +??? mutex_lock(&head->subsys->lock);
>> +??? list_del_init(&head->entry);
>> +??? mutex_unlock(&head->subsys->lock);
>> ????? nvme_mpath_remove_disk(head);
>> ????? ida_simple_remove(&head->subsys->ns_ida, head->instance);
>> ????? cleanup_srcu_struct(&head->srcu);
>> @@ -3806,8 +3809,6 @@ static void nvme_alloc_ns(struct nvme_ctrl 
>> *ctrl, unsigned nsid,
>> ?? out_unlink_ns:
>> ????? mutex_lock(&ctrl->subsys->lock);
>> ????? list_del_rcu(&ns->siblings);
>> -??? if (list_empty(&ns->head->list))
>> -??????? list_del_init(&ns->head->entry);
>> ????? mutex_unlock(&ctrl->subsys->lock);
>> ????? nvme_put_ns_head(ns->head);
>> ?? out_free_queue:
>> @@ -3828,8 +3829,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>> ????? mutex_lock(&ns->ctrl->subsys->lock);
>> ????? list_del_rcu(&ns->siblings);
>> -??? if (list_empty(&ns->head->list))
>> -??????? list_del_init(&ns->head->entry);
>> ????? mutex_unlock(&ns->ctrl->subsys->lock);
>> ????? synchronize_rcu(); /* guarantee not available in head->list */
>> @@ -3849,7 +3848,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>> ????? list_del_init(&ns->list);
>> ????? up_write(&ns->ctrl->namespaces_rwsem);
>> -??? nvme_mpath_check_last_path(ns);
>> +??? nvme_mpath_check_last_path(ns->head);
>> ????? nvme_put_ns(ns);
>> ? }
>> diff --git a/drivers/nvme/host/multipath.c 
>> b/drivers/nvme/host/multipath.c
>> index 23573fe3fc7d..31153f6ec582 100644
>> --- a/drivers/nvme/host/multipath.c
>> +++ b/drivers/nvme/host/multipath.c
>> @@ -266,6 +266,8 @@ inline struct nvme_ns *nvme_find_path(struct 
>> nvme_ns_head *head)
>> ????? int node = numa_node_id();
>> ????? struct nvme_ns *ns;
>> +??? if (!(head->disk->flags & GENHD_FL_UP))
>> +??????? return NULL;
>> ????? ns = srcu_dereference(head->current_path[node], &head->srcu);
>> ????? if (unlikely(!ns))
>> ????????? return __nvme_find_path(head, node);
>> @@ -281,6 +283,8 @@ static bool nvme_available_path(struct 
>> nvme_ns_head *head)
>> ? {
>> ????? struct nvme_ns *ns;
>> +??? if (!(head->disk->flags & GENHD_FL_UP))
>> +??????? return false;
> 
> nvme_available_path should have no business looking at the head gendisk,
> it should just understand if a PATH (a.k.a a controller) exists.
> 
Agreed. I was only overly cautious here; will be dropping this check.

> IMO, the fact that it does should tell that we should take a step back
> and think about this. We are trying to keep an zombie nshead around
> just for the possibility the host will reconnect (not as part of
> error recovery, but as a brand new connect). Why shouldn't we just
> remove it and restore it as a brand new nshead when the host attaches
> again?
> 
This patch has now evolved quite a bit, and in fact diverged slightly 
from the description. The original intent indeed was to keep the nshead 
around until the last reference drops, such that if a controller gets 
reattached it will be able to connect the namespaces to the correct 
(existing) ns_head.
However, as it turned out this was just a band-aid, and the real fix is 
to get the reference counts between 'struct ns' and 'struct ns_head' 
correct: if the last path to a ns_head drops, we should be removing the 
ns_head by calling del_gendisk() and removing it from the list of ns_heads.

As noted by Keith the first part is done correctly in this patch (namely 
del_gendisk() is called when the last path drops), but the second bit of 
detaching it from the list of ns_heads is _not_ done correctly.
Both should be happening at the same time to avoid any race conditions.

Will be sending an updated patch.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From hare at suse.de  Tue Jun 22 00:01:40 2021
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 22 Jun 2021 09:01:40 +0200
Subject: [PATCHv7] nvme: fix refcounting imbalance when all paths are down
Message-ID: <20210622070140.22488-1-hare@suse.de>

When the last path to a ns_head drops the current code
removes the ns_head from the subsystem list, but will only
delete the disk itself if the last reference to the ns_head
drops. This is causing an refcounting imbalance eg when
applications have a reference to the disk, as then they'll
never get notified that the disk is in fact dead.
With this patch we are calling 'del_gendisk' and remove it
from the list of ns_heads at the same time, ensuring that
the disk can be properly removed and applications get the
appropriate notifications.

Changes to v6:
- Move the list_del() into nvme_mpath_check_last_path()
- Drop the tests for GENHD_FL_UP
Changes to v5:
- Synchronize between nvme_init_ns_head() and nvme_mpath_check_last_path()
- Check for removed gendisk in nvme_ns_head_submit_bio()
Changes to v4:
- Call del_gendisk() in nvme_mpath_check_last_path() to avoid deadlock
Changes to v3:
- Simplify if() clause to detect duplicate namespaces
Changes to v2:
- Drop memcpy() statement
Changes to v1:
- Always check NSIDs after reattach

Signed-off-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/core.c      |  6 +-----
 drivers/nvme/host/multipath.c | 28 +++++++++++++++++++++++-----
 drivers/nvme/host/nvme.h      | 11 ++---------
 3 files changed, 26 insertions(+), 19 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index c7ef0b6684b5..106322dafe45 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3799,8 +3799,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid,
  out_unlink_ns:
 	mutex_lock(&ctrl->subsys->lock);
 	list_del_rcu(&ns->siblings);
-	if (list_empty(&ns->head->list))
-		list_del_init(&ns->head->entry);
 	mutex_unlock(&ctrl->subsys->lock);
 	nvme_put_ns_head(ns->head);
  out_free_queue:
@@ -3821,8 +3819,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	mutex_lock(&ns->ctrl->subsys->lock);
 	list_del_rcu(&ns->siblings);
-	if (list_empty(&ns->head->list))
-		list_del_init(&ns->head->entry);
 	mutex_unlock(&ns->ctrl->subsys->lock);
 
 	synchronize_rcu(); /* guarantee not available in head->list */
@@ -3842,7 +3838,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	list_del_init(&ns->list);
 	up_write(&ns->ctrl->namespaces_rwsem);
 
-	nvme_mpath_check_last_path(ns);
+	nvme_mpath_check_last_path(ns->head);
 	nvme_put_ns(ns);
 }
 
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 23573fe3fc7d..bf9dfaefa77d 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -771,20 +771,38 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id)
 #endif
 }
 
-void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+void nvme_mpath_check_last_path(struct nvme_ns_head *head)
 {
+	bool last_path = false;
 	if (!head->disk)
 		return;
-	if (head->disk->flags & GENHD_FL_UP) {
-		nvme_cdev_del(&head->cdev, &head->cdev_device);
-		del_gendisk(head->disk);
+
+	/* Synchronize with nvme_init_ns_head() */
+	mutex_lock(&head->subsys->lock);
+	if (list_empty(&head->list)) {
+		list_del_init(&head->entry);
+		last_path = true;
 	}
+	mutex_unlock(&head->subsys->lock);
+	if (last_path) {
+		kblockd_schedule_work(&head->requeue_work);
+		if (head->disk->flags & GENHD_FL_UP) {
+			nvme_cdev_del(&head->cdev, &head->cdev_device);
+			del_gendisk(head->disk);
+		}
+	}
+}
+
+void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+{
+	if (!head->disk)
+		return;
 	blk_set_queue_dying(head->disk->queue);
 	/* make sure all pending bios are cleaned up */
 	kblockd_schedule_work(&head->requeue_work);
 	flush_work(&head->requeue_work);
 	blk_cleanup_queue(head->disk->queue);
-	if (!test_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
+	if (!test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
 		/*
 		 * if device_add_disk wasn't called, prevent
 		 * disk release to put a bogus reference on the
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 75420ceacc10..6b1caabe861b 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -716,14 +716,7 @@ void nvme_mpath_uninit(struct nvme_ctrl *ctrl);
 void nvme_mpath_stop(struct nvme_ctrl *ctrl);
 bool nvme_mpath_clear_current_path(struct nvme_ns *ns);
 void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl);
-
-static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)
-{
-	struct nvme_ns_head *head = ns->head;
-
-	if (head->disk && list_empty(&head->list))
-		kblockd_schedule_work(&head->requeue_work);
-}
+void nvme_mpath_check_last_path(struct nvme_ns_head *head);
 
 static inline void nvme_trace_bio_complete(struct request *req)
 {
@@ -772,7 +765,7 @@ static inline bool nvme_mpath_clear_current_path(struct nvme_ns *ns)
 static inline void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl)
 {
 }
-static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)
+static inline void nvme_mpath_check_last_path(struct nvme_ns_head *head)
 {
 }
 static inline void nvme_trace_bio_complete(struct request *req)
-- 
2.26.2



From hch at infradead.org  Tue Jun 22 00:05:03 2021
From: hch at infradead.org (Christoph Hellwig)
Date: Tue, 22 Jun 2021 09:05:03 +0200
Subject: [GIT PULL] second round of nvme updates for Linux 5.14
Message-ID: <YNGLn2MsJDwwbZbv@infradead.org>

The following changes since commit e0d245e2230998e66dfda10fb8c413f29196eb1c:

  Merge branch 'md-next' of https://git.kernel.org/pub/scm/linux/kernel/git/song/md into for-5.14/drivers (2021-06-15 15:42:56 -0600)

are available in the Git repository at:

  git://git.infradead.org/nvme.git tags/nvme-5.14-2021-06-22

for you to fetch changes up to 3c3ee16532c1be92350a2a88bd19283b7bdf32e9:

  nvmet: use NVMET_MAX_NAMESPACES to set nn value (2021-06-21 08:34:10 +0200)

----------------------------------------------------------------
nvme updates for Linux 5.14:

 - move the ACPI StorageD3 code to drivers/acpi/ and add quirks for
   certain AMD CPUs (Mario Limonciello)
 - zoned device support for nvmet (Chaitanya Kulkarni)
 - fix the rules for changing the serial number in nvmet (Noam Gottlieb)
 - various small fixes and cleanups (Dan Carpenter, JK Kim,
   Chaitanya Kulkarni, Hannes Reinecke, Wesley Sheng, Geert Uytterhoeven,
   Daniel Wagner)

----------------------------------------------------------------
Chaitanya Kulkarni (25):
      nvme: factor out a nvme_validate_passthru_nsid helper
      nvme-pci: remove trailing lines for helpers
      nvme: add a helper to check ctrl sgl support
      nvme-fc: use ctrl sgl check helper
      nvme-pci: use ctrl sgl check helper
      nvme-tcp: use ctrl sgl check helper
      nvme-fabrics: remove memset in nvmf_reg_read64()
      nvme-fabrics: remove memset in nvmf_reg_write32()
      nvme-fabrics: remove memset in connect admin q
      nvme-fabrics: remove memset in connect io q
      nvmet: use req->cmd directly in bdev-ns fast path
      nvmet: use req->cmd directly in file-ns fast path
      nvmet: use u32 for nvmet_subsys max_nsid
      nvmet: use u32 type for the local variable nsid
      nvmet: use nvme status value directly
      nvmet: remove local variable
      block: export blk_next_bio()
      nvmet: add req cns error complete helper
      nvmet: add nvmet_req_bio put helper for backends
      nvmet: add Command Set Identifier support
      nvmet: add ZBD over ZNS backend support
      nvmet: remove zeroout memset call for struct
      nvme-pci: remove zeroout memset call for struct
      nvme: remove zeroout memset call for struct
      nvmet: use NVMET_MAX_NAMESPACES to set nn value

Dan Carpenter (1):
      nvme-tcp: fix error codes in nvme_tcp_setup_ctrl()

Daniel Wagner (2):
      nvme: verify MNAN value if ANA is enabled
      nvme: remove superfluous bio_set_dev in nvme_requeue_work

Geert Uytterhoeven (1):
      nvme: fix grammar in the CONFIG_NVME_MULTIPATH kconfig help text

Hannes Reinecke (1):
      nvmet-fc: do not check for invalid target port in nvmet_fc_handle_fcp_rqst()

JK Kim (1):
      nvme-pci: fix var. type for increasing cq_head

Mario Limonciello (2):
      ACPI: Check StorageD3Enable _DSD property in ACPI code
      ACPI: Add quirks for AMD Renoir/Lucienne CPUs to force the D3 hint

Noam Gottlieb (4):
      nvmet: change sn size and check validity
      nvmet: make sn stable once connection was established
      nvmet: allow mn change if subsys not discovered
      nvmet: make ver stable once connection established

Wesley Sheng (1):
      nvme.h: add missing nvme_lba_range_type endianness annotations

 block/blk-lib.c                   |   1 +
 drivers/acpi/device_pm.c          |  32 ++
 drivers/acpi/internal.h           |   9 +
 drivers/acpi/x86/utils.c          |  25 ++
 drivers/nvme/host/Kconfig         |   2 +-
 drivers/nvme/host/core.c          |  19 +-
 drivers/nvme/host/fabrics.c       |  12 +-
 drivers/nvme/host/fc.c            |   2 +-
 drivers/nvme/host/ioctl.c         |  26 +-
 drivers/nvme/host/multipath.c     |  12 +-
 drivers/nvme/host/nvme.h          |   5 +
 drivers/nvme/host/pci.c           |  60 +---
 drivers/nvme/host/tcp.c           |   4 +-
 drivers/nvme/target/Makefile      |   1 +
 drivers/nvme/target/admin-cmd.c   | 155 +++++++---
 drivers/nvme/target/configfs.c    | 102 +++++--
 drivers/nvme/target/core.c        |  98 ++++--
 drivers/nvme/target/discovery.c   |   8 +-
 drivers/nvme/target/fc.c          |  10 +-
 drivers/nvme/target/io-cmd-bdev.c |  33 +-
 drivers/nvme/target/io-cmd-file.c |   4 +-
 drivers/nvme/target/nvmet.h       |  41 ++-
 drivers/nvme/target/passthru.c    |   3 +-
 drivers/nvme/target/rdma.c        |   3 +-
 drivers/nvme/target/zns.c         | 615 ++++++++++++++++++++++++++++++++++++++
 include/linux/acpi.h              |   5 +
 include/linux/bio.h               |   2 +
 include/linux/nvme.h              |  12 +-
 28 files changed, 1073 insertions(+), 228 deletions(-)
 create mode 100644 drivers/nvme/target/zns.c


From sagi at grimberg.me  Tue Jun 22 00:42:57 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 22 Jun 2021 00:42:57 -0700
Subject: [PATCH] nvmet-rdma: process wr wait list from a non-atomic context
Message-ID: <20210622074257.1273123-1-sagi@grimberg.me>

If we fail to post a rsp capsule, we release the rsp and
if we happen to have a work waiting on the wr_wait_list, hence
we need to process them, however the context can be softirq
hence we need to schedule this on a kthread as I/O submission
may schedule.

Reported-by: Michal Kalderon <mkalderon at marvell.com>
Tested-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/target/rdma.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 891174ccd44b..9684d8972a3e 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -102,6 +102,7 @@ struct nvmet_rdma_queue {
 
 	struct work_struct	release_work;
 	struct list_head	rsp_wait_list;
+	struct work_struct	wr_wait_work;
 	struct list_head	rsp_wr_wait_list;
 	spinlock_t		rsp_wr_wait_lock;
 
@@ -517,8 +518,10 @@ static int nvmet_rdma_post_recv(struct nvmet_rdma_device *ndev,
 	return ret;
 }
 
-static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
+static void nvmet_rdma_process_wr_wait_list(struct work_struct *w)
 {
+	struct nvmet_rdma_queue *queue =
+		container_of(w, struct nvmet_rdma_queue, wr_wait_work);
 	spin_lock(&queue->rsp_wr_wait_lock);
 	while (!list_empty(&queue->rsp_wr_wait_list)) {
 		struct nvmet_rdma_rsp *rsp;
@@ -677,7 +680,7 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
 		nvmet_req_free_sgls(&rsp->req);
 
 	if (unlikely(!list_empty_careful(&queue->rsp_wr_wait_list)))
-		nvmet_rdma_process_wr_wait_list(queue);
+		schedule_work(&queue->wr_wait_work);
 
 	nvmet_rdma_put_rsp(rsp);
 }
@@ -1445,6 +1448,7 @@ nvmet_rdma_alloc_queue(struct nvmet_rdma_device *ndev,
 	 * inside a CM callback would trigger a deadlock. (great API design..)
 	 */
 	INIT_WORK(&queue->release_work, nvmet_rdma_release_queue_work);
+	INIT_WORK(&queue->wr_wait_work, nvmet_rdma_process_wr_wait_list);
 	queue->dev = ndev;
 	queue->cm_id = cm_id;
 	queue->port = port->nport;
-- 
2.27.0



From sagi at grimberg.me  Tue Jun 22 01:52:15 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Tue, 22 Jun 2021 01:52:15 -0700
Subject: [PATCHv7] nvme: fix refcounting imbalance when all paths are down
In-Reply-To: <20210622070140.22488-1-hare@suse.de>
References: <20210622070140.22488-1-hare@suse.de>
Message-ID: <6679737f-1af8-bc9e-5af1-ce3fac5b4d01@grimberg.me>



On 6/22/21 12:01 AM, Hannes Reinecke wrote:
> When the last path to a ns_head drops the current code
> removes the ns_head from the subsystem list, but will only
> delete the disk itself if the last reference to the ns_head
> drops. This is causing an refcounting imbalance eg when
> applications have a reference to the disk, as then they'll
> never get notified that the disk is in fact dead.
> With this patch we are calling 'del_gendisk' and remove it
> from the list of ns_heads at the same time, ensuring that
> the disk can be properly removed and applications get the
> appropriate notifications.
> 
> Changes to v6:
> - Move the list_del() into nvme_mpath_check_last_path()
> - Drop the tests for GENHD_FL_UP
> Changes to v5:
> - Synchronize between nvme_init_ns_head() and nvme_mpath_check_last_path()
> - Check for removed gendisk in nvme_ns_head_submit_bio()
> Changes to v4:
> - Call del_gendisk() in nvme_mpath_check_last_path() to avoid deadlock
> Changes to v3:
> - Simplify if() clause to detect duplicate namespaces
> Changes to v2:
> - Drop memcpy() statement
> Changes to v1:
> - Always check NSIDs after reattach
> 
> Signed-off-by: Hannes Reinecke <hare at suse.de>
> ---
>   drivers/nvme/host/core.c      |  6 +-----
>   drivers/nvme/host/multipath.c | 28 +++++++++++++++++++++++-----
>   drivers/nvme/host/nvme.h      | 11 ++---------
>   3 files changed, 26 insertions(+), 19 deletions(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index c7ef0b6684b5..106322dafe45 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -3799,8 +3799,6 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid,
>    out_unlink_ns:
>   	mutex_lock(&ctrl->subsys->lock);
>   	list_del_rcu(&ns->siblings);
> -	if (list_empty(&ns->head->list))
> -		list_del_init(&ns->head->entry);
>   	mutex_unlock(&ctrl->subsys->lock);
>   	nvme_put_ns_head(ns->head);
>    out_free_queue:
> @@ -3821,8 +3819,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>   
>   	mutex_lock(&ns->ctrl->subsys->lock);
>   	list_del_rcu(&ns->siblings);
> -	if (list_empty(&ns->head->list))
> -		list_del_init(&ns->head->entry);
>   	mutex_unlock(&ns->ctrl->subsys->lock);
>   
>   	synchronize_rcu(); /* guarantee not available in head->list */
> @@ -3842,7 +3838,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>   	list_del_init(&ns->list);
>   	up_write(&ns->ctrl->namespaces_rwsem);
>   
> -	nvme_mpath_check_last_path(ns);
> +	nvme_mpath_check_last_path(ns->head);
>   	nvme_put_ns(ns);
>   }
>   
> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
> index 23573fe3fc7d..bf9dfaefa77d 100644
> --- a/drivers/nvme/host/multipath.c
> +++ b/drivers/nvme/host/multipath.c
> @@ -771,20 +771,38 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id)
>   #endif
>   }
>   
> -void nvme_mpath_remove_disk(struct nvme_ns_head *head)
> +void nvme_mpath_check_last_path(struct nvme_ns_head *head)
>   {
> +	bool last_path = false;
>   	if (!head->disk)
>   		return;
> -	if (head->disk->flags & GENHD_FL_UP) {
> -		nvme_cdev_del(&head->cdev, &head->cdev_device);
> -		del_gendisk(head->disk); > +
> +	/* Synchronize with nvme_init_ns_head() */
> +	mutex_lock(&head->subsys->lock);
> +	if (list_empty(&head->list)) {
> +		list_del_init(&head->entry);
> +		last_path = true;
>   	}

What happens now if CONFIG_NVME_MULTIPATH is not set? I'm not sure this
should be done here...


From hare at suse.de  Tue Jun 22 02:03:28 2021
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 22 Jun 2021 11:03:28 +0200
Subject: [PATCHv7] nvme: fix refcounting imbalance when all paths are down
In-Reply-To: <6679737f-1af8-bc9e-5af1-ce3fac5b4d01@grimberg.me>
References: <20210622070140.22488-1-hare@suse.de>
 <6679737f-1af8-bc9e-5af1-ce3fac5b4d01@grimberg.me>
Message-ID: <a5baf9d1-47af-7a9a-e08f-f75c12429413@suse.de>

On 6/22/21 10:52 AM, Sagi Grimberg wrote:
> 
> 
> On 6/22/21 12:01 AM, Hannes Reinecke wrote:
>> When the last path to a ns_head drops the current code
>> removes the ns_head from the subsystem list, but will only
>> delete the disk itself if the last reference to the ns_head
>> drops. This is causing an refcounting imbalance eg when
>> applications have a reference to the disk, as then they'll
>> never get notified that the disk is in fact dead.
>> With this patch we are calling 'del_gendisk' and remove it
>> from the list of ns_heads at the same time, ensuring that
>> the disk can be properly removed and applications get the
>> appropriate notifications.
>>
>> Changes to v6:
>> - Move the list_del() into nvme_mpath_check_last_path()
>> - Drop the tests for GENHD_FL_UP
>> Changes to v5:
>> - Synchronize between nvme_init_ns_head() and
>> nvme_mpath_check_last_path()
>> - Check for removed gendisk in nvme_ns_head_submit_bio()
>> Changes to v4:
>> - Call del_gendisk() in nvme_mpath_check_last_path() to avoid deadlock
>> Changes to v3:
>> - Simplify if() clause to detect duplicate namespaces
>> Changes to v2:
>> - Drop memcpy() statement
>> Changes to v1:
>> - Always check NSIDs after reattach
>>
>> Signed-off-by: Hannes Reinecke <hare at suse.de>
>> ---
>> ? drivers/nvme/host/core.c????? |? 6 +-----
>> ? drivers/nvme/host/multipath.c | 28 +++++++++++++++++++++++-----
>> ? drivers/nvme/host/nvme.h????? | 11 ++---------
>> ? 3 files changed, 26 insertions(+), 19 deletions(-)
>>
>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>> index c7ef0b6684b5..106322dafe45 100644
>> --- a/drivers/nvme/host/core.c
>> +++ b/drivers/nvme/host/core.c
>> @@ -3799,8 +3799,6 @@ static void nvme_alloc_ns(struct nvme_ctrl
>> *ctrl, unsigned nsid,
>> ?? out_unlink_ns:
>> ????? mutex_lock(&ctrl->subsys->lock);
>> ????? list_del_rcu(&ns->siblings);
>> -??? if (list_empty(&ns->head->list))
>> -??????? list_del_init(&ns->head->entry);
>> ????? mutex_unlock(&ctrl->subsys->lock);
>> ????? nvme_put_ns_head(ns->head);
>> ?? out_free_queue:
>> @@ -3821,8 +3819,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>> ? ????? mutex_lock(&ns->ctrl->subsys->lock);
>> ????? list_del_rcu(&ns->siblings);
>> -??? if (list_empty(&ns->head->list))
>> -??????? list_del_init(&ns->head->entry);
>> ????? mutex_unlock(&ns->ctrl->subsys->lock);
>> ? ????? synchronize_rcu(); /* guarantee not available in head->list */
>> @@ -3842,7 +3838,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
>> ????? list_del_init(&ns->list);
>> ????? up_write(&ns->ctrl->namespaces_rwsem);
>> ? -??? nvme_mpath_check_last_path(ns);
>> +??? nvme_mpath_check_last_path(ns->head);
>> ????? nvme_put_ns(ns);
>> ? }
>> ? diff --git a/drivers/nvme/host/multipath.c
>> b/drivers/nvme/host/multipath.c
>> index 23573fe3fc7d..bf9dfaefa77d 100644
>> --- a/drivers/nvme/host/multipath.c
>> +++ b/drivers/nvme/host/multipath.c
>> @@ -771,20 +771,38 @@ void nvme_mpath_add_disk(struct nvme_ns *ns,
>> struct nvme_id_ns *id)
>> ? #endif
>> ? }
>> ? -void nvme_mpath_remove_disk(struct nvme_ns_head *head)
>> +void nvme_mpath_check_last_path(struct nvme_ns_head *head)
>> ? {
>> +??? bool last_path = false;
>> ????? if (!head->disk)
>> ????????? return;
>> -??? if (head->disk->flags & GENHD_FL_UP) {
>> -??????? nvme_cdev_del(&head->cdev, &head->cdev_device);
>> -??????? del_gendisk(head->disk); > +
>> +??? /* Synchronize with nvme_init_ns_head() */
>> +??? mutex_lock(&head->subsys->lock);
>> +??? if (list_empty(&head->list)) {
>> +??????? list_del_init(&head->entry);
>> +??????? last_path = true;
>> ????? }
> 
> What happens now if CONFIG_NVME_MULTIPATH is not set? I'm not sure this
> should be done here...

I don't mind _where_ this code ends up.
As long as we can agree that we should do both, list_del_init() and
del_gendisk() at the same time when we find that the last path is dead.
Do we?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare at suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), GF: Felix Imend?rffer


From kbusch at kernel.org  Tue Jun 22 07:57:18 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 22 Jun 2021 23:57:18 +0900
Subject: [PATCHv4 0/4] block and nvme passthrough error handling
In-Reply-To: <20210610214437.641245-1-kbusch@kernel.org>
References: <20210610214437.641245-1-kbusch@kernel.org>
Message-ID: <20210622145718.GA11584@redsun51.ssa.fujisawa.hgst.com>

Jens,

Do you have any thoughts on this series? I think it's good to go and
have received reviews, and just want to check if you are okay to take
this through block.

Thanks,
Keith


From kbusch at kernel.org  Tue Jun 22 08:06:36 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 22 Jun 2021 08:06:36 -0700
Subject: [PATCH 1/2] nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <20210622002710.26591-2-cachen@purestorage.com>
References: <20210622002710.26591-1-cachen@purestorage.com>
 <20210622002710.26591-2-cachen@purestorage.com>
Message-ID: <20210622150636.GA2359631@dhcp-10-100-145-180.wdc.com>

On Mon, Jun 21, 2021 at 05:27:09PM -0700, Casey Chen wrote:
> +	/*
> +	 * Free IRQ resources as soon as NVMEQ_ENABLED bit transitions
> +	 * from set to unset. If there is a window to it is truely freed,
> +	 * pci_free_irq_vectors() jumping into this window will crash.
> +	 * And take lock to avoid racing with pci_free_irq_vectors() in
> +	 * nvme_dev_disable() path.
> +	 */
> +	mutex_lock(&dev->shutdown_lock)

Sorry, I wasn't clear in previous review. All of the shutdown_locks
taken after initialization need to by mutex_trylock()'s. If you have to
wait to get the lock, the device is not going to be suitable for
continuing initialization after the lock is available.

And looking at this again, if trylock is successful, I think you need to
verify controller state is suitable for continuing the initialization.


From kbusch at kernel.org  Tue Jun 22 12:16:02 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 22 Jun 2021 12:16:02 -0700
Subject: [PATCH 2/2] nvme-pci: Fix UAF introduced by
 nvme_dev_remove_admin()
In-Reply-To: <20210622002710.26591-3-cachen@purestorage.com>
References: <20210622002710.26591-1-cachen@purestorage.com>
 <20210622002710.26591-3-cachen@purestorage.com>
Message-ID: <20210622191602.GA2360166@dhcp-10-100-145-180.wdc.com>

On Mon, Jun 21, 2021 at 05:27:10PM -0700, Casey Chen wrote:
> nvme_dev_remove_admin() could free admin_q and admin_tagset while
> they are being accessed by nvme_dev_disable(), which could come from
> nvme_remove_dead_ctrl() by nvme_reset_work() during cleanup.
> 
> Commit cb4bfda62afa ("nvme-pci: fix hot removal during error handling") was
> to avoid requests being stuck on a removed controller by killing admin queue.
> But the later fix c8e9e9b7646e ("nvme-pci: unquiesce admin queue on shutdown"),
> together with nvme_dev_disable(dev, true) right before nvme_dev_remove_admin()
> could help dispatch requests and fail them early, so we don't need
> nvme_dev_remove_admin() any more.
> 
> Fixes: cb4bfda62afa ("nvme-pci: fix hot removal during error handling")

Thanks, this change looks good.

Reviewed-by: Keith Busch <kbusch at kernel.org>

> upstream_status: upstream
> ---
>  drivers/nvme/host/pci.c | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 5abc4c3be454..49ef2027bcb0 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -3023,7 +3023,6 @@ static void nvme_remove(struct pci_dev *pdev)
>  	if (!pci_device_is_present(pdev)) {
>  		nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DEAD);
>  		nvme_dev_disable(dev, true);
> -		nvme_dev_remove_admin(dev);
>  	}
>  
>  	flush_work(&dev->ctrl.reset_work);
> -- 
> 2.17.1


From cachen at purestorage.com  Tue Jun 22 17:59:30 2021
From: cachen at purestorage.com (Casey Chen)
Date: Tue, 22 Jun 2021 17:59:30 -0700
Subject: [PATCH 1/2] nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <20210622150636.GA2359631@dhcp-10-100-145-180.wdc.com>
References: <20210622002710.26591-1-cachen@purestorage.com>
 <20210622002710.26591-2-cachen@purestorage.com>
 <20210622150636.GA2359631@dhcp-10-100-145-180.wdc.com>
Message-ID: <33919248-6c62-0974-ce88-3c0d0885e837@purestorage.com>

On 6/22/21 8:06 AM, Keith Busch wrote:
> On Mon, Jun 21, 2021 at 05:27:09PM -0700, Casey Chen wrote:
>> +	/*
>> +	 * Free IRQ resources as soon as NVMEQ_ENABLED bit transitions
>> +	 * from set to unset. If there is a window to it is truely freed,
>> +	 * pci_free_irq_vectors() jumping into this window will crash.
>> +	 * And take lock to avoid racing with pci_free_irq_vectors() in
>> +	 * nvme_dev_disable() path.
>> +	 */
>> +	mutex_lock(&dev->shutdown_lock)
> Sorry, I wasn't clear in previous review. All of the shutdown_locks
> taken after initialization need to by mutex_trylock()'s. If you have to
> wait to get the lock, the device is not going to be suitable for
> continuing initialization after the lock is available.
>
> And looking at this again, if trylock is successful, I think you need to
> verify controller state is suitable for continuing the initialization.
I assume you mean all the mutex_lock() except the one in 
nvme_dev_disable(), right ? I agree with replacing all mutex_lock() with 
mutex_trylock() except the one in nvme_dev_disable(). But I doubt about 
checking controller state after lock being acquired. Two reasons:

1. What state can use to check ? Suppose there is a state set on 
surprise removal, what if the state check happen before it is set while 
surprise removal is on its way ?

2. Even we don't check controller state after acquiring lock, the 
initialization still fail fairly soon if surprise removal happens. We 
don't sacrifice any performance.

Casey



From kbusch at kernel.org  Tue Jun 22 18:53:31 2021
From: kbusch at kernel.org (Keith Busch)
Date: Tue, 22 Jun 2021 18:53:31 -0700
Subject: [PATCH 1/2] nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <33919248-6c62-0974-ce88-3c0d0885e837@purestorage.com>
References: <20210622002710.26591-1-cachen@purestorage.com>
 <20210622002710.26591-2-cachen@purestorage.com>
 <20210622150636.GA2359631@dhcp-10-100-145-180.wdc.com>
 <33919248-6c62-0974-ce88-3c0d0885e837@purestorage.com>
Message-ID: <20210623015331.GB2360166@dhcp-10-100-145-180.wdc.com>

On Tue, Jun 22, 2021 at 05:59:30PM -0700, Casey Chen wrote:
> On 6/22/21 8:06 AM, Keith Busch wrote:
> > On Mon, Jun 21, 2021 at 05:27:09PM -0700, Casey Chen wrote:
> > > +	/*
> > > +	 * Free IRQ resources as soon as NVMEQ_ENABLED bit transitions
> > > +	 * from set to unset. If there is a window to it is truely freed,
> > > +	 * pci_free_irq_vectors() jumping into this window will crash.
> > > +	 * And take lock to avoid racing with pci_free_irq_vectors() in
> > > +	 * nvme_dev_disable() path.
> > > +	 */
> > > +	mutex_lock(&dev->shutdown_lock)
> > Sorry, I wasn't clear in previous review. All of the shutdown_locks
> > taken after initialization need to by mutex_trylock()'s. If you have to
> > wait to get the lock, the device is not going to be suitable for
> > continuing initialization after the lock is available.
> > 
> > And looking at this again, if trylock is successful, I think you need to
> > verify controller state is suitable for continuing the initialization.
> I assume you mean all the mutex_lock() except the one in nvme_dev_disable(),
> right ? I agree with replacing all mutex_lock() with mutex_trylock() except
> the one in nvme_dev_disable(). 

I only mean the locks you are adding. The existing usages are correct
as-is.

> But I doubt about checking controller state
> after lock being acquired. Two reasons:
>
> 1. What state can use to check ?

If the state is not NVME_CTRL_CONNECTING after successfully taking the
shutdown_lock, then the controller is not in a state that can proceed
with IO queue setup.

> Suppose there is a state set on surprise
> removal, what if the state check happen before it is set while surprise
> removal is on its way ?
> 
> 2. Even we don't check controller state after acquiring lock, the
> initialization still fail fairly soon if surprise removal happens. We don't
> sacrifice any performance.

Nothing to do with performance; we're not in io path. The suggestion is
about avoiding access to a non-existent device. The timing of a surprise
removal can always defeat such a check, so my suggestion is not critical
since we expect that to be handled anyway, but in my experience,
platforms work better if the OS prevents such access when it knows it
will fail. It's not full-proof.

But as I said, everything should work even without the check, so feel
free to omit it.


From cachen at purestorage.com  Tue Jun 22 21:00:08 2021
From: cachen at purestorage.com (Casey Chen)
Date: Tue, 22 Jun 2021 21:00:08 -0700
Subject: [PATCH 1/2] nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <20210623015331.GB2360166@dhcp-10-100-145-180.wdc.com>
References: <20210622002710.26591-1-cachen@purestorage.com>
 <20210622002710.26591-2-cachen@purestorage.com>
 <20210622150636.GA2359631@dhcp-10-100-145-180.wdc.com>
 <33919248-6c62-0974-ce88-3c0d0885e837@purestorage.com>
 <20210623015331.GB2360166@dhcp-10-100-145-180.wdc.com>
Message-ID: <0faf018d-570b-4c26-a093-3d375bd70c26@purestorage.com>


On 6/22/21 6:53 PM, Keith Busch wrote:
> On Tue, Jun 22, 2021 at 05:59:30PM -0700, Casey Chen wrote:
>> On 6/22/21 8:06 AM, Keith Busch wrote:
>>> On Mon, Jun 21, 2021 at 05:27:09PM -0700, Casey Chen wrote:
>>>> +	/*
>>>> +	 * Free IRQ resources as soon as NVMEQ_ENABLED bit transitions
>>>> +	 * from set to unset. If there is a window to it is truely freed,
>>>> +	 * pci_free_irq_vectors() jumping into this window will crash.
>>>> +	 * And take lock to avoid racing with pci_free_irq_vectors() in
>>>> +	 * nvme_dev_disable() path.
>>>> +	 */
>>>> +	mutex_lock(&dev->shutdown_lock)
>>> Sorry, I wasn't clear in previous review. All of the shutdown_locks
>>> taken after initialization need to by mutex_trylock()'s. If you have to
>>> wait to get the lock, the device is not going to be suitable for
>>> continuing initialization after the lock is available.
>>>
>>> And looking at this again, if trylock is successful, I think you need to
>>> verify controller state is suitable for continuing the initialization.
>> I assume you mean all the mutex_lock() except the one in nvme_dev_disable(),
>> right ? I agree with replacing all mutex_lock() with mutex_trylock() except
>> the one in nvme_dev_disable().
> I only mean the locks you are adding. The existing usages are correct
> as-is.
>
>> But I doubt about checking controller state
>> after lock being acquired. Two reasons:
>>
>> 1. What state can use to check ?
> If the state is not NVME_CTRL_CONNECTING after successfully taking the
> shutdown_lock, then the controller is not in a state that can proceed
> with IO queue setup.
>
>> Suppose there is a state set on surprise
>> removal, what if the state check happen before it is set while surprise
>> removal is on its way ?
>>
>> 2. Even we don't check controller state after acquiring lock, the
>> initialization still fail fairly soon if surprise removal happens. We don't
>> sacrifice any performance.
> Nothing to do with performance; we're not in io path. The suggestion is
> about avoiding access to a non-existent device. The timing of a surprise
> removal can always defeat such a check, so my suggestion is not critical
> since we expect that to be handled anyway, but in my experience,
> platforms work better if the OS prevents such access when it knows it
> will fail. It's not full-proof.
>
> But as I said, everything should work even without the check, so feel
> free to omit it.

Thanks for explanation. I prefer not having the check even I had came up 
with a solution as below.

This looks cumbersome. If you can make it better, I might adopt it. 
Otherwise, I will send new patches without the check. :)

+/*
+ * Try getting shutdown_lock while setting up IO queues.
+ * Caller remember to unlock.
+ */
+static int nvme_setup_io_queues_trylock(struct nvme_dev *dev)
+{
+?????? /*
+?????? * Lock is being held by nvme_dev_disable(), fail early.
+?????? */
+?????? if (!mutex_trylock(&dev->shutdown_lock))
+?????????????? return -ENODEV;
+
+?????? /*
+?????? * Controller is in wrong state, fail early.
+??????? */
+?????? if (dev->ctrl.state == NVME_CTRL_CONNECTING)
+?????????????? return -ENODEV;
+
+?????? return 0;
+}
+

Sample call:

-?????? mutex_lock(&dev->shutdown_lock);
+?????? if ((result = nvme_setup_io_queues_trylock(dev)))
+?????????????? return result;



From cachen at purestorage.com  Tue Jun 22 21:14:21 2021
From: cachen at purestorage.com (Casey Chen)
Date: Tue, 22 Jun 2021 21:14:21 -0700
Subject: [PATCH 1/2] nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <0faf018d-570b-4c26-a093-3d375bd70c26@purestorage.com>
References: <20210622002710.26591-1-cachen@purestorage.com>
 <20210622002710.26591-2-cachen@purestorage.com>
 <20210622150636.GA2359631@dhcp-10-100-145-180.wdc.com>
 <33919248-6c62-0974-ce88-3c0d0885e837@purestorage.com>
 <20210623015331.GB2360166@dhcp-10-100-145-180.wdc.com>
 <0faf018d-570b-4c26-a093-3d375bd70c26@purestorage.com>
Message-ID: <9681def7-a751-f4d8-8a92-c5bee027a97a@purestorage.com>


On 6/22/21 9:00 PM, Casey Chen wrote:
>
> On 6/22/21 6:53 PM, Keith Busch wrote:
>> On Tue, Jun 22, 2021 at 05:59:30PM -0700, Casey Chen wrote:
>>> On 6/22/21 8:06 AM, Keith Busch wrote:
>>>> On Mon, Jun 21, 2021 at 05:27:09PM -0700, Casey Chen wrote:
>>>>> +??? /*
>>>>> +???? * Free IRQ resources as soon as NVMEQ_ENABLED bit transitions
>>>>> +???? * from set to unset. If there is a window to it is truely 
>>>>> freed,
>>>>> +???? * pci_free_irq_vectors() jumping into this window will crash.
>>>>> +???? * And take lock to avoid racing with pci_free_irq_vectors() in
>>>>> +???? * nvme_dev_disable() path.
>>>>> +???? */
>>>>> +??? mutex_lock(&dev->shutdown_lock)
>>>> Sorry, I wasn't clear in previous review. All of the shutdown_locks
>>>> taken after initialization need to by mutex_trylock()'s. If you 
>>>> have to
>>>> wait to get the lock, the device is not going to be suitable for
>>>> continuing initialization after the lock is available.
>>>>
>>>> And looking at this again, if trylock is successful, I think you 
>>>> need to
>>>> verify controller state is suitable for continuing the initialization.
>>> I assume you mean all the mutex_lock() except the one in 
>>> nvme_dev_disable(),
>>> right ? I agree with replacing all mutex_lock() with mutex_trylock() 
>>> except
>>> the one in nvme_dev_disable().
>> I only mean the locks you are adding. The existing usages are correct
>> as-is.
>>
>>> But I doubt about checking controller state
>>> after lock being acquired. Two reasons:
>>>
>>> 1. What state can use to check ?
>> If the state is not NVME_CTRL_CONNECTING after successfully taking the
>> shutdown_lock, then the controller is not in a state that can proceed
>> with IO queue setup.
>>
>>> Suppose there is a state set on surprise
>>> removal, what if the state check happen before it is set while surprise
>>> removal is on its way ?
>>>
>>> 2. Even we don't check controller state after acquiring lock, the
>>> initialization still fail fairly soon if surprise removal happens. 
>>> We don't
>>> sacrifice any performance.
>> Nothing to do with performance; we're not in io path. The suggestion is
>> about avoiding access to a non-existent device. The timing of a surprise
>> removal can always defeat such a check, so my suggestion is not critical
>> since we expect that to be handled anyway, but in my experience,
>> platforms work better if the OS prevents such access when it knows it
>> will fail. It's not full-proof.
>>
>> But as I said, everything should work even without the check, so feel
>> free to omit it.
>
> Thanks for explanation. I prefer not having the check even I had came 
> up with a solution as below.
>
> This looks cumbersome. If you can make it better, I might adopt it. 
> Otherwise, I will send new patches without the check. :)
>
> +/*
> + * Try getting shutdown_lock while setting up IO queues.
> + * Caller remember to unlock.
> + */
> +static int nvme_setup_io_queues_trylock(struct nvme_dev *dev)
> +{
> +?????? /*
> +?????? * Lock is being held by nvme_dev_disable(), fail early.
> +?????? */
> +?????? if (!mutex_trylock(&dev->shutdown_lock))
> +?????????????? return -ENODEV;
> +
> +?????? /*
> +?????? * Controller is in wrong state, fail early.
> +??????? */
> +?????? if (dev->ctrl.state == NVME_CTRL_CONNECTING)
> +?????????????? return -ENODEV;
> +
> +?????? return 0;
> +}
> +
>
> Sample call:
>
> -?????? mutex_lock(&dev->shutdown_lock);
> +?????? if ((result = nvme_setup_io_queues_trylock(dev)))
> +?????????????? return result;
>

Sorry I sent in a rush, should be:

+/*
+ * Try getting shutdown_lock while setting up IO queues.
+ * Caller remember to unlock if success.
+ */
+static int nvme_setup_io_queues_trylock(struct nvme_dev *dev)
+{
+?????? /*
+?????? * Lock is being held by nvme_dev_disable(), fail early.
+?????? */
+?????? if (!mutex_trylock(&dev->shutdown_lock))
+?????????????? return -ENODEV;
+
+?????? /*
+?????? * Controller is in wrong state, fail early.
+??????? */
+?????? if (dev->ctrl.state != NVME_CTRL_CONNECTING) {
+?????????????? mutex_unlock(&dev->shutdown_lock);
+?????????????? return -ENODEV;
+?????? }
+
+?????? return 0;
+}
+



From yi.zhang at redhat.com  Wed Jun 23 03:01:00 2021
From: yi.zhang at redhat.com (Yi Zhang)
Date: Wed, 23 Jun 2021 18:01:00 +0800
Subject: [bug report] NVMe/IB: reset_controller need more than 1min
In-Reply-To: <CAHj4cs_3eLZd=vxRRrnBU2W4H38mqttcy0ZdSw6uw4KvbJWgeQ@mail.gmail.com>
References: <CAHj4cs8cT23z+h2i+g6o3OQqEhWnHS88JO4jNoQo0Nww-sdkYg@mail.gmail.com>
 <3c86dc88-97d9-5a71-20e1-a90279f47db5@grimberg.me>
 <CAHj4cs_3eLZd=vxRRrnBU2W4H38mqttcy0ZdSw6uw4KvbJWgeQ@mail.gmail.com>
Message-ID: <CAHj4cs_VZ7C7ciKy-q51a+Gc=uce0GDKRHNmUdoGOd7KSvURpA@mail.gmail.com>

Hello

Gentle ping here, this issue still exists on latest 5.13-rc7

# time nvme reset /dev/nvme0

real 0m12.636s
user 0m0.002s
sys 0m0.005s
# time nvme reset /dev/nvme0

real 0m12.641s
user 0m0.000s
sys 0m0.007s
# time nvme reset /dev/nvme0

real 1m16.133s
user 0m0.000s
sys 0m0.007s

On Sat, May 22, 2021 at 12:27 PM Yi Zhang <yi.zhang at redhat.com> wrote:
>
> On Sat, May 22, 2021 at 2:00 AM Sagi Grimberg <sagi at grimberg.me> wrote:
> >
> >
> > > Hi
> > > I found this issue on 5.13-rc2 with NVMe/IB environment, could anyone
> > > help check it?
> > > Thanks.
> > >
> > > $ time echo 1 >/sys/block/nvme0n1/device/reset_controller
> > > real 0m10.678s
> > > user 0m0.000s
> > > sys 0m0.000s
> > > $ time echo 1 >/sys/block/nvme0n1/device/reset_controller
> > > real 1m11.530s
> > > user 0m0.000s
> > > sys 0m0.000s
> > >
> > > target:
> > > $ dmesg | grep nvme
> > > [  276.891454] nvmet: creating controller 1 for subsystem testnqn for
> > > NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> > > [  287.374412] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
> > > [  287.399317] nvmet: ctrl 1 fatal error occurred!
> > > [  348.412672] nvmet: creating controller 1 for subsystem testnqn for
> > > NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> > >
> > > client:
> > > $ dmesg | grep nvme
> > > [  281.704475] nvme nvme0: creating 40 I/O queues.
> > > [  285.557759] nvme nvme0: mapped 40/0/0 default/read/poll queues.
> > > [  353.187809] nvme nvme0: I/O 8 QID 0 timeout
> > > [  353.193100] nvme nvme0: Property Set error: 881, offset 0x14
> > > [  353.226082] nvme nvme0: creating 40 I/O queues.
> > > [  357.088266] nvme nvme0: mapped 40/0/0 default/read/poll queues.
> >
> > It appears that there is an admin timeout that is either triggered
> > by the reset or unrelated.
> >
> > Can you run nvme reset /dev/nvme0 instead so we can see the "resetting
> > controller" print?
> >
> Yes, here is the log:
> ------------------------0
> + nvme reset /dev/nvme0
> real 0m10.737s
> user 0m0.004s
> sys 0m0.004s
> ------------------------1
> + nvme reset /dev/nvme0
> real 1m11.335s
> user 0m0.002s
> sys 0m0.005s
>
> target:
> [  934.306016] nvmet: creating controller 1 for subsystem testnqn for
> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> [  944.875021] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
> [  944.900051] nvmet: ctrl 1 fatal error occurred!
> [ 1005.628340] nvmet: creating controller 1 for subsystem testnqn for
> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
>
> client:
> [  857.264029] nvme nvme0: resetting controller
> [  864.115369] nvme nvme0: creating 40 I/O queues.
> [  867.996746] nvme nvme0: mapped 40/0/0 default/read/poll queues.
> [  868.001673] nvme nvme0: resetting controller
> [  935.396789] nvme nvme0: I/O 9 QID 0 timeout
> [  935.402036] nvme nvme0: Property Set error: 881, offset 0x14
> [  935.438080] nvme nvme0: creating 40 I/O queues.
> [  939.332125] nvme nvme0: mapped 40/0/0 default/read/poll queues.
>
>
> --
> Best Regards,
>   Yi Zhang



From hare at suse.de  Wed Jun 23 07:32:48 2021
From: hare at suse.de (Hannes Reinecke)
Date: Wed, 23 Jun 2021 16:32:48 +0200
Subject: [PATCH 0/2] nvme-fabrics: short-circuit connect retries
Message-ID: <20210623143250.82445-1-hare@suse.de>

Hi all,

commit f25f8ef70ce2 ("nvme-fc: short-circuit reconnect retries")
allowed the fc transport to honour the DNR bit during reconnect
retries, allowing to speed up error recovery.
These patches implement the same logic for RDMA and TCP.

As usual, comments and reviews are welcome.

Hannes Reinecke (2):
  nvme-tcp: short-circuit reconnect retries
  nvme-rdma: short-circuit reconnect retries

 drivers/nvme/host/rdma.c | 23 ++++++++++++++++-------
 drivers/nvme/host/tcp.c  | 22 ++++++++++++++--------
 2 files changed, 30 insertions(+), 15 deletions(-)

-- 
2.29.2



From hare at suse.de  Wed Jun 23 07:32:49 2021
From: hare at suse.de (Hannes Reinecke)
Date: Wed, 23 Jun 2021 16:32:49 +0200
Subject: [PATCH 1/2] nvme-tcp: short-circuit reconnect retries
In-Reply-To: <20210623143250.82445-1-hare@suse.de>
References: <20210623143250.82445-1-hare@suse.de>
Message-ID: <20210623143250.82445-2-hare@suse.de>

Returning an nvme status from nvme_tcp_setup_ctrl() indicates
that the association was established and we have received a status
from the controller; consequently we should honour the DNR bit.
If not any future reconnect attempts will just return the same error, so
we can short-circuit the reconnect attempts and fail the connection
directly.

Signed-off-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/tcp.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 34f4b3402f7c..ec5f511203b8 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1943,8 +1943,10 @@ static void nvme_tcp_teardown_io_queues(struct nvme_ctrl *ctrl,
 	nvme_tcp_destroy_io_queues(ctrl, remove);
 }
 
-static void nvme_tcp_reconnect_or_remove(struct nvme_ctrl *ctrl)
+static void nvme_tcp_reconnect_or_remove(struct nvme_ctrl *ctrl, int status)
 {
+	bool recon = status > 0 && (status & NVME_SC_DNR) ? false : true;
+
 	/* If we are resetting/deleting then do nothing */
 	if (ctrl->state != NVME_CTRL_CONNECTING) {
 		WARN_ON_ONCE(ctrl->state == NVME_CTRL_NEW ||
@@ -1952,13 +1954,13 @@ static void nvme_tcp_reconnect_or_remove(struct nvme_ctrl *ctrl)
 		return;
 	}
 
-	if (nvmf_should_reconnect(ctrl)) {
+	if (recon && nvmf_should_reconnect(ctrl)) {
 		dev_info(ctrl->device, "Reconnecting in %d seconds...\n",
 			ctrl->opts->reconnect_delay);
 		queue_delayed_work(nvme_wq, &to_tcp_ctrl(ctrl)->connect_work,
 				ctrl->opts->reconnect_delay * HZ);
 	} else {
-		dev_info(ctrl->device, "Removing controller...\n");
+		dev_info(ctrl->device, "Removing controller (%d)...\n", status);
 		nvme_delete_ctrl(ctrl);
 	}
 }
@@ -2038,10 +2040,12 @@ static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work)
 	struct nvme_tcp_ctrl *tcp_ctrl = container_of(to_delayed_work(work),
 			struct nvme_tcp_ctrl, connect_work);
 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
+	int ret;
 
 	++ctrl->nr_reconnects;
 
-	if (nvme_tcp_setup_ctrl(ctrl, false))
+	ret = nvme_tcp_setup_ctrl(ctrl, false);
+	if (ret)
 		goto requeue;
 
 	dev_info(ctrl->device, "Successfully reconnected (%d attempt)\n",
@@ -2054,7 +2058,7 @@ static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work)
 requeue:
 	dev_info(ctrl->device, "Failed reconnect attempt %d\n",
 			ctrl->nr_reconnects);
-	nvme_tcp_reconnect_or_remove(ctrl);
+	nvme_tcp_reconnect_or_remove(ctrl, ret);
 }
 
 static void nvme_tcp_error_recovery_work(struct work_struct *work)
@@ -2077,7 +2081,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 		return;
 	}
 
-	nvme_tcp_reconnect_or_remove(ctrl);
+	nvme_tcp_reconnect_or_remove(ctrl, -ENOTCONN);
 }
 
 static void nvme_tcp_teardown_ctrl(struct nvme_ctrl *ctrl, bool shutdown)
@@ -2103,6 +2107,7 @@ static void nvme_reset_ctrl_work(struct work_struct *work)
 {
 	struct nvme_ctrl *ctrl =
 		container_of(work, struct nvme_ctrl, reset_work);
+	int ret;
 
 	nvme_stop_ctrl(ctrl);
 	nvme_tcp_teardown_ctrl(ctrl, false);
@@ -2114,14 +2119,15 @@ static void nvme_reset_ctrl_work(struct work_struct *work)
 		return;
 	}
 
-	if (nvme_tcp_setup_ctrl(ctrl, false))
+	ret = nvme_tcp_setup_ctrl(ctrl, false);
+	if (ret)
 		goto out_fail;
 
 	return;
 
 out_fail:
 	++ctrl->nr_reconnects;
-	nvme_tcp_reconnect_or_remove(ctrl);
+	nvme_tcp_reconnect_or_remove(ctrl, ret);
 }
 
 static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl)
-- 
2.29.2



From hare at suse.de  Wed Jun 23 07:32:50 2021
From: hare at suse.de (Hannes Reinecke)
Date: Wed, 23 Jun 2021 16:32:50 +0200
Subject: [PATCH 2/2] nvme-rdma: short-circuit reconnect retries
In-Reply-To: <20210623143250.82445-1-hare@suse.de>
References: <20210623143250.82445-1-hare@suse.de>
Message-ID: <20210623143250.82445-3-hare@suse.de>

Returning an nvme status from nvme_rdma_setup_ctrl() indicates
that the association was established and we have received a status
from the controller; consequently we should honour the DNR bit.
If not any future reconnect attempts will just return the same error, so
we can short-circuit the reconnect attempts and fail the connection
directly.

Signed-off-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/rdma.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 4697a94c0945..92c0bdf2bc74 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1067,8 +1067,11 @@ static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl)
 	kfree(ctrl);
 }
 
-static void nvme_rdma_reconnect_or_remove(struct nvme_rdma_ctrl *ctrl)
+static void nvme_rdma_reconnect_or_remove(struct nvme_rdma_ctrl *ctrl,
+		int status)
 {
+	bool recon = status > 0 && (status & NVME_SC_DNR) ? false : true;
+
 	/* If we are resetting/deleting then do nothing */
 	if (ctrl->ctrl.state != NVME_CTRL_CONNECTING) {
 		WARN_ON_ONCE(ctrl->ctrl.state == NVME_CTRL_NEW ||
@@ -1076,12 +1079,14 @@ static void nvme_rdma_reconnect_or_remove(struct nvme_rdma_ctrl *ctrl)
 		return;
 	}
 
-	if (nvmf_should_reconnect(&ctrl->ctrl)) {
+	if (recon && nvmf_should_reconnect(&ctrl->ctrl)) {
 		dev_info(ctrl->ctrl.device, "Reconnecting in %d seconds...\n",
 			ctrl->ctrl.opts->reconnect_delay);
 		queue_delayed_work(nvme_wq, &ctrl->reconnect_work,
 				ctrl->ctrl.opts->reconnect_delay * HZ);
 	} else {
+		dev_info(ctrl->ctrl.device, "Removing controller (%d)...\n",
+			 status);
 		nvme_delete_ctrl(&ctrl->ctrl);
 	}
 }
@@ -1166,10 +1171,12 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
 {
 	struct nvme_rdma_ctrl *ctrl = container_of(to_delayed_work(work),
 			struct nvme_rdma_ctrl, reconnect_work);
+	int ret;
 
 	++ctrl->ctrl.nr_reconnects;
 
-	if (nvme_rdma_setup_ctrl(ctrl, false))
+	ret = nvme_rdma_setup_ctrl(ctrl, false);
+	if (ret)
 		goto requeue;
 
 	dev_info(ctrl->ctrl.device, "Successfully reconnected (%d attempts)\n",
@@ -1182,7 +1189,7 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
 requeue:
 	dev_info(ctrl->ctrl.device, "Failed reconnect attempt %d\n",
 			ctrl->ctrl.nr_reconnects);
-	nvme_rdma_reconnect_or_remove(ctrl);
+	nvme_rdma_reconnect_or_remove(ctrl, ret);
 }
 
 static void nvme_rdma_error_recovery_work(struct work_struct *work)
@@ -1203,7 +1210,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 		return;
 	}
 
-	nvme_rdma_reconnect_or_remove(ctrl);
+	nvme_rdma_reconnect_or_remove(ctrl, -ENOTCONN);
 }
 
 static void nvme_rdma_error_recovery(struct nvme_rdma_ctrl *ctrl)
@@ -2259,6 +2266,7 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work)
 {
 	struct nvme_rdma_ctrl *ctrl =
 		container_of(work, struct nvme_rdma_ctrl, ctrl.reset_work);
+	int ret;
 
 	nvme_stop_ctrl(&ctrl->ctrl);
 	nvme_rdma_shutdown_ctrl(ctrl, false);
@@ -2269,14 +2277,15 @@ static void nvme_rdma_reset_ctrl_work(struct work_struct *work)
 		return;
 	}
 
-	if (nvme_rdma_setup_ctrl(ctrl, false))
+	ret = nvme_rdma_setup_ctrl(ctrl, false);
+	if (ret)
 		goto out_fail;
 
 	return;
 
 out_fail:
 	++ctrl->ctrl.nr_reconnects;
-	nvme_rdma_reconnect_or_remove(ctrl);
+	nvme_rdma_reconnect_or_remove(ctrl, ret);
 }
 
 static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = {
-- 
2.29.2



From kbusch at kernel.org  Wed Jun 23 07:50:07 2021
From: kbusch at kernel.org (Keith Busch)
Date: Wed, 23 Jun 2021 07:50:07 -0700
Subject: [PATCH 1/2] nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <9681def7-a751-f4d8-8a92-c5bee027a97a@purestorage.com>
References: <20210622002710.26591-1-cachen@purestorage.com>
 <20210622002710.26591-2-cachen@purestorage.com>
 <20210622150636.GA2359631@dhcp-10-100-145-180.wdc.com>
 <33919248-6c62-0974-ce88-3c0d0885e837@purestorage.com>
 <20210623015331.GB2360166@dhcp-10-100-145-180.wdc.com>
 <0faf018d-570b-4c26-a093-3d375bd70c26@purestorage.com>
 <9681def7-a751-f4d8-8a92-c5bee027a97a@purestorage.com>
Message-ID: <20210623145007.GC2360166@dhcp-10-100-145-180.wdc.com>

On Tue, Jun 22, 2021 at 09:14:21PM -0700, Casey Chen wrote:
> +/*
> + * Try getting shutdown_lock while setting up IO queues.
> + * Caller remember to unlock if success.
> + */
> +static int nvme_setup_io_queues_trylock(struct nvme_dev *dev)
> +{
> +?????? /*
> +?????? * Lock is being held by nvme_dev_disable(), fail early.
> +?????? */
> +?????? if (!mutex_trylock(&dev->shutdown_lock))
> +?????????????? return -ENODEV;
> +
> +?????? /*
> +?????? * Controller is in wrong state, fail early.
> +??????? */
> +?????? if (dev->ctrl.state != NVME_CTRL_CONNECTING) {
> +?????????????? mutex_unlock(&dev->shutdown_lock);
> +?????????????? return -ENODEV;
> +?????? }
> +
> +?????? return 0;
> +}
> +

Yes, thank you, this is aligned with what I suggested.


From Martin.Belanger at dell.com  Wed Jun 23 10:19:22 2021
From: Martin.Belanger at dell.com (Belanger, Martin)
Date: Wed, 23 Jun 2021 17:19:22 +0000
Subject: [PATCH] nvme-tcp: Use __dev_get_by_name instead dev_get_by_name
 for OPT_HOST_IFACE
In-Reply-To: <CAJ2QiJLWh3_gWf8Bx8c201hGzVGcqfEhT-J_OLD=2740w1+Qsw@mail.gmail.com>
References: <20210618133956.10637-1-pkushwaha@marvell.com>
 <20210618134244.GA16005@lst.de>
 <78d5833d-2038-811c-f6a4-2f14a3996fc5@grimberg.me>
 <CAJ2QiJLWh3_gWf8Bx8c201hGzVGcqfEhT-J_OLD=2740w1+Qsw@mail.gmail.com>
Message-ID: <SJ0PR19MB45442B71E8DAE398970BC1C6F2089@SJ0PR19MB4544.namprd19.prod.outlook.com>

> On Sat, Jun 19, 2021 at 12:44 AM Sagi Grimberg <sagi at grimberg.me> wrote:
> >
> >
> > >> dev_get_by_name() finds network device by name but it also
> > >> increases reference count.
> > >> Increasing the ref count,
> > >> If nvme-tcp queue is present and the network device driver is
> > >> removed before nvme_tcp, we will face the following continuous log:
> > >>    "kernel:unregister_netdevice: waiting for <eth> to become
> > >>    free. Usage count = 2"
> > >> And rmmod further halts. Similar case arises during reboot/shutdown
> > >> with nvme-tcp queue present and both never completes.
> > >>
> > >> As a fix we will use __dev_get_by_name() which find network device
> > >> by name without increasing any reference counter.
> > >
> > > And when you remove it without the refcount we'll now have a stale
> > > kernel pointer?
> >
> > The netdev is not actually needed, its to make sure that it exist as
> > reference in the host_iface afaict (the netdev is never referenced
> > afterwards), and I'm pretty sure there was a matching put somewhere in
> > the former versions of this patch.
> 
> Patch 3ede8f72a9a2 (?nvme-tcp: allow selecting the network interface for
> connections") exposed to this problem in the following cases:
> A)  create nvme-tcp queue, disconnect queue and remove network driver
> B)  create nvme-tcp queue and remove network driver
> 
> ?A? can be solved by putting dev_put() before kfree(ctrl):
> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index
> c7bd37103cf4..93639b5b795f 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -2155,6 +2155,8 @@ static void nvme_tcp_free_ctrl(struct nvme_ctrl
> *nctrl)
>         nvmf_free_options(nctrl->opts);
> free_ctrl:
>         kfree(ctrl->queues);
> +       if (ctrl->ndev)
> +               dev_put(ctrl->ndev);
>         kfree(ctrl);
> }
> 
> @@ -2586,6 +2588,8 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct
> device *dev,
> out_kfree_queues:
>         kfree(ctrl->queues);
> out_free_ctrl:
> +       if (ctrl->ndev)
> +               dev_put(ctrl->ndev);
>         kfree(ctrl);
>         return ERR_PTR(ret);
> }
> 
> But even using dev_put() will not solve ?B? in case of the removal of the
> network driver before nvme-tcp.
> 
> This was our initial motivation for using  __dev_get_by_name().
> It will solve both ?A? and ?B? and it will behave the same with or without the
> usage of the OPT_HOST_IFACE option.
> i.e. If we remove the network driver while controllers are connected it will end
> with nvme timeouts.
> 
> --pk

Sagi is right. The function dev_get_by_name() was used to verify that the interface name exists. We don't need to increase the ref count here and using __dev_get_by_name() is the right thing to do.
Martin

From malin1024 at gmail.com  Wed Jun 23 11:55:48 2021
From: malin1024 at gmail.com (Shai Malin)
Date: Wed, 23 Jun 2021 21:55:48 +0300
Subject: [PATCH v2 0/8] NVMeTCP Offload ULP
In-Reply-To: <20210621064528.GA5906@lst.de>
References: <20210621051046.28783-1-smalin@marvell.com>
 <20210621064528.GA5906@lst.de>
Message-ID: <CAKKgK4xuLbQpoPvCbJWS=xzLkUvcFoC-V71O_pDTsnUTi8brvg@mail.gmail.com>

On 6/21/21 9:45 AM, Christoph Hellwig wrote:
> This seems to lack drivers actually using the infrastructure.  That is not
> only completly not acceptable for merging code into the kernel, but also
> make the code impossible to review.

We will send v3 together with the offload device driver.


From malin1024 at gmail.com  Wed Jun 23 11:58:30 2021
From: malin1024 at gmail.com (Shai Malin)
Date: Wed, 23 Jun 2021 21:58:30 +0300
Subject: [PATCH v2 3/8] nvme-fabrics: Expose nvmf_check_required_opts()
 globally
In-Reply-To: <20210621064956.GA6019@lst.de>
References: <20210621051046.28783-1-smalin@marvell.com>
 <20210621051046.28783-4-smalin@marvell.com>
 <20210621064956.GA6019@lst.de>
Message-ID: <CAKKgK4y4m-SrV-C0pu_ro4XqR=h1s=XYh9AgLQRTGxHJPb_U3A@mail.gmail.com>

On Mon, 21 Jun 2021 at 09:49, Christoph Hellwig <hch at lst.de> wrote:
> On Mon, Jun 21, 2021 at 08:10:41AM +0300, Shai Malin wrote:
> > From: Prabhakar Kushwaha <pkushwaha at marvell.com>
> >
> > nvmf_check_required_opts() is used to check if user provided opts has
> > the required_opts or not. if not, it will log which options are not
> > provided.
> >
> > It can be leveraged by nvme-tcp-offload to check if provided opts are
> > supported by this specific vendor driver or not.
>
> WTF is a "vendor" driver?  Vendors are not really a concept we use anywhere
> in Linux.

We were trying to refer to the "offload device driver" as "vendor driver".
Will be fixed in v3.


From malin1024 at gmail.com  Wed Jun 23 12:02:48 2021
From: malin1024 at gmail.com (Shai Malin)
Date: Wed, 23 Jun 2021 22:02:48 +0300
Subject: [PATCH v2 4/8] nvme-tcp-offload: Add device scan implementation
In-Reply-To: <20210621065024.GB6019@lst.de>
References: <20210621051046.28783-1-smalin@marvell.com>
 <20210621051046.28783-5-smalin@marvell.com>
 <20210621065024.GB6019@lst.de>
Message-ID: <CAKKgK4w6wpMfdiGrvNDRrrMztdvijCYcRQwCgchMG9onOh=Pew@mail.gmail.com>

On Mon, 21 Jun 2021 at 09:50, Christoph Hellwig <hch at lst.de> wrote:
> On Mon, Jun 21, 2021 at 08:10:42AM +0300, Shai Malin wrote:
> > +static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nctrl)
>
> Please fix all these overly long lines in the whole series.

Sure. Will be fixed in v3.


From sagi at grimberg.me  Wed Jun 23 14:32:45 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 23 Jun 2021 14:32:45 -0700
Subject: [bug report] NVMe/IB: reset_controller need more than 1min
In-Reply-To: <CAHj4cs_VZ7C7ciKy-q51a+Gc=uce0GDKRHNmUdoGOd7KSvURpA@mail.gmail.com>
References: <CAHj4cs8cT23z+h2i+g6o3OQqEhWnHS88JO4jNoQo0Nww-sdkYg@mail.gmail.com>
 <3c86dc88-97d9-5a71-20e1-a90279f47db5@grimberg.me>
 <CAHj4cs_3eLZd=vxRRrnBU2W4H38mqttcy0ZdSw6uw4KvbJWgeQ@mail.gmail.com>
 <CAHj4cs_VZ7C7ciKy-q51a+Gc=uce0GDKRHNmUdoGOd7KSvURpA@mail.gmail.com>
Message-ID: <84208be5-a7a9-5261-398c-fa9bda3efbe3@grimberg.me>


> Hello
> 
> Gentle ping here, this issue still exists on latest 5.13-rc7
> 
> # time nvme reset /dev/nvme0
> 
> real 0m12.636s
> user 0m0.002s
> sys 0m0.005s
> # time nvme reset /dev/nvme0
> 
> real 0m12.641s
> user 0m0.000s
> sys 0m0.007s

Strange that even normal resets take so long...
What device are you using?

> # time nvme reset /dev/nvme0
> 
> real 1m16.133s
> user 0m0.000s
> sys 0m0.007s

There seems to be a spurious command timeout here, but maybe this
is due to the fact that the queues take so long to connect and
the target expires the keep-alive timer.

Does this patch help?
--
diff --git a/drivers/nvme/target/fabrics-cmd.c 
b/drivers/nvme/target/fabrics-cmd.c
index 7d0f3523fdab..f4a7db1ab3e5 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -142,6 +142,14 @@ static u16 nvmet_install_queue(struct nvmet_ctrl 
*ctrl, struct nvmet_req *req)
                 }
         }

+       /*
+        * Controller establishment flow may take some time, and the 
host may not
+        * send us keep-alive during this period, hence reset the
+        * traffic based keep-alive timer so we don't trigger a
+        * controller teardown as a result of a keep-alive expiration.
+        */
+       ctrl->reset_tbkas = true;
+
         return 0;

  err:
--

>> target:
>> [  934.306016] nvmet: creating controller 1 for subsystem testnqn for
>> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
>> [  944.875021] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
>> [  944.900051] nvmet: ctrl 1 fatal error occurred!
>> [ 1005.628340] nvmet: creating controller 1 for subsystem testnqn for
>> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
>>
>> client:
>> [  857.264029] nvme nvme0: resetting controller
>> [  864.115369] nvme nvme0: creating 40 I/O queues.
>> [  867.996746] nvme nvme0: mapped 40/0/0 default/read/poll queues.
>> [  868.001673] nvme nvme0: resetting controller
>> [  935.396789] nvme nvme0: I/O 9 QID 0 timeout
>> [  935.402036] nvme nvme0: Property Set error: 881, offset 0x14
>> [  935.438080] nvme nvme0: creating 40 I/O queues.
>> [  939.332125] nvme nvme0: mapped 40/0/0 default/read/poll queues.


From sagi at grimberg.me  Wed Jun 23 14:34:15 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 23 Jun 2021 14:34:15 -0700
Subject: [PATCH] nvme-tcp: Use __dev_get_by_name instead dev_get_by_name
 for OPT_HOST_IFACE
In-Reply-To: <20210618133956.10637-1-pkushwaha@marvell.com>
References: <20210618133956.10637-1-pkushwaha@marvell.com>
Message-ID: <ac92bc9e-0799-9aec-ce77-479a133fc3d3@grimberg.me>

Reviewed-by: Sagi Grimberg <sagi at grimberg.me>


From sagi at grimberg.me  Wed Jun 23 14:38:30 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 23 Jun 2021 14:38:30 -0700
Subject: [PATCH 0/2] nvme-fabrics: short-circuit connect retries
In-Reply-To: <20210623143250.82445-1-hare@suse.de>
References: <20210623143250.82445-1-hare@suse.de>
Message-ID: <8d7c79d9-0928-8c71-74fd-7185d67e7895@grimberg.me>


> Hi all,
> 
> commit f25f8ef70ce2 ("nvme-fc: short-circuit reconnect retries")
> allowed the fc transport to honour the DNR bit during reconnect
> retries, allowing to speed up error recovery.

How does this speed up error recovery?


From sagi at grimberg.me  Wed Jun 23 14:41:31 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 23 Jun 2021 14:41:31 -0700
Subject: [PATCHv7] nvme: fix refcounting imbalance when all paths are down
In-Reply-To: <a5baf9d1-47af-7a9a-e08f-f75c12429413@suse.de>
References: <20210622070140.22488-1-hare@suse.de>
 <6679737f-1af8-bc9e-5af1-ce3fac5b4d01@grimberg.me>
 <a5baf9d1-47af-7a9a-e08f-f75c12429413@suse.de>
Message-ID: <2d42c1ee-82c3-b53b-0fdc-baf9add01ec2@grimberg.me>


>>> +void nvme_mpath_check_last_path(struct nvme_ns_head *head)
>>>  ? {
>>> +??? bool last_path = false;
>>>  ????? if (!head->disk)
>>>  ????????? return;
>>> -??? if (head->disk->flags & GENHD_FL_UP) {
>>> -??????? nvme_cdev_del(&head->cdev, &head->cdev_device);
>>> -??????? del_gendisk(head->disk); > +
>>> +??? /* Synchronize with nvme_init_ns_head() */
>>> +??? mutex_lock(&head->subsys->lock);
>>> +??? if (list_empty(&head->list)) {
>>> +??????? list_del_init(&head->entry);
>>> +??????? last_path = true;
>>>  ????? }
>>
>> What happens now if CONFIG_NVME_MULTIPATH is not set? I'm not sure this
>> should be done here...
> 
> I don't mind _where_ this code ends up.
> As long as we can agree that we should do both, list_del_init() and
> del_gendisk() at the same time when we find that the last path is dead.
> Do we?

Yes, and independently from user/fs references...


From hare at suse.de  Wed Jun 23 22:51:15 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 24 Jun 2021 07:51:15 +0200
Subject: [PATCH 0/2] nvme-fabrics: short-circuit connect retries
In-Reply-To: <8d7c79d9-0928-8c71-74fd-7185d67e7895@grimberg.me>
References: <20210623143250.82445-1-hare@suse.de>
 <8d7c79d9-0928-8c71-74fd-7185d67e7895@grimberg.me>
Message-ID: <204a6656-ef83-b688-4979-26fe17f57974@suse.de>

On 6/23/21 11:38 PM, Sagi Grimberg wrote:
> 
>> Hi all,
>>
>> commit f25f8ef70ce2 ("nvme-fc: short-circuit reconnect retries")
>> allowed the fc transport to honour the DNR bit during reconnect
>> retries, allowing to speed up error recovery.
> 
> How does this speed up error recovery?

Well, not exactly error recovery (as there is nothing to recover).
But we won't attempt pointless retries, thereby reducing the noise in 
the message log.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From hch at lst.de  Wed Jun 23 23:34:07 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 24 Jun 2021 08:34:07 +0200
Subject: [PATCH] nvmet-rdma: process wr wait list from a non-atomic context
In-Reply-To: <20210622074257.1273123-1-sagi@grimberg.me>
References: <20210622074257.1273123-1-sagi@grimberg.me>
Message-ID: <20210624063407.GA26612@lst.de>

Thanks,

applied to nvme-5.14.


From hch at lst.de  Wed Jun 23 23:35:51 2021
From: hch at lst.de (Christoph Hellwig)
Date: Thu, 24 Jun 2021 08:35:51 +0200
Subject: [PATCH] nvme-tcp: Use __dev_get_by_name instead
 dev_get_by_name for OPT_HOST_IFACE
In-Reply-To: <20210618133956.10637-1-pkushwaha@marvell.com>
References: <20210618133956.10637-1-pkushwaha@marvell.com>
Message-ID: <20210624063551.GB26612@lst.de>

I've applied the slightly updated version below to make it a little
more clear what is going on:

---
>From 9de7d173c10b6be09fe9d5b7010ef182465897a1 Mon Sep 17 00:00:00 2001
From: Prabhakar Kushwaha <pkushwaha at marvell.com>
Date: Fri, 18 Jun 2021 16:39:56 +0300
Subject: nvme-tcp: use __dev_get_by_name instead dev_get_by_name for
 OPT_HOST_IFACE

dev_get_by_name() finds network device by name but it also increases the
reference count.

If a nvme-tcp queue is present and the network device driver is removed
before nvme_tcp, we will face the following continuous log:

  "kernel:unregister_netdevice: waiting for <eth> to become free. Usage count = 2"

And rmmod further halts. Similar case arises during reboot/shutdown
with nvme-tcp queue present and both never completes.

To fix this, use __dev_get_by_name() which finds network device by
name without increasing any reference counter.

Fixes: 3ede8f72a9a2 ("nvme-tcp: allow selecting the network interface for connections")
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Reviewed-by: Sagi Grimberg <sagi at grimberg.me>
[hch: remove the ->ndev member entirely]
Signed-off-by: Christoph Hellwig <hch at lst.de>
---
 drivers/nvme/host/tcp.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index c7bd37103cf4..2b46c6e0226f 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -123,7 +123,6 @@ struct nvme_tcp_ctrl {
 	struct blk_mq_tag_set	admin_tag_set;
 	struct sockaddr_storage addr;
 	struct sockaddr_storage src_addr;
-	struct net_device	*ndev;
 	struct nvme_ctrl	ctrl;
 
 	struct work_struct	err_work;
@@ -2533,8 +2532,7 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev,
 	}
 
 	if (opts->mask & NVMF_OPT_HOST_IFACE) {
-		ctrl->ndev = dev_get_by_name(&init_net, opts->host_iface);
-		if (!ctrl->ndev) {
+		if (!__dev_get_by_name(&init_net, opts->host_iface)) {
 			pr_err("invalid interface passed: %s\n",
 			       opts->host_iface);
 			ret = -ENODEV;
-- 
2.30.2



From lengchao at huawei.com  Thu Jun 24 00:29:45 2021
From: lengchao at huawei.com (Chao Leng)
Date: Thu, 24 Jun 2021 15:29:45 +0800
Subject: [PATCH 0/2] nvme-fabrics: short-circuit connect retries
In-Reply-To: <204a6656-ef83-b688-4979-26fe17f57974@suse.de>
References: <20210623143250.82445-1-hare@suse.de>
 <8d7c79d9-0928-8c71-74fd-7185d67e7895@grimberg.me>
 <204a6656-ef83-b688-4979-26fe17f57974@suse.de>
Message-ID: <cdf40226-12ed-5f87-8549-10b4defceae5@huawei.com>



On 2021/6/24 13:51, Hannes Reinecke wrote:
> On 6/23/21 11:38 PM, Sagi Grimberg wrote:
>>
>>> Hi all,
>>>
>>> commit f25f8ef70ce2 ("nvme-fc: short-circuit reconnect retries")
>>> allowed the fc transport to honour the DNR bit during reconnect
>>> retries, allowing to speed up error recovery.
>>
>> How does this speed up error recovery?
> 
> Well, not exactly error recovery (as there is nothing to recover).
> But we won't attempt pointless retries, thereby reducing the noise in the message log.
This conflict with the tcp and rdma target.
You may need to delete the improper NVME_SC_DNR at the target.
However, this will cause compatibility issues between different versions.
> 
> Cheers,
> 
> Hannes


From hare at suse.de  Thu Jun 24 01:10:17 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 24 Jun 2021 10:10:17 +0200
Subject: [PATCH 0/2] nvme-fabrics: short-circuit connect retries
In-Reply-To: <cdf40226-12ed-5f87-8549-10b4defceae5@huawei.com>
References: <20210623143250.82445-1-hare@suse.de>
 <8d7c79d9-0928-8c71-74fd-7185d67e7895@grimberg.me>
 <204a6656-ef83-b688-4979-26fe17f57974@suse.de>
 <cdf40226-12ed-5f87-8549-10b4defceae5@huawei.com>
Message-ID: <91ff9961-6537-05d1-c552-18a02d13bf0c@suse.de>

On 6/24/21 9:29 AM, Chao Leng wrote:
> 
> 
> On 2021/6/24 13:51, Hannes Reinecke wrote:
>> On 6/23/21 11:38 PM, Sagi Grimberg wrote:
>>>
>>>> Hi all,
>>>>
>>>> commit f25f8ef70ce2 ("nvme-fc: short-circuit reconnect retries")
>>>> allowed the fc transport to honour the DNR bit during reconnect
>>>> retries, allowing to speed up error recovery.
>>>
>>> How does this speed up error recovery?
>>
>> Well, not exactly error recovery (as there is nothing to recover).
>> But we won't attempt pointless retries, thereby reducing the noise in
>> the message log.
> This conflict with the tcp and rdma target.
> You may need to delete the improper NVME_SC_DNR at the target.
> However, this will cause compatibility issues between different versions.

Which ones?
I checked the DNR usage in the target code, and they seem to set it
correctly (ie the result would not change when the command is retried).
With the possible exception of ENOSPC handling, as this is arguably
dynamic and might change with a retry.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare at suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), GF: Felix Imend?rffer


From hare at suse.de  Thu Jun 24 01:55:46 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 24 Jun 2021 10:55:46 +0200
Subject: [PATCHv8] nvme: fix refcounting imbalance when all paths are down
Message-ID: <20210624085546.123824-1-hare@suse.de>

When the last path to a ns_head drops the current code
removes the ns_head from the subsystem list, but will only
delete the disk itself if the last reference to the ns_head
drops. This is causing an refcounting imbalance eg when
applications have a reference to the disk, as then they'll
never get notified that the disk is in fact dead.
This patch moves the call 'del_gendisk' into nvme_mpath_check_last_path(),
ensuring that the disk can be properly removed and applications get the
appropriate notifications.

Changes to v7:
- Move the list_del() call outside of nvme_mpath_check_last_path()
Changes to v6:
- Move the list_del() into nvme_mpath_check_last_path()
- Drop the tests for GENHD_FL_UP
Changes to v5:
- Synchronize between nvme_init_ns_head() and nvme_mpath_check_last_path()
- Check for removed gendisk in nvme_ns_head_submit_bio()
Changes to v4:
- Call del_gendisk() in nvme_mpath_check_last_path() to avoid deadlock
Changes to v3:
- Simplify if() clause to detect duplicate namespaces
Changes to v2:
- Drop memcpy() statement
Changes to v1:
- Always check NSIDs after reattach

Signed-off-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/core.c      | 14 +++++++++++---
 drivers/nvme/host/multipath.c | 11 +++++++++--
 drivers/nvme/host/nvme.h      | 11 ++---------
 3 files changed, 22 insertions(+), 14 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index c7ef0b6684b5..b7f25cdf91c8 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3813,6 +3813,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid,
 
 static void nvme_ns_remove(struct nvme_ns *ns)
 {
+	bool last_path = false;
+
 	if (test_and_set_bit(NVME_NS_REMOVING, &ns->flags))
 		return;
 
@@ -3821,8 +3823,6 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	mutex_lock(&ns->ctrl->subsys->lock);
 	list_del_rcu(&ns->siblings);
-	if (list_empty(&ns->head->list))
-		list_del_init(&ns->head->entry);
 	mutex_unlock(&ns->ctrl->subsys->lock);
 
 	synchronize_rcu(); /* guarantee not available in head->list */
@@ -3842,7 +3842,15 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	list_del_init(&ns->list);
 	up_write(&ns->ctrl->namespaces_rwsem);
 
-	nvme_mpath_check_last_path(ns);
+	/* Synchronize with nvme_init_ns_head() */
+	mutex_lock(&ns->head->subsys->lock);
+	if (list_empty(&ns->head->list)) {
+		list_del_init(&ns->head->entry);
+		last_path = true;
+	}
+	mutex_unlock(&ns->head->subsys->lock);
+	if (last_path)
+		nvme_mpath_check_last_path(ns->head);
 	nvme_put_ns(ns);
 }
 
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 23573fe3fc7d..04307638ffe7 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -771,20 +771,27 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, struct nvme_id_ns *id)
 #endif
 }
 
-void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+void nvme_mpath_check_last_path(struct nvme_ns_head *head)
 {
 	if (!head->disk)
 		return;
+	kblockd_schedule_work(&head->requeue_work);
 	if (head->disk->flags & GENHD_FL_UP) {
 		nvme_cdev_del(&head->cdev, &head->cdev_device);
 		del_gendisk(head->disk);
 	}
+}
+
+void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+{
+	if (!head->disk)
+		return;
 	blk_set_queue_dying(head->disk->queue);
 	/* make sure all pending bios are cleaned up */
 	kblockd_schedule_work(&head->requeue_work);
 	flush_work(&head->requeue_work);
 	blk_cleanup_queue(head->disk->queue);
-	if (!test_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
+	if (!test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
 		/*
 		 * if device_add_disk wasn't called, prevent
 		 * disk release to put a bogus reference on the
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 75420ceacc10..6b1caabe861b 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -716,14 +716,7 @@ void nvme_mpath_uninit(struct nvme_ctrl *ctrl);
 void nvme_mpath_stop(struct nvme_ctrl *ctrl);
 bool nvme_mpath_clear_current_path(struct nvme_ns *ns);
 void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl);
-
-static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)
-{
-	struct nvme_ns_head *head = ns->head;
-
-	if (head->disk && list_empty(&head->list))
-		kblockd_schedule_work(&head->requeue_work);
-}
+void nvme_mpath_check_last_path(struct nvme_ns_head *head);
 
 static inline void nvme_trace_bio_complete(struct request *req)
 {
@@ -772,7 +765,7 @@ static inline bool nvme_mpath_clear_current_path(struct nvme_ns *ns)
 static inline void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl)
 {
 }
-static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)
+static inline void nvme_mpath_check_last_path(struct nvme_ns_head *head)
 {
 }
 static inline void nvme_trace_bio_complete(struct request *req)
-- 
2.29.2



From hare at suse.de  Thu Jun 24 02:10:07 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 24 Jun 2021 11:10:07 +0200
Subject: [PATCH 1/1] nvme-fabrics: Generate uevent on namespace AENs
In-Reply-To: <20210503065235.GB32627@lst.de>
References: <BL0PR13MB429142FD8B9FCC57C2FE451D9C409@BL0PR13MB4291.namprd13.prod.outlook.com>
 <20210429063056.GA2482@lst.de>
 <SJ0PR19MB454499BF2899EF772B1EA11CF25F9@SJ0PR19MB4544.namprd19.prod.outlook.com>
 <BYAPR04MB4965BD2AE65469C96CBF1354865F9@BYAPR04MB4965.namprd04.prod.outlook.com>
 <BYAPR19MB29351987D7ABBA3F28791D978F5E9@BYAPR19MB2935.namprd19.prod.outlook.com>
 <20210430162104.GA907317@dhcp-10-100-145-180.wdc.com>
 <20210503065235.GB32627@lst.de>
Message-ID: <eafbb7f9-5481-6138-15d2-bdcde3a29929@suse.de>

On 5/3/21 8:52 AM, Christoph Hellwig wrote:
> On Fri, Apr 30, 2021 at 09:21:04AM -0700, Keith Busch wrote:
>> On Fri, Apr 30, 2021 at 05:14:55AM +0000, Rao, Vinay wrote:
>>> + Madhu
>>>
>>> Hi Chaitanya, 
>>>
>>> Some of the AEN's like ANA state change and Name Space attribute change events are of interest outside the NVME driver. 
>>>
>>> It would be good to consider these events to be propagated up to user space. I am not yet convinced on how propagating this above to user space is problematic.
>>
>> The namespace attribute change event pairs with the Changed Namespace
>> List log, and reading that log will change the result for a subsequent
>> reader. User space racing with the kernel on a log access when there are
>> read side effects creates non-deterministic behavior, and that is
>> problematic.
> 
> The only way I could think of making this work is by:
> 
> forcing the RAE bit on for all log pages that the kernel cares about,
> and only delivering the uevent on controllers that actually do support
> the RAE bit (IIRC it got added in 1.3, but I'd have to check).
> 
Yep, that sounds like a good idea.
I would even go so far as to inhibit any user-space commands which read
those log pages and which have the RAE bit _not_ set, as this has the
potential of messing up the kernel operations...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare at suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), GF: Felix Imend?rffer


From hare at suse.de  Thu Jun 24 02:17:33 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 24 Jun 2021 11:17:33 +0200
Subject: [PATCH 1/1] nvme-fabrics: Generate uevent on namespace AENs
In-Reply-To: <20210503065235.GB32627@lst.de>
References: <BL0PR13MB429142FD8B9FCC57C2FE451D9C409@BL0PR13MB4291.namprd13.prod.outlook.com>
 <20210429063056.GA2482@lst.de>
 <SJ0PR19MB454499BF2899EF772B1EA11CF25F9@SJ0PR19MB4544.namprd19.prod.outlook.com>
 <BYAPR04MB4965BD2AE65469C96CBF1354865F9@BYAPR04MB4965.namprd04.prod.outlook.com>
 <BYAPR19MB29351987D7ABBA3F28791D978F5E9@BYAPR19MB2935.namprd19.prod.outlook.com>
 <20210430162104.GA907317@dhcp-10-100-145-180.wdc.com>
 <20210503065235.GB32627@lst.de>
Message-ID: <45d90119-0193-0110-0b66-803ffc84bc69@suse.de>

On 5/3/21 8:52 AM, Christoph Hellwig wrote:
> On Fri, Apr 30, 2021 at 09:21:04AM -0700, Keith Busch wrote:
>> On Fri, Apr 30, 2021 at 05:14:55AM +0000, Rao, Vinay wrote:
>>> + Madhu
>>>
>>> Hi Chaitanya, 
>>>
>>> Some of the AEN's like ANA state change and Name Space attribute change events are of interest outside the NVME driver. 
>>>
>>> It would be good to consider these events to be propagated up to user space. I am not yet convinced on how propagating this above to user space is problematic.
>>
>> The namespace attribute change event pairs with the Changed Namespace
>> List log, and reading that log will change the result for a subsequent
>> reader. User space racing with the kernel on a log access when there are
>> read side effects creates non-deterministic behavior, and that is
>> problematic.
> 
> The only way I could think of making this work is by:
> 
> forcing the RAE bit on for all log pages that the kernel cares about,
> and only delivering the uevent on controllers that actually do support
> the RAE bit (IIRC it got added in 1.3, but I'd have to check).
> 

... and you are correct, it went into the base spec into 1.3.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare at suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), GF: Felix Imend?rffer


From hare at suse.de  Thu Jun 24 02:24:27 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 24 Jun 2021 11:24:27 +0200
Subject: [LSF/MM/BPF Topic] Towards more useful nvme-passthrough
In-Reply-To: <20210609105050.127009-1-joshi.k@samsung.com>
References: <CGME20210609105347epcas5p42ab916655fca311157a38d54f79f95e7@epcas5p4.samsung.com>
 <20210609105050.127009-1-joshi.k@samsung.com>
Message-ID: <d9e6e3cc-fedb-eb45-7a3e-5df24e67455b@suse.de>

On 6/9/21 12:50 PM, Kanchan Joshi wrote:
> Background & objectives:
> ------------------------
> 
> The NVMe passthrough interface
> 
> Good part: allows new device-features to be usable (at least in raw
> form) without having to build block-generic cmds, in-kernel users,
> emulations and file-generic user-interfaces - all this take some time to
> evolve.
> 
> Bad part: passthrough interface has remain tied to synchronous ioctl,
> which is a blocker for performance-centric usage scenarios. User-space
> can take the pain of implementing async-over-sync on its own but it does
> not make much sense in a world that already has io_uring.
> 
> Passthrough is lean in the sense it cuts through layers of abstractions
> and reaches to NVMe fast. One of the objective here is to build a
> scalable pass-through that can be readily used to play with new/emerging
> NVMe features.  Another is to surpass/match existing raw/direct block
> I/O performance with this new in-kernel path.
> 
> Recent developments:
> --------------------
> - NVMe now has a per-namespace char interface that remains available/usable
>   even for unsupported features and for new command-sets [1].
> 
> - Jens has proposed async-ioctl like facility 'uring_cmd' in io_uring. This
>   introduces new possibilities (beyond storage); async-passthrough is one of
> those. Last posted version is V4 [2].
> 
> - I have posted work on async nvme passthrough over block-dev [3]. Posted work
>   is in V4 (in sync with the infra of [2]).
> 
> Early performance numbers:
> --------------------------
> fio, randread, 4k bs, 1 job
> Kiops, with varying QD:
> 
> QD      Sync-PT         io_uring        Async-PT
> 1         10.8            10.6            10.6
> 2         10.9            24.5            24
> 4         10.6            45              46
> 8         10.9            90              89
> 16        11.0            169             170
> 32        10.6            308             307
> 64        10.8            503             506
> 128       10.9            592             596
> 
> Further steps/discussion points:
> --------------------------------
> 1.Async-passthrough over nvme char-dev
> It is in a shape to receive feedback, but I am not sure if community
> would like to take a look at that before settling on uring-cmd infra.
> 
> 2.Once above gets in shape, bring other perf-centric features of io_uring to
> this path -
> A. SQPoll and register-file: already functional.
> B. Passthrough polling: This can be enabled for block and looks feasible for
> char-interface as well.  Keith recently posted enabling polling for user
> pass-through [4]
> C. Pre-mapped buffers: Early thought is to let the buffers registered by
> io_uring, and add a new passthrough ioctl/uring_cmd in driver which does
> everything that passthrough does except pinning/unpinning the pages.
> 
> 3. Are there more things in the "io_uring->nvme->[block-layer]->nvme" path
> which can be optimized.
> 
> Ideally I'd like to cover good deal of ground before Dec. But there seems
> plenty of possibilities on this path.  Discussion would help in how best to
> move forward, and cement the ideas.
> 
> [1] https://lore.kernel.org/linux-nvme/20210421074504.57750-1-minwoo.im.dev at gmail.com/
> [2] https://lore.kernel.org/linux-nvme/20210317221027.366780-1-axboe at kernel.dk/
> [3] https://lore.kernel.org/linux-nvme/20210325170540.59619-1-joshi.k at samsung.com/
> [4] https://lore.kernel.org/linux-block/20210517171443.GB2709391 at dhcp-10-100-145-180.wdc.com/#t
> 
I do like the idea.

What I would like to see is to make the ioring_cmd infrastructure
generally available, such that we can port the SCSI sg asynchronous
interface over to this.
Doug Gilbert has been fighting a lone battle to improve the sg
asynchronous interface, as the current one is deemed a security hazard.
But in the absence of a generic interface he had to design his own
ioctls, with all the expected pushback.
Plus there are only so many people who care about sg internals :-(

Being able to use ioring_cmd would be a neat way out of this.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare at suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), GF: Felix Imend?rffer


From hare at suse.de  Thu Jun 24 03:49:20 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 24 Jun 2021 12:49:20 +0200
Subject: [PATCH 1/1] nvme-fabrics: Generate uevent on namespace AENs
In-Reply-To: <20210430162104.GA907317@dhcp-10-100-145-180.wdc.com>
References: <BL0PR13MB429142FD8B9FCC57C2FE451D9C409@BL0PR13MB4291.namprd13.prod.outlook.com>
 <20210429063056.GA2482@lst.de>
 <SJ0PR19MB454499BF2899EF772B1EA11CF25F9@SJ0PR19MB4544.namprd19.prod.outlook.com>
 <BYAPR04MB4965BD2AE65469C96CBF1354865F9@BYAPR04MB4965.namprd04.prod.outlook.com>
 <BYAPR19MB29351987D7ABBA3F28791D978F5E9@BYAPR19MB2935.namprd19.prod.outlook.com>
 <20210430162104.GA907317@dhcp-10-100-145-180.wdc.com>
Message-ID: <097e56b6-2b13-e00c-ef84-509fa0e887ca@suse.de>

On 4/30/21 6:21 PM, Keith Busch wrote:
> On Fri, Apr 30, 2021 at 05:14:55AM +0000, Rao, Vinay wrote:
>> + Madhu
>>
>> Hi Chaitanya, 
>>
>> Some of the AEN's like ANA state change and Name Space attribute change events are of interest outside the NVME driver. 
>>
>> It would be good to consider these events to be propagated up to user space. I am not yet convinced on how propagating
>> this above to user space is problematic.
> 
> The namespace attribute change event pairs with the Changed Namespace
> List log, and reading that log will change the result for a subsequent
> reader. User space racing with the kernel on a log access when there are
> read side effects creates non-deterministic behavior, and that is
> problematic.
> 
It would, if we were relying on the Changed Namespace List log.
But in the end we're just reading it to clear the AEN flag; the contents
are completely immaterial to us as we're doing a full namespace scan
afterwards anyway.
So I don't think that that'll be a problem for us.

What will be problematic, though, is if we were to allow userspace to
register an AEN; as soon as the controller allows more than one pending
AERs there nothing preventing it from doing so.
If he does our internal states will be messed up as we're not receiving
all events. I'd rather error out any attempt here, but then we can't do
a full passthrough, and I guess things like SPDK might become unhappy.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare at suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), GF: Felix Imend?rffer


From Ran.Anner at dell.com  Thu Jun 24 05:33:50 2021
From: Ran.Anner at dell.com (Anner, Ran)
Date: Thu, 24 Jun 2021 12:33:50 +0000
Subject: [PATCH] nvmet: allow setting ctrl to ready without iosqes/iocqes
 set
In-Reply-To: <20210511163348.GA18486@lst.de>
References: <20210502153106.18970-1-ran.anner@dell.com>
 <20210504091236.GA22900@lst.de>
 <MW2PR1901MB20756A164FC6DE36BE5D718291549@MW2PR1901MB2075.namprd19.prod.outlook.com>
 <20210511163348.GA18486@lst.de>
Message-ID: <BL0PR1901MB206733CD44E09D208EEA9D0791079@BL0PR1901MB2067.namprd19.prod.outlook.com>

Reviving this thread.

-----Original Message-----
From: Christoph Hellwig <hch at lst.de> 
Sent: Tuesday, May 11, 2021 19:34
To: Anner, Ran
Cc: Christoph Hellwig
Subject: Re: [PATCH] nvmet: allow setting ctrl to ready without iosqes/iocqes set


[EXTERNAL EMAIL] 

Please resend the mail with the mainling list in the loop.

On Mon, May 10, 2021 at 12:27:36PM +0000, Anner, Ran wrote:
> Hi Christoph.
> 
> This is for IO controller.
> We saw a host behavior where only the cc.en bit was set and the host was waiting for csts to indicate ready state.
> >From what I saw in the nvme specification we should only fail connect commands that create IO queues in case IOSQES/IOCQES is not set and csts should not be effected.
> 
> Thanks
> 
> -----Original Message-----
> From: Christoph Hellwig <hch at lst.de>
> Sent: Tuesday, May 4, 2021 12:13
> To: Anner, Ran
> Cc: hch at lst.de; linux-nvme at lists.infradead.org
> Subject: Re: [PATCH] nvmet: allow setting ctrl to ready without 
> iosqes/iocqes set
> 
> 
> [EXTERNAL EMAIL]
> 
> On Sun, May 02, 2021 at 06:31:06PM +0300, ran.anner at dell.com wrote:
> > From: Ran Anner <Ran.Anner at dell.com>
> > 
> > According to the spec we should fail creation of IO queues if iosqes 
> > or iocqes properties are not set but does not mention preventing 
> > from changing ctrl status to ready.
> > We have seen host implementation which sets the property to 0x1 and 
> > waits endlessly until ctrl status changes to ready.
> 
> So is that for a discovery controller?  The spec says the field shall be read-only for them, although we don't reject writes to the property right now.
---end quoted text---


From hare at suse.de  Thu Jun 24 06:48:43 2021
From: hare at suse.de (Hannes Reinecke)
Date: Thu, 24 Jun 2021 15:48:43 +0200
Subject: [PATCH] nvme-fabrics: make cntlid settable
Message-ID: <20210624134843.138389-1-hare@suse.de>

As per spec the controller ID can be set if a subsystem implements the
the static controller model, so add a 'cntlid' option to the connect
string.

Signed-off-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/fabrics.c | 19 ++++++++++++++++++-
 drivers/nvme/host/fabrics.h |  2 ++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 1e6a7cc056ca..c836625a6745 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -115,6 +115,9 @@ int nvmf_get_address(struct nvme_ctrl *ctrl, char *buf, int size)
 	if (ctrl->opts->mask & NVMF_OPT_HOST_IFACE)
 		len += scnprintf(buf + len, size - len, "%shost_iface=%s",
 				(len) ? "," : "", ctrl->opts->host_iface);
+	if (ctrl->opts->mask & NVMF_OPT_CNTLID)
+		len += scnprintf(buf + len, size - len, "%scntlid=%d",
+				(len) ? "," : "", ctrl->opts->cntlid);
 	len += scnprintf(buf + len, size - len, "\n");
 
 	return len;
@@ -385,7 +388,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		return -ENOMEM;
 
 	uuid_copy(&data->hostid, &ctrl->opts->host->id);
-	data->cntlid = cpu_to_le16(0xffff);
+	data->cntlid = cpu_to_le16(ctrl->opts->cntlid);
 	strncpy(data->subsysnqn, ctrl->opts->subsysnqn, NVMF_NQN_SIZE);
 	strncpy(data->hostnqn, ctrl->opts->host->nqn, NVMF_NQN_SIZE);
 
@@ -541,6 +544,7 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_HOST_TRADDR,		"host_traddr=%s"	},
 	{ NVMF_OPT_HOST_IFACE,		"host_iface=%s"		},
 	{ NVMF_OPT_HOST_ID,		"hostid=%s"		},
+	{ NVMF_OPT_CNTLID,		"cntlid=%d"		},
 	{ NVMF_OPT_DUP_CONNECT,		"duplicate_connect"	},
 	{ NVMF_OPT_DISABLE_SQFLOW,	"disable_sqflow"	},
 	{ NVMF_OPT_HDR_DIGEST,		"hdr_digest"		},
@@ -572,6 +576,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	opts->hdr_digest = false;
 	opts->data_digest = false;
 	opts->tos = -1; /* < 0 == use transport default */
+	opts->cntlid = 0xffff;
 
 	options = o = kstrdup(buf, GFP_KERNEL);
 	if (!options)
@@ -773,6 +778,18 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 			}
 			kfree(p);
 			break;
+		case NVMF_OPT_CNTLID:
+			if (match_int(args, &token)) {
+				ret = -EINVAL;
+				goto out;
+			}
+			if (token < 0 || token > 0xffff) {
+				pr_err("Invalid controller id %d\n", token);
+				ret = -EINVAL;
+				goto out;
+			}
+			opts->cntlid = token;
+			break;
 		case NVMF_OPT_DUP_CONNECT:
 			opts->duplicate_connect = true;
 			break;
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index c31dad69a773..8e32a64cefff 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -67,6 +67,7 @@ enum {
 	NVMF_OPT_TOS		= 1 << 19,
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
+	NVMF_OPT_CNTLID		= 1 << 22,
 };
 
 /**
@@ -112,6 +113,7 @@ struct nvmf_ctrl_options {
 	char			*trsvcid;
 	char			*host_traddr;
 	char			*host_iface;
+	unsigned int		cntlid;
 	size_t			queue_size;
 	unsigned int		nr_io_queues;
 	unsigned int		reconnect_delay;
-- 
2.29.2



From axboe at kernel.dk  Thu Jun 24 08:39:27 2021
From: axboe at kernel.dk (Jens Axboe)
Date: Thu, 24 Jun 2021 09:39:27 -0600
Subject: [GIT PULL] second round of nvme updates for Linux 5.14
In-Reply-To: <YNGLn2MsJDwwbZbv@infradead.org>
References: <YNGLn2MsJDwwbZbv@infradead.org>
Message-ID: <e6c5fddd-6829-a3ee-a270-c7e3a81a0c9a@kernel.dk>

On 6/22/21 1:05 AM, Christoph Hellwig wrote:
> The following changes since commit e0d245e2230998e66dfda10fb8c413f29196eb1c:
> 
>   Merge branch 'md-next' of https://git.kernel.org/pub/scm/linux/kernel/git/song/md into for-5.14/drivers (2021-06-15 15:42:56 -0600)
> 
> are available in the Git repository at:
> 
>   git://git.infradead.org/nvme.git tags/nvme-5.14-2021-06-22

Pulled, thanks.

-- 
Jens Axboe



From yi.zhang at redhat.com  Thu Jun 24 09:14:42 2021
From: yi.zhang at redhat.com (Yi Zhang)
Date: Fri, 25 Jun 2021 00:14:42 +0800
Subject: [bug report] NVMe/IB: reset_controller need more than 1min
In-Reply-To: <84208be5-a7a9-5261-398c-fa9bda3efbe3@grimberg.me>
References: <CAHj4cs8cT23z+h2i+g6o3OQqEhWnHS88JO4jNoQo0Nww-sdkYg@mail.gmail.com>
 <3c86dc88-97d9-5a71-20e1-a90279f47db5@grimberg.me>
 <CAHj4cs_3eLZd=vxRRrnBU2W4H38mqttcy0ZdSw6uw4KvbJWgeQ@mail.gmail.com>
 <CAHj4cs_VZ7C7ciKy-q51a+Gc=uce0GDKRHNmUdoGOd7KSvURpA@mail.gmail.com>
 <84208be5-a7a9-5261-398c-fa9bda3efbe3@grimberg.me>
Message-ID: <CAHj4cs8dgNNE5qcX3Y4ykuTYU8z_kea6=q64Pn_2vsdodgOJZQ@mail.gmail.com>

On Thu, Jun 24, 2021 at 5:32 AM Sagi Grimberg <sagi at grimberg.me> wrote:
>
>
> > Hello
> >
> > Gentle ping here, this issue still exists on latest 5.13-rc7
> >
> > # time nvme reset /dev/nvme0
> >
> > real 0m12.636s
> > user 0m0.002s
> > sys 0m0.005s
> > # time nvme reset /dev/nvme0
> >
> > real 0m12.641s
> > user 0m0.000s
> > sys 0m0.007s
>
> Strange that even normal resets take so long...
> What device are you using?

Hi Sagi

Here is the device info:
Mellanox Technologies MT27700 Family [ConnectX-4]

>
> > # time nvme reset /dev/nvme0
> >
> > real 1m16.133s
> > user 0m0.000s
> > sys 0m0.007s
>
> There seems to be a spurious command timeout here, but maybe this
> is due to the fact that the queues take so long to connect and
> the target expires the keep-alive timer.
>
> Does this patch help?

The issue still exists, let me know if you need more testing for it. :)


> --
> diff --git a/drivers/nvme/target/fabrics-cmd.c
> b/drivers/nvme/target/fabrics-cmd.c
> index 7d0f3523fdab..f4a7db1ab3e5 100644
> --- a/drivers/nvme/target/fabrics-cmd.c
> +++ b/drivers/nvme/target/fabrics-cmd.c
> @@ -142,6 +142,14 @@ static u16 nvmet_install_queue(struct nvmet_ctrl
> *ctrl, struct nvmet_req *req)
>                  }
>          }
>
> +       /*
> +        * Controller establishment flow may take some time, and the
> host may not
> +        * send us keep-alive during this period, hence reset the
> +        * traffic based keep-alive timer so we don't trigger a
> +        * controller teardown as a result of a keep-alive expiration.
> +        */
> +       ctrl->reset_tbkas = true;
> +
>          return 0;
>
>   err:
> --
>
> >> target:
> >> [  934.306016] nvmet: creating controller 1 for subsystem testnqn for
> >> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> >> [  944.875021] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
> >> [  944.900051] nvmet: ctrl 1 fatal error occurred!
> >> [ 1005.628340] nvmet: creating controller 1 for subsystem testnqn for
> >> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> >>
> >> client:
> >> [  857.264029] nvme nvme0: resetting controller
> >> [  864.115369] nvme nvme0: creating 40 I/O queues.
> >> [  867.996746] nvme nvme0: mapped 40/0/0 default/read/poll queues.
> >> [  868.001673] nvme nvme0: resetting controller
> >> [  935.396789] nvme nvme0: I/O 9 QID 0 timeout
> >> [  935.402036] nvme nvme0: Property Set error: 881, offset 0x14
> >> [  935.438080] nvme nvme0: creating 40 I/O queues.
> >> [  939.332125] nvme nvme0: mapped 40/0/0 default/read/poll queues.
>


-- 
Best Regards,
  Yi Zhang



From smalin at marvell.com  Thu Jun 24 10:23:50 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:23:50 +0300
Subject: [PATCH v3 00/20] NVMeTCP Offload ULP
Message-ID: <20210624172410.14225-1-smalin@marvell.com>

With the goal of enabling a generic infrastructure that allows NVMe/TCP
offload devices like NICs to seamlessly plug into the NVMe-oF stack, this
patch series introduces the nvme-tcp-offload ULP host layer, which will
be a new transport type called "tcp-offload" and will serve as an 
abstraction layer to work with device specific nvme-tcp offload drivers.

NVMeTCP offload is a full offload of the NVMeTCP protocol, this includes 
both the TCP level and the NVMeTCP level.

The nvme-tcp-offload transport can co-exist with the existing tcp and 
other transports. The tcp offload was designed so that stack changes are 
kept to a bare minimum: only registering new transports.
All other APIs, ops etc. are identical to the regular tcp transport.
Representing the TCP offload as a new transport allows clear and manageable
differentiation between the connections which should use the offload path 
and those that are not offloaded (even on the same device).

The nvme-tcp-offload layers and API compared to nvme-tcp and nvme-rdma:

* NVMe layer: *

       [ nvme/nvme-fabrics/blk-mq ]
             |
        (nvme API and blk-mq API)
             |
             |			 
* Transport layer: *

      [ nvme-rdma ] [ nvme-tcp ] [ nvme-tcp-offload ]
             |        |             |
           (Verbs) 
             |        |             |
             |     (Socket)
             |        |             |
             |        |        (nvme-tcp-offload API)
             |        |             |
             |        |             |
* Transport Driver: *

             |        |             |
      [ RDMA driver ]       
                      |             |
             [ Network driver ]
                                    |
                       [ NVMeTCP Offload driver ]

Upstream plan:
==============
As discussed in RFCV7, "NVMeTCP Offload ULP and QEDN Device Driver" 
contains 3 parts:
https://lore.kernel.org/linux-nvme/20210531225222.16992-1-smalin at marvell.com/

This series contains part1 and part3, intended for linux-nvme:
- Part 1: The nvme-tcp-offload patches
- Part 3: Marvell's Offload device driver(qedn) patches. 
          It has "compilation dependency" on both Part 1 and Part 2. 

Part 2 is already accepted in net-next.git:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=eda1bc65b0dc1b03006e427430ba23746ec44714


Usage:
======
The user will interact with the network-device in order to configure 
the ip/vlan - Logically similar to the RDMA model.
The NVMeTCP configuration is populated as part of the 
nvme connect command.

Example:
Assign IP to the net-device (from any existing Linux tool):

    ip addr add 100.100.0.101/24 dev p1p1

This IP will be used by both net-device and offload-device.

In order to connect from "sw" nvme-tcp through the net-device:

    nvme connect -t tcp -s 4420 -a 100.100.0.100 -n testnqn

In order to connect from "offload" nvme-tcp through the offload-device:

    nvme connect -t tcp_offload -s 4420 -a 100.100.0.100 -n testnqn
	
An alternative approach, and as a future enhancement that will not impact this 
series will be to modify nvme-cli with a new flag that will determine 
if "-t tcp" should be the regular nvme-tcp (which will be the default) 
or nvme-tcp-offload.
Exmaple:
    nvme connect -t tcp -s 4420 -a 100.100.0.100 -n testnqn -[new flag]


Queue Initialization Design:
============================
The nvme-tcp-offload ULP module shall register with the existing 
nvmf_transport_ops (.name = "tcp_offload"), nvme_ctrl_ops and blk_mq_ops.
The nvme-tcp-offload driver shall register to nvme-tcp-offload ULP
with the following ops:
- claim_dev() - in order to resolve the route to the target according to
                the paired net_dev.
- create_queue() - in order to create offloaded nvme-tcp queue.

The nvme-tcp-offload ULP module shall manage all the controller level
functionalities, call claim_dev and based on the return values shall call
the relevant module create_queue in order to create the admin queue and
the IO queues.


IO-path Design:
===============
The nvme-tcp-offload shall work at the IO-level - the nvme-tcp-offload 
ULP module shall pass the request (the IO) to the nvme-tcp-offload 
driver and later, the nvme-tcp-offload driver returns the request
completion (the IO completion).
No additional handling is needed in between; this design will reduce the
CPU utilization as we will describe below.

The nvme-tcp-offload driver shall register to nvme-tcp-offload ULP
with the following IO-path ops:
- send_req() - in order to pass the request to the handling of the
               offload driver that shall pass it to the offload device 
			   specific device.
- poll_queue()

Once the IO completes, the nvme-tcp-offload driver shall call 
command.done() that will invoke the nvme-tcp-offload ULP layer to
complete the request.


Teardown and errors:
====================
In case of NVMeTCP queue error the nvme-tcp-offload driver shall
call the nvme_tcp_ofld_report_queue_err.
The nvme-tcp-offload driver shall register to nvme-tcp-offload ULP
with the following teardown ops:
- drain_queue()
- destroy_queue()


The Marvell qedn driver:
========================
The new driver will be added under "drivers/nvme/hw" and will be enabled
by the Kconfig "Marvell NVM Express over Fabrics TCP offload".
As part of the qedn init, the driver will register as a pci device driver 
and will work with the Marvell fastlinQ NIC.
As part of the probe, the driver will register to the nvme_tcp_offload
(ULP) and to the qed module (qed_nvmetcp_ops) - similar to other
"qed_*_ops" which are used by the qede, qedr, qedf and qedi device
drivers.


Changes since RFC v1:
=====================
- nvme-tcp-offload: Fix nvme_tcp_ofld_ops return values.
- nvme-tcp-offload: Remove NVMF_TRTYPE_TCP_OFFLOAD.
- nvme-tcp-offload: Add nvme_tcp_ofld_poll() implementation.
- nvme-tcp-offload: Fix nvme_tcp_ofld_queue_rq() to check map_sg() and 
  send_req() return values.

Changes since RFC v2:
=====================
- nvme-tcp-offload: Fixes in controller and queue level (patches 3-6).
- qedn: Add the Marvell's NVMeTCP HW offload device driver init and probe
  (patches 8-11).
  
Changes since RFC v3:
=====================
- nvme-tcp-offload: Add the full implementation of the nvme-tcp-offload layer 
  including the new ops: setup_ctrl(), release_ctrl(), commit_rqs() and new 
  flows (ASYNC and timeout).
- nvme-tcp-offload: Add device maximums: max_hw_sectors, max_segments.
- nvme-tcp-offload: layer design and optimization changes.
- qedn: Add full implementation for the conn level, IO path and error handling.

Changes since RFC v4:
=====================
(Many thanks to Hannes Reinecke for his feedback)
- nvme_tcp_offload: Add num_hw_vectors in order to limit the number of queues.
- nvme_tcp_offload: Add per device private_data.
- nvme_tcp_offload: Fix header digest, data digest and tos initialization.
- qedn: Remove the qedn_global list.
- qedn: Remove the workqueue flow from send_req.
- qedn: Add db_recovery support.

Changes since RFC v5:
=====================
(Many thanks to Sagi Grimberg for his feedback)
- nvme-fabrics: Expose nvmf_check_required_opts() globally (as a new patch).
- nvme_tcp_offload: Remove io-queues BLK_MQ_F_BLOCKING.
- nvme_tcp_offload: Fix the nvme_tcp_ofld_stop_queue (drain_queue) flow.
- nvme_tcp_offload: Fix the nvme_tcp_ofld_free_queue (destroy_queue) flow.
- nvme_tcp_offload: Change rwsem to mutex.
- nvme_tcp_offload: remove redundant fields.
- nvme_tcp_offload: Remove the "new" from setup_ctrl().
- nvme_tcp_offload: Remove the init_req() and commit_rqs() ops.
- nvme_tcp_offload: Minor fixes in nvme_tcp_ofld_create_ctrl() ansd 
  nvme_tcp_ofld_free_queue().
- nvme_tcp_offload: Patch 8 (timeout and async) was squeashed into 
  patch 7 (io level).
- qedn: Fix the free_queue flow and the destroy_queue flow.
- qedn: Remove version number.

Changes since RFC v6:
=====================
- No changes in nvme_tcp_offload
- qedn: Remove redundant logic in the io-queues core affinity initialization.
- qedn: Remove qedn_validate_cccid_in_range().

Changes since v1:
=====================
- nvme_tcp_offload: Add support for NVME_OPT_HOST_IFACE.
- nvme_tcp_offload: Kconfig fix (thanks to Petr Mladek).
- nvme_tcp_offload: return code fix (thanks to Dan Carpenter).

Changes since v2:
=====================
- nvme_tcp_offload: Fix overly long lines.
- nvme_tcp_offload: use correct terminology for vendor driver.
- qedn: Added qedn driver as part of series.


Arie Gershberg (2):
  nvme-tcp-offload: Add controller level implementation
  nvme-tcp-offload: Add controller level error recovery implementation

Dean Balandin (3):
  nvme-tcp-offload: Add device scan implementation
  nvme-tcp-offload: Add queue level implementation
  nvme-tcp-offload: Add IO level implementation

Nikolay Assa (1):
  qedn: Add qedn_claim_dev API support

Prabhakar Kushwaha (7):
  nvme-fabrics: Move NVMF_ALLOWED_OPTS and NVMF_REQUIRED_OPTS
    definitions
  nvme-fabrics: Expose nvmf_check_required_opts() globally
  qedn: Add connection-level slowpath functionality
  qedn: Add support of configuring HW filter block
  qedn: Add support of Task and SGL
  qedn: Add support of NVME ICReq & ICResp
  qedn: Add support of ASYNC

Shai Malin (7):
  nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW offload ULP
  qedn: Add qedn - Marvell's NVMeTCP HW offload device driver
  qedn: Add qedn probe
  qedn: Add IRQ and fast-path resources initializations
  qedn: Add IO level qedn_send_req and fw_cq workqueue
  qedn: Add IO level fastpath functionality
  qedn: Add Connection and IO level recovery flows

 MAINTAINERS                      |   18 +
 drivers/nvme/Kconfig             |    1 +
 drivers/nvme/Makefile            |    1 +
 drivers/nvme/host/Kconfig        |   15 +
 drivers/nvme/host/Makefile       |    3 +
 drivers/nvme/host/fabrics.c      |   12 +-
 drivers/nvme/host/fabrics.h      |    9 +
 drivers/nvme/host/tcp-offload.c  | 1345 ++++++++++++++++++++++++++++++
 drivers/nvme/host/tcp-offload.h  |  207 +++++
 drivers/nvme/hw/Kconfig          |    9 +
 drivers/nvme/hw/Makefile         |    3 +
 drivers/nvme/hw/qedn/Makefile    |    4 +
 drivers/nvme/hw/qedn/qedn.h      |  402 +++++++++
 drivers/nvme/hw/qedn/qedn_conn.c | 1076 ++++++++++++++++++++++++
 drivers/nvme/hw/qedn/qedn_main.c | 1109 ++++++++++++++++++++++++
 drivers/nvme/hw/qedn/qedn_task.c |  873 +++++++++++++++++++
 16 files changed, 5078 insertions(+), 9 deletions(-)
 create mode 100644 drivers/nvme/host/tcp-offload.c
 create mode 100644 drivers/nvme/host/tcp-offload.h
 create mode 100644 drivers/nvme/hw/Kconfig
 create mode 100644 drivers/nvme/hw/Makefile
 create mode 100644 drivers/nvme/hw/qedn/Makefile
 create mode 100644 drivers/nvme/hw/qedn/qedn.h
 create mode 100644 drivers/nvme/hw/qedn/qedn_conn.c
 create mode 100644 drivers/nvme/hw/qedn/qedn_main.c
 create mode 100644 drivers/nvme/hw/qedn/qedn_task.c

-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:23:51 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:23:51 +0300
Subject: [PATCH v3 01/20] nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW
 offload ULP
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-2-smalin@marvell.com>

This patch will present the structure for the NVMeTCP offload common
layer driver. This module is added under "drivers/nvme/host/" and future
offload drivers which will register to it will be placed under
"drivers/nvme/hw".
This new driver will be enabled by the Kconfig "NVM Express over Fabrics
TCP offload commmon layer".
In order to support the new transport type, for host mode, no change is
needed.

Each new offload device specific driver will register to this ULP during
its probe function, by filling out the nvme_tcp_ofld_dev->ops and
nvme_tcp_ofld_dev->private_data and calling nvme_tcp_ofld_register_dev
with the initialized struct.

The internal implementation:
- tcp-offload.h:
  Includes all common structs and ops to be used and shared by offload
  drivers.

- tcp-offload.c:
  Includes the init function which registers as a NVMf transport just
  like any other transport.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 MAINTAINERS                     |   8 ++
 drivers/nvme/host/Kconfig       |  15 +++
 drivers/nvme/host/Makefile      |   3 +
 drivers/nvme/host/tcp-offload.c | 124 ++++++++++++++++++++
 drivers/nvme/host/tcp-offload.h | 199 ++++++++++++++++++++++++++++++++
 5 files changed, 349 insertions(+)
 create mode 100644 drivers/nvme/host/tcp-offload.c
 create mode 100644 drivers/nvme/host/tcp-offload.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 81e1edeceae4..01fbebdc7722 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13093,6 +13093,14 @@ F:	drivers/nvme/host/
 F:	include/linux/nvme.h
 F:	include/uapi/linux/nvme_ioctl.h
 
+NVM EXPRESS TCP OFFLOAD TRANSPORT DRIVERS
+M:	Shai Malin <smalin at marvell.com>
+M:	Ariel Elior <aelior at marvell.com>
+L:	linux-nvme at lists.infradead.org
+S:	Supported
+F:	drivers/nvme/host/tcp-offload.c
+F:	drivers/nvme/host/tcp-offload.h
+
 NVM EXPRESS FC TRANSPORT DRIVERS
 M:	James Smart <james.smart at broadcom.com>
 L:	linux-nvme at lists.infradead.org
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 102292289cdf..1993734d0104 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -84,3 +84,18 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_TCP_OFFLOAD
+	tristate "NVM Express over Fabrics TCP offload common layer"
+	depends on INET
+	depends on BLK_DEV_NVME
+	select NVME_FABRICS
+	help
+	  This provides support for the NVMe over Fabrics protocol using
+	  the TCP offload transport. This allows you to use remote block devices
+	  exported using the NVMe protocol set.
+
+	  To configure a NVMe over Fabrics controller use the nvme-cli tool
+	  from https://github.com/linux-nvme/nvme-cli.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index cbc509784b2e..3c3fdf83ce38 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -8,6 +8,7 @@ obj-$(CONFIG_NVME_FABRICS)		+= nvme-fabrics.o
 obj-$(CONFIG_NVME_RDMA)			+= nvme-rdma.o
 obj-$(CONFIG_NVME_FC)			+= nvme-fc.o
 obj-$(CONFIG_NVME_TCP)			+= nvme-tcp.o
+obj-$(CONFIG_NVME_TCP_OFFLOAD)	+= nvme-tcp-offload.o
 
 nvme-core-y				:= core.o ioctl.o
 nvme-core-$(CONFIG_TRACING)		+= trace.o
@@ -26,3 +27,5 @@ nvme-rdma-y				+= rdma.o
 nvme-fc-y				+= fc.o
 
 nvme-tcp-y				+= tcp.o
+
+nvme-tcp-offload-y		+= tcp-offload.o
diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
new file mode 100644
index 000000000000..10b87f5b875b
--- /dev/null
+++ b/drivers/nvme/host/tcp-offload.c
@@ -0,0 +1,124 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+/* Kernel includes */
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+/* Driver includes */
+#include "tcp-offload.h"
+
+static LIST_HEAD(nvme_tcp_ofld_devices);
+static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
+
+/**
+ * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
+ * function.
+ * @dev:	NVMeTCP offload device instance to be registered to the
+ *		common tcp offload instance.
+ *
+ * API function that registers the type of offload device specific driver
+ * being implemented to the common NVMe over TCP offload library. Part of
+ * the overall init sequence of starting up an offload driver.
+ */
+int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev)
+{
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+
+	if (!ops->claim_dev ||
+	    !ops->setup_ctrl ||
+	    !ops->release_ctrl ||
+	    !ops->create_queue ||
+	    !ops->drain_queue ||
+	    !ops->destroy_queue ||
+	    !ops->poll_queue ||
+	    !ops->send_req)
+		return -EINVAL;
+
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_add_tail(&dev->entry, &nvme_tcp_ofld_devices);
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_register_dev);
+
+/**
+ * nvme_tcp_ofld_unregister_dev() - NVMeTCP Offload Library unregistration
+ * function.
+ * @dev:	NVMeTCP offload device instance to be unregistered from the
+ *		common tcp offload instance.
+ *
+ * API function that unregisters the type of offload device specific driver
+ * being implemented from the common NVMe over TCP offload library.
+ * Part of the overall exit sequence of unloading the implemented driver.
+ */
+void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev)
+{
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_del(&dev->entry);
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
+
+/**
+ * nvme_tcp_ofld_report_queue_err() - NVMeTCP Offload report error event
+ * callback function. Pointed to by nvme_tcp_ofld_queue->report_err.
+ * @queue:	NVMeTCP offload queue instance on which the error has occurred.
+ *
+ * API function that allows the offload device specific driver to reports
+ * errors to the common offload layer, to invoke error recovery.
+ */
+int nvme_tcp_ofld_report_queue_err(struct nvme_tcp_ofld_queue *queue)
+{
+	/* Placeholder - invoke error recovery flow */
+
+	return 0;
+}
+
+/**
+ * nvme_tcp_ofld_req_done() - NVMeTCP Offload request done callback
+ * function. Pointed to by nvme_tcp_ofld_req->done.
+ * Handles both NVME_TCP_F_DATA_SUCCESS flag and NVMe CQ.
+ * @req:	NVMeTCP offload request to complete.
+ * @result:     The nvme_result.
+ * @status:     The completion status.
+ *
+ * API function that allows the offload device specific driver to report
+ * request completions to the common offload layer.
+ */
+void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
+			    union nvme_result *result,
+			    __le16 status)
+{
+	/* Placeholder - complete request with/without error */
+}
+
+static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
+	.name		= "tcp_offload",
+	.module		= THIS_MODULE,
+	.required_opts	= NVMF_OPT_TRADDR,
+	.allowed_opts	= NVMF_OPT_TRSVCID | NVMF_OPT_NR_WRITE_QUEUES  |
+			  NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO |
+			  NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HDR_DIGEST |
+			  NVMF_OPT_DATA_DIGEST | NVMF_OPT_NR_POLL_QUEUES |
+			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE,
+};
+
+static int __init nvme_tcp_ofld_init_module(void)
+{
+	nvmf_register_transport(&nvme_tcp_ofld_transport);
+
+	return 0;
+}
+
+static void __exit nvme_tcp_ofld_cleanup_module(void)
+{
+	nvmf_unregister_transport(&nvme_tcp_ofld_transport);
+}
+
+module_init(nvme_tcp_ofld_init_module);
+module_exit(nvme_tcp_ofld_cleanup_module);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
new file mode 100644
index 000000000000..bfa759177a07
--- /dev/null
+++ b/drivers/nvme/host/tcp-offload.h
@@ -0,0 +1,199 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+/* Linux includes */
+#include <linux/dma-mapping.h>
+#include <linux/scatterlist.h>
+#include <linux/types.h>
+#include <linux/nvme-tcp.h>
+
+/* Driver includes */
+#include "nvme.h"
+#include "fabrics.h"
+
+/* Forward declarations */
+struct nvme_tcp_ofld_ops;
+
+/* Representation of a offload device. This is the struct used to register
+ * to the offload layer by the offload device specific driver, during its probe
+ * function.
+ * Allocated by offload device specific driver.
+ */
+struct nvme_tcp_ofld_dev {
+	struct list_head entry;
+	struct nvme_tcp_ofld_ops *ops;
+
+	/* Offload device specific driver context */
+	int num_hw_vectors;
+};
+
+/* Per IO struct holding the nvme_request and command
+ * Allocated by blk-mq.
+ */
+struct nvme_tcp_ofld_req {
+	struct nvme_request req;
+	struct nvme_command nvme_cmd;
+	struct list_head queue_entry;
+	struct nvme_tcp_ofld_queue *queue;
+
+	/* Offload device specific driver context */
+	void *private_data;
+
+	/* async flag is used to distinguish between async and IO flow
+	 * in common send_req() of nvme_tcp_ofld_ops.
+	 */
+	bool async;
+
+	void (*done)(struct nvme_tcp_ofld_req *req,
+		     union nvme_result *result,
+		     __le16 status);
+};
+
+enum nvme_tcp_ofld_queue_flags {
+	NVME_TCP_OFLD_Q_ALLOCATED = 0,
+	NVME_TCP_OFLD_Q_LIVE = 1,
+};
+
+/* Allocated by nvme_tcp_ofld */
+struct nvme_tcp_ofld_queue {
+	/* Offload device associated to this queue */
+	struct nvme_tcp_ofld_dev *dev;
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	unsigned long flags;
+	size_t cmnd_capsule_len;
+
+	u8 hdr_digest;
+	u8 data_digest;
+	u8 tos;
+
+	/* Offload device specific driver context */
+	void *private_data;
+
+	/* Error callback function */
+	int (*report_err)(struct nvme_tcp_ofld_queue *queue);
+};
+
+/* Connectivity (routing) params used for establishing a connection */
+struct nvme_tcp_ofld_ctrl_con_params {
+	struct sockaddr_storage remote_ip_addr;
+
+	/* If NVMF_OPT_HOST_TRADDR is provided it will be set in local_ip_addr
+	 * in nvme_tcp_ofld_create_ctrl().
+	 * If NVMF_OPT_HOST_TRADDR is not provided the local_ip_addr will be
+	 * initialized by claim_dev().
+	 */
+	struct sockaddr_storage local_ip_addr;
+};
+
+/* Allocated by nvme_tcp_ofld */
+struct nvme_tcp_ofld_ctrl {
+	struct nvme_ctrl nctrl;
+	struct list_head list;
+	struct net_device *ndev;
+	struct nvme_tcp_ofld_dev *dev;
+
+	/* admin and IO queues */
+	struct blk_mq_tag_set tag_set;
+	struct blk_mq_tag_set admin_tag_set;
+	struct nvme_tcp_ofld_queue *queues;
+
+	struct work_struct err_work;
+	struct delayed_work connect_work;
+
+	/*
+	 * Each entry in the array indicates the number of queues of
+	 * corresponding type.
+	 */
+	u32 io_queues[HCTX_MAX_TYPES];
+
+	/* Connectivity params */
+	struct nvme_tcp_ofld_ctrl_con_params conn_params;
+
+	/* Offload device driver context */
+	void *private_data;
+};
+
+struct nvme_tcp_ofld_ops {
+	const char *name;
+	struct module *module;
+
+	/* For offload device specific driver to report what opts it supports.
+	 * It could be different than the ULP supported opts due to hardware
+	 * limitations. Also it could be different among different offload
+	 * specific device drivers.
+	 */
+	int required_opts; /* bitmap using enum nvmf_parsing_opts */
+	int allowed_opts; /* bitmap using enum nvmf_parsing_opts */
+
+	/* For offload device specific max num of segments and IO sizes */
+	u32 max_hw_sectors;
+	u32 max_segments;
+
+	/**
+	 * claim_dev: Return True if addr is reachable via offload device.
+	 * @dev: The offload device to check.
+	 * @ctrl: The offload ctrl have the conn_params field. The
+	 * conn_params is to be filled with routing params by the
+	 * offload device specific driver.
+	 */
+	int (*claim_dev)(struct nvme_tcp_ofld_dev *dev,
+			 struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * setup_ctrl: Setup device specific controller structures.
+	 * @ctrl: The offload ctrl.
+	 */
+	int (*setup_ctrl)(struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * release_ctrl: Release/Free device specific controller structures.
+	 * @ctrl: The offload ctrl.
+	 */
+	int (*release_ctrl)(struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * create_queue: Create offload queue and establish TCP + NVMeTCP
+	 * (icreq+icresp) connection. Return true on successful connection.
+	 * Based on nvme_tcp_alloc_queue.
+	 * @queue: The queue itself - used as input and output.
+	 * @qid: The queue ID associated with the requested queue.
+	 * @q_size: The queue depth.
+	 */
+	int (*create_queue)(struct nvme_tcp_ofld_queue *queue, int qid,
+			    size_t queue_size);
+
+	/**
+	 * drain_queue: Drain a given queue - blocking function call.
+	 * Return from this function ensures that no additional
+	 * completions will arrive on this queue and that the HW will
+	 * not access host memory.
+	 * @queue: The queue to drain.
+	 */
+	void (*drain_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * destroy_queue: Close the TCP + NVMeTCP connection of a given queue
+	 * and make sure its no longer active (no completions will arrive on the
+	 * queue).
+	 * @queue: The queue to destroy.
+	 */
+	void (*destroy_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * poll_queue: Poll a given queue for completions.
+	 * @queue: The queue to poll.
+	 */
+	int (*poll_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * send_req: Dispatch a request. Returns the execution status.
+	 * @req: Ptr to request to be sent.
+	 */
+	int (*send_req)(struct nvme_tcp_ofld_req *req);
+};
+
+/* Exported functions for offload device specific drivers */
+int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
+void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:23:53 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:23:53 +0300
Subject: [PATCH v3 03/20] nvme-fabrics: Expose nvmf_check_required_opts()
 globally
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-4-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

nvmf_check_required_opts() is used to check if user provided opts has
the required_opts or not. if not, it will log which options are not
provided.

It can be leveraged by nvme-tcp-offload to check if provided opts are
supported by this specific offload driver or not.

So expose nvmf_check_required_opts() globally.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/fabrics.c | 5 +++--
 drivers/nvme/host/fabrics.h | 2 ++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 21d67775d2af..7830d2540492 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -864,8 +864,8 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	return ret;
 }
 
-static int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
-		unsigned int required_opts)
+int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
+			     unsigned int required_opts)
 {
 	if ((opts->mask & required_opts) != required_opts) {
 		int i;
@@ -883,6 +883,7 @@ static int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(nvmf_check_required_opts);
 
 bool nvmf_ip_options_match(struct nvme_ctrl *ctrl,
 		struct nvmf_ctrl_options *opts)
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 38ac7b757d78..15d9c15ef8a6 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -197,5 +197,7 @@ int nvmf_get_address(struct nvme_ctrl *ctrl, char *buf, int size);
 bool nvmf_should_reconnect(struct nvme_ctrl *ctrl);
 bool nvmf_ip_options_match(struct nvme_ctrl *ctrl,
 		struct nvmf_ctrl_options *opts);
+int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
+			     unsigned int required_opts);
 
 #endif /* _NVME_FABRICS_H */
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:23:54 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:23:54 +0300
Subject: [PATCH v3 04/20] nvme-tcp-offload: Add device scan implementation
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-5-smalin@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

As part of create_ctrl(), it scans the registered devices and calls
the claim_dev op on each of them, to find the first devices that matches
the connection params. Once the correct devices is found (claim_dev
returns true), we raise the refcnt of that device and return that device
as the device to be used for ctrl currently being created.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/tcp-offload.c | 91 +++++++++++++++++++++++++++++++++
 drivers/nvme/host/tcp-offload.h |  1 +
 2 files changed, 92 insertions(+)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 10b87f5b875b..d0f4b83549b9 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -13,6 +13,11 @@
 static LIST_HEAD(nvme_tcp_ofld_devices);
 static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
 
+static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nc)
+{
+	return container_of(nc, struct nvme_tcp_ofld_ctrl, nctrl);
+}
+
 /**
  * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
  * function.
@@ -96,6 +101,91 @@ void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
 	/* Placeholder - complete request with/without error */
 }
 
+static struct nvme_tcp_ofld_dev *
+nvme_tcp_ofld_lookup_dev(struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+	struct nvme_tcp_ofld_dev *dev;
+	char *iface;
+
+	if (nctrl->opts->mask & NVMF_OPT_HOST_IFACE) {
+		iface = nctrl->opts->host_iface;
+		ctrl->ndev = __dev_get_by_name(&init_net, iface);
+		if (!ctrl->ndev) {
+			pr_err("invalid interface passed: %s\n", iface);
+			return NULL;
+		}
+	}
+
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_for_each_entry(dev, &nvme_tcp_ofld_devices, entry) {
+	/* ctrl includes the destination ip, source ip (if provided) and
+	 * network interface (if provided).
+	 */
+		if (dev->ops->claim_dev(dev, ctrl))
+			goto out;
+	}
+
+	dev = NULL;
+out:
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+
+	return dev;
+}
+
+static struct nvme_ctrl *
+nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	struct nvme_tcp_ofld_dev *dev;
+	struct nvme_ctrl *nctrl;
+	int rc = 0;
+
+	ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
+	if (!ctrl)
+		return ERR_PTR(-ENOMEM);
+
+	nctrl = &ctrl->nctrl;
+
+	/* Init nvme_tcp_ofld_ctrl & nvme_ctrl params based on received opts */
+
+	/* Find device that can reach the dest addr */
+	dev = nvme_tcp_ofld_lookup_dev(ctrl);
+	if (!dev) {
+		pr_info("no device found for addr %s:%s.\n",
+			opts->traddr, opts->trsvcid);
+		rc = -EINVAL;
+		goto out_free_ctrl;
+	}
+
+	/* Increase driver refcnt */
+	if (!try_module_get(dev->ops->module)) {
+		pr_err("try_module_get failed\n");
+		rc = -ENODEV;
+		goto out_free_ctrl;
+	}
+
+	ctrl->dev = dev;
+
+	if (ctrl->dev->ops->max_hw_sectors)
+		nctrl->max_hw_sectors = ctrl->dev->ops->max_hw_sectors;
+	if (ctrl->dev->ops->max_segments)
+		nctrl->max_segments = ctrl->dev->ops->max_segments;
+
+	/* Init queues */
+
+	/* Call nvme_init_ctrl */
+
+	/* Setup ctrl */
+
+	return nctrl;
+
+out_free_ctrl:
+	kfree(ctrl);
+
+	return ERR_PTR(rc);
+}
+
 static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
 	.name		= "tcp_offload",
 	.module		= THIS_MODULE,
@@ -105,6 +195,7 @@ static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
 			  NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HDR_DIGEST |
 			  NVMF_OPT_DATA_DIGEST | NVMF_OPT_NR_POLL_QUEUES |
 			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE,
+	.create_ctrl	= nvme_tcp_ofld_create_ctrl,
 };
 
 static int __init nvme_tcp_ofld_init_module(void)
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index bfa759177a07..d1c2c6171897 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -8,6 +8,7 @@
 #include <linux/scatterlist.h>
 #include <linux/types.h>
 #include <linux/nvme-tcp.h>
+#include <linux/netdevice.h>
 
 /* Driver includes */
 #include "nvme.h"
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:23:55 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:23:55 +0300
Subject: [PATCH v3 05/20] nvme-tcp-offload: Add controller level implementation
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-6-smalin@marvell.com>

From: Arie Gershberg <agershberg at marvell.com>

In this patch we implement controller level functionality including:
- create_ctrl.
- delete_ctrl.
- free_ctrl.

The implementation is similar to other nvme fabrics modules, the main
difference being that the nvme-tcp-offload ULP calls the offload specific
claim_dev() op with the given TCP/IP parameters to determine which device
will be used for this controller.
Once found, the offload specific device and controller will be paired and
kept in a controller list managed by the ULP.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/tcp-offload.c | 487 +++++++++++++++++++++++++++++++-
 1 file changed, 482 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index d0f4b83549b9..806247722a35 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -12,6 +12,10 @@
 
 static LIST_HEAD(nvme_tcp_ofld_devices);
 static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
+static LIST_HEAD(nvme_tcp_ofld_ctrl_list);
+static DEFINE_MUTEX(nvme_tcp_ofld_ctrl_mutex);
+static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops;
+static struct blk_mq_ops nvme_tcp_ofld_mq_ops;
 
 static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nc)
 {
@@ -133,21 +137,441 @@ nvme_tcp_ofld_lookup_dev(struct nvme_tcp_ofld_ctrl *ctrl)
 	return dev;
 }
 
+static struct blk_mq_tag_set *
+nvme_tcp_ofld_alloc_tagset(struct nvme_ctrl *nctrl, bool admin)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct blk_mq_tag_set *set;
+	int rc;
+
+	if (admin) {
+		set = &ctrl->admin_tag_set;
+		memset(set, 0, sizeof(*set));
+		set->ops = &nvme_tcp_ofld_admin_mq_ops;
+		set->queue_depth = NVME_AQ_MQ_TAG_DEPTH;
+		set->reserved_tags = NVMF_RESERVED_TAGS;
+		set->numa_node = nctrl->numa_node;
+		set->flags = BLK_MQ_F_BLOCKING;
+		set->cmd_size = sizeof(struct nvme_tcp_ofld_req);
+		set->driver_data = ctrl;
+		set->nr_hw_queues = 1;
+		set->timeout = NVME_ADMIN_TIMEOUT;
+	} else {
+		set = &ctrl->tag_set;
+		memset(set, 0, sizeof(*set));
+		set->ops = &nvme_tcp_ofld_mq_ops;
+		set->queue_depth = nctrl->sqsize + 1;
+		set->reserved_tags = NVMF_RESERVED_TAGS;
+		set->numa_node = nctrl->numa_node;
+		set->flags = BLK_MQ_F_SHOULD_MERGE;
+		set->cmd_size = sizeof(struct nvme_tcp_ofld_req);
+		set->driver_data = ctrl;
+		set->nr_hw_queues = nctrl->queue_count - 1;
+		set->timeout = NVME_IO_TIMEOUT;
+		set->nr_maps = nctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2;
+	}
+
+	rc = blk_mq_alloc_tag_set(set);
+	if (rc)
+		return ERR_PTR(rc);
+
+	return set;
+}
+
+static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
+					       bool new)
+{
+	int rc;
+
+	/* Placeholder - alloc_admin_queue */
+	if (new) {
+		nctrl->admin_tagset =
+				nvme_tcp_ofld_alloc_tagset(nctrl, true);
+		if (IS_ERR(nctrl->admin_tagset)) {
+			rc = PTR_ERR(nctrl->admin_tagset);
+			nctrl->admin_tagset = NULL;
+			goto out_destroy_queue;
+		}
+
+		nctrl->fabrics_q = blk_mq_init_queue(nctrl->admin_tagset);
+		if (IS_ERR(nctrl->fabrics_q)) {
+			rc = PTR_ERR(nctrl->fabrics_q);
+			nctrl->fabrics_q = NULL;
+			goto out_free_tagset;
+		}
+
+		nctrl->admin_q = blk_mq_init_queue(nctrl->admin_tagset);
+		if (IS_ERR(nctrl->admin_q)) {
+			rc = PTR_ERR(nctrl->admin_q);
+			nctrl->admin_q = NULL;
+			goto out_cleanup_fabrics_q;
+		}
+	}
+
+	/* Placeholder - nvme_tcp_ofld_start_queue */
+
+	rc = nvme_enable_ctrl(nctrl);
+	if (rc)
+		goto out_stop_queue;
+
+	blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	rc = nvme_init_ctrl_finish(nctrl);
+	if (rc)
+		goto out_quiesce_queue;
+
+	return 0;
+
+out_quiesce_queue:
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+
+out_stop_queue:
+	/* Placeholder - stop offload queue */
+	nvme_cancel_admin_tagset(nctrl);
+
+out_cleanup_fabrics_q:
+	if (new)
+		blk_cleanup_queue(nctrl->fabrics_q);
+out_free_tagset:
+	if (new)
+		blk_mq_free_tag_set(nctrl->admin_tagset);
+out_destroy_queue:
+	/* Placeholder - free admin queue */
+
+	return rc;
+}
+
+static int
+nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
+{
+	int rc;
+
+	/* Placeholder - alloc_io_queues */
+
+	if (new) {
+		nctrl->tagset = nvme_tcp_ofld_alloc_tagset(nctrl, false);
+		if (IS_ERR(nctrl->tagset)) {
+			rc = PTR_ERR(nctrl->tagset);
+			nctrl->tagset = NULL;
+			goto out_free_io_queues;
+		}
+
+		nctrl->connect_q = blk_mq_init_queue(nctrl->tagset);
+		if (IS_ERR(nctrl->connect_q)) {
+			rc = PTR_ERR(nctrl->connect_q);
+			nctrl->connect_q = NULL;
+			goto out_free_tag_set;
+		}
+	}
+
+	/* Placeholder - start_io_queues */
+
+	if (!new) {
+		nvme_start_queues(nctrl);
+		if (!nvme_wait_freeze_timeout(nctrl, NVME_IO_TIMEOUT)) {
+			/*
+			 * If we timed out waiting for freeze we are likely to
+			 * be stuck.  Fail the controller initialization just
+			 * to be safe.
+			 */
+			rc = -ENODEV;
+			goto out_wait_freeze_timed_out;
+		}
+		blk_mq_update_nr_hw_queues(nctrl->tagset,
+					   nctrl->queue_count - 1);
+		nvme_unfreeze(nctrl);
+	}
+
+	return 0;
+
+out_wait_freeze_timed_out:
+	nvme_stop_queues(nctrl);
+	nvme_sync_io_queues(nctrl);
+
+	/* Placeholder - Stop IO queues */
+
+	if (new)
+		blk_cleanup_queue(nctrl->connect_q);
+out_free_tag_set:
+	if (new)
+		blk_mq_free_tag_set(nctrl->tagset);
+out_free_io_queues:
+	/* Placeholder - free_io_queues */
+
+	return rc;
+}
+
+static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvmf_ctrl_options *opts = nctrl->opts;
+	int rc = 0;
+
+	rc = ctrl->dev->ops->setup_ctrl(ctrl);
+	if (rc)
+		return rc;
+
+	rc = nvme_tcp_ofld_configure_admin_queue(nctrl, new);
+	if (rc)
+		goto out_release_ctrl;
+
+	if (nctrl->icdoff) {
+		dev_err(nctrl->device, "icdoff is not supported!\n");
+		rc = -EOPNOTSUPP;
+		goto destroy_admin;
+	}
+
+	if (!nvme_ctrl_sgl_supported(nctrl)) {
+		dev_err(nctrl->device, "Mandatory sgls are not supported!\n");
+		rc = -EOPNOTSUPP;
+		goto destroy_admin;
+	}
+
+	if (opts->queue_size > nctrl->sqsize + 1)
+		dev_warn(nctrl->device,
+			 "queue_size %zu > ctrl sqsize %u, clamping down\n",
+			 opts->queue_size, nctrl->sqsize + 1);
+
+	if (nctrl->sqsize + 1 > nctrl->maxcmd) {
+		dev_warn(nctrl->device,
+			 "sqsize %u > ctrl maxcmd %u, clamping down\n",
+			 nctrl->sqsize + 1, nctrl->maxcmd);
+		nctrl->sqsize = nctrl->maxcmd - 1;
+	}
+
+	if (nctrl->queue_count > 1) {
+		rc = nvme_tcp_ofld_configure_io_queues(nctrl, new);
+		if (rc)
+			goto destroy_admin;
+	}
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_LIVE)) {
+		/*
+		 * state change failure is ok if we started ctrl delete,
+		 * unless we're during creation of a new controller to
+		 * avoid races with teardown flow.
+		 */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+		WARN_ON_ONCE(new);
+		rc = -EINVAL;
+		goto destroy_io;
+	}
+
+	nvme_start_ctrl(nctrl);
+
+	return 0;
+
+destroy_io:
+	/* Placeholder - stop and destroy io queues*/
+destroy_admin:
+	/* Placeholder - stop and destroy admin queue*/
+out_release_ctrl:
+	ctrl->dev->ops->release_ctrl(ctrl);
+
+	return rc;
+}
+
+static int
+nvme_tcp_ofld_check_dev_opts(struct nvmf_ctrl_options *opts,
+			     struct nvme_tcp_ofld_ops *ofld_ops)
+{
+	unsigned int nvme_tcp_ofld_opt_mask = NVMF_ALLOWED_OPTS |
+			ofld_ops->allowed_opts | ofld_ops->required_opts;
+	struct nvmf_ctrl_options dev_opts_mask;
+
+	if (opts->mask & ~nvme_tcp_ofld_opt_mask) {
+		pr_warn("One or more nvmf options missing from ofld drvr %s.\n",
+			ofld_ops->name);
+
+		dev_opts_mask.mask = nvme_tcp_ofld_opt_mask;
+
+		return nvmf_check_required_opts(&dev_opts_mask, opts->mask);
+	}
+
+	return 0;
+}
+
+static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+
+	if (list_empty(&ctrl->list))
+		goto free_ctrl;
+
+	ctrl->dev->ops->release_ctrl(ctrl);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_del(&ctrl->list);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+
+	nvmf_free_options(nctrl->opts);
+free_ctrl:
+	module_put(dev->ops->module);
+	kfree(ctrl->queues);
+	kfree(ctrl);
+}
+
+static void
+nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove)
+{
+	/* Placeholder - teardown_admin_queue */
+}
+
+static void
+nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
+{
+	/* Placeholder - teardown_io_queues */
+}
+
+static void
+nvme_tcp_ofld_teardown_ctrl(struct nvme_ctrl *nctrl, bool shutdown)
+{
+	/* Placeholder - err_work and connect_work */
+	nvme_tcp_ofld_teardown_io_queues(nctrl, shutdown);
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	if (shutdown)
+		nvme_shutdown_ctrl(nctrl);
+	else
+		nvme_disable_ctrl(nctrl);
+	nvme_tcp_ofld_teardown_admin_queue(nctrl, shutdown);
+}
+
+static void nvme_tcp_ofld_delete_ctrl(struct nvme_ctrl *nctrl)
+{
+	nvme_tcp_ofld_teardown_ctrl(nctrl, true);
+}
+
+static int
+nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
+			   struct request *rq,
+			   unsigned int hctx_idx,
+			   unsigned int numa_node)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+
+	/* Placeholder - init request */
+
+	req->done = nvme_tcp_ofld_req_done;
+
+	return 0;
+}
+
+static blk_status_t
+nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
+		       const struct blk_mq_queue_data *bd)
+{
+	/* Call nvme_setup_cmd(...) */
+
+	/* Call ops->send_req(...) */
+
+	return BLK_STS_OK;
+}
+
+static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
+	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.init_request	= nvme_tcp_ofld_init_request,
+	/*
+	 * All additional ops will be also implemented and registered similar to
+	 * tcp.c
+	 */
+};
+
+static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
+	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.init_request	= nvme_tcp_ofld_init_request,
+	/*
+	 * All additional ops will be also implemented and registered similar to
+	 * tcp.c
+	 */
+};
+
+static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
+	.name			= "tcp_offload",
+	.module			= THIS_MODULE,
+	.flags			= NVME_F_FABRICS,
+	.reg_read32		= nvmf_reg_read32,
+	.reg_read64		= nvmf_reg_read64,
+	.reg_write32		= nvmf_reg_write32,
+	.free_ctrl		= nvme_tcp_ofld_free_ctrl,
+	.delete_ctrl		= nvme_tcp_ofld_delete_ctrl,
+	.get_address		= nvmf_get_address,
+};
+
+static bool
+nvme_tcp_ofld_existing_controller(struct nvmf_ctrl_options *opts)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	bool found = false;
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_for_each_entry(ctrl, &nvme_tcp_ofld_ctrl_list, list) {
+		found = nvmf_ip_options_match(&ctrl->nctrl, opts);
+		if (found)
+			break;
+	}
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+
+	return found;
+}
+
 static struct nvme_ctrl *
 nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 {
+	struct nvme_tcp_ofld_queue *queue;
 	struct nvme_tcp_ofld_ctrl *ctrl;
 	struct nvme_tcp_ofld_dev *dev;
 	struct nvme_ctrl *nctrl;
-	int rc = 0;
+	int i, rc = 0;
 
 	ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
 	if (!ctrl)
 		return ERR_PTR(-ENOMEM);
 
+	INIT_LIST_HEAD(&ctrl->list);
 	nctrl = &ctrl->nctrl;
+	nctrl->opts = opts;
+	nctrl->queue_count = opts->nr_io_queues + opts->nr_write_queues +
+			     opts->nr_poll_queues + 1;
+	nctrl->sqsize = opts->queue_size - 1;
+	nctrl->kato = opts->kato;
+	if (!(opts->mask & NVMF_OPT_TRSVCID)) {
+		opts->trsvcid =
+			kstrdup(__stringify(NVME_TCP_DISC_PORT), GFP_KERNEL);
+		if (!opts->trsvcid) {
+			rc = -ENOMEM;
+			goto out_free_ctrl;
+		}
+		opts->mask |= NVMF_OPT_TRSVCID;
+	}
+
+	rc = inet_pton_with_scope(&init_net, AF_UNSPEC, opts->traddr,
+				  opts->trsvcid,
+				  &ctrl->conn_params.remote_ip_addr);
+	if (rc) {
+		pr_err("malformed address passed: %s:%s\n",
+		       opts->traddr, opts->trsvcid);
+		goto out_free_ctrl;
+	}
 
-	/* Init nvme_tcp_ofld_ctrl & nvme_ctrl params based on received opts */
+	if (opts->mask & NVMF_OPT_HOST_TRADDR) {
+		rc = inet_pton_with_scope(&init_net, AF_UNSPEC,
+					  opts->host_traddr, NULL,
+					  &ctrl->conn_params.local_ip_addr);
+		if (rc) {
+			pr_err("malformed src address passed: %s\n",
+			       opts->host_traddr);
+			goto out_free_ctrl;
+		}
+	}
+
+	if (!opts->duplicate_connect &&
+	    nvme_tcp_ofld_existing_controller(opts)) {
+		rc = -EALREADY;
+		goto out_free_ctrl;
+	}
 
 	/* Find device that can reach the dest addr */
 	dev = nvme_tcp_ofld_lookup_dev(ctrl);
@@ -165,6 +589,10 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 		goto out_free_ctrl;
 	}
 
+	rc = nvme_tcp_ofld_check_dev_opts(opts, dev->ops);
+	if (rc)
+		goto out_module_put;
+
 	ctrl->dev = dev;
 
 	if (ctrl->dev->ops->max_hw_sectors)
@@ -172,14 +600,55 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 	if (ctrl->dev->ops->max_segments)
 		nctrl->max_segments = ctrl->dev->ops->max_segments;
 
-	/* Init queues */
+	ctrl->queues = kcalloc(nctrl->queue_count,
+			       sizeof(struct nvme_tcp_ofld_queue),
+			       GFP_KERNEL);
+	if (!ctrl->queues) {
+		rc = -ENOMEM;
+		goto out_module_put;
+	}
+
+	for (i = 0; i < nctrl->queue_count; ++i) {
+		queue = &ctrl->queues[i];
+		queue->ctrl = ctrl;
+		queue->dev = dev;
+		queue->report_err = nvme_tcp_ofld_report_queue_err;
+	}
+
+	rc = nvme_init_ctrl(nctrl, ndev, &nvme_tcp_ofld_ctrl_ops, 0);
+	if (rc)
+		goto out_free_queues;
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		WARN_ON_ONCE(1);
+		rc = -EINTR;
+		goto out_uninit_ctrl;
+	}
 
-	/* Call nvme_init_ctrl */
+	rc = nvme_tcp_ofld_setup_ctrl(nctrl, true);
+	if (rc)
+		goto out_uninit_ctrl;
 
-	/* Setup ctrl */
+	dev_info(nctrl->device, "new ctrl: NQN \"%s\", addr %pISp\n",
+		 opts->subsysnqn, &ctrl->conn_params.remote_ip_addr);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_add_tail(&ctrl->list, &nvme_tcp_ofld_ctrl_list);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
 
 	return nctrl;
 
+out_uninit_ctrl:
+	nvme_uninit_ctrl(nctrl);
+	nvme_put_ctrl(nctrl);
+	if (rc > 0)
+		rc = -EIO;
+
+	return ERR_PTR(rc);
+out_free_queues:
+	kfree(ctrl->queues);
+out_module_put:
+	module_put(dev->ops->module);
 out_free_ctrl:
 	kfree(ctrl);
 
@@ -207,7 +676,15 @@ static int __init nvme_tcp_ofld_init_module(void)
 
 static void __exit nvme_tcp_ofld_cleanup_module(void)
 {
+	struct nvme_tcp_ofld_ctrl *ctrl;
+
 	nvmf_unregister_transport(&nvme_tcp_ofld_transport);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_for_each_entry(ctrl, &nvme_tcp_ofld_ctrl_list, list)
+		nvme_delete_ctrl(&ctrl->nctrl);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+	flush_workqueue(nvme_delete_wq);
 }
 
 module_init(nvme_tcp_ofld_init_module);
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:23:52 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:23:52 +0300
Subject: [PATCH v3 02/20] nvme-fabrics: Move NVMF_ALLOWED_OPTS and
 NVMF_REQUIRED_OPTS definitions
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-3-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

Move NVMF_ALLOWED_OPTS and NVMF_REQUIRED_OPTS definitions
to header file, so it can be used by the different HW devices.

NVMeTCP offload devices might have different limitations of the
allowed options, for example, a device that does not support all the
queue types. With tcp and rdma, only the nvme-tcp and nvme-rdma layers
handle those attributes and the HW devices do not create any limitations
for the allowed options.

An alternative design could be to add separate fields in
nvme_tcp_ofld_ops such as max_hw_sectors and max_segments that
we already have in this series.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Acked-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/fabrics.c | 7 -------
 drivers/nvme/host/fabrics.h | 7 +++++++
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 1e6a7cc056ca..21d67775d2af 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -947,13 +947,6 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
 
-#define NVMF_REQUIRED_OPTS	(NVMF_OPT_TRANSPORT | NVMF_OPT_NQN)
-#define NVMF_ALLOWED_OPTS	(NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \
-				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
-				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
-				 NVMF_OPT_DISABLE_SQFLOW |\
-				 NVMF_OPT_FAIL_FAST_TMO)
-
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
 {
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index c31dad69a773..38ac7b757d78 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -69,6 +69,13 @@ enum {
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 };
 
+#define NVMF_REQUIRED_OPTS	(NVMF_OPT_TRANSPORT | NVMF_OPT_NQN)
+#define NVMF_ALLOWED_OPTS	(NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \
+				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
+				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
+				 NVMF_OPT_DISABLE_SQFLOW |\
+				 NVMF_OPT_FAIL_FAST_TMO)
+
 /**
  * struct nvmf_ctrl_options - Used to hold the options specified
  *			      with the parsing opts enum.
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:23:56 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:23:56 +0300
Subject: [PATCH v3 06/20] nvme-tcp-offload: Add controller level error
 recovery implementation
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-7-smalin@marvell.com>

From: Arie Gershberg <agershberg at marvell.com>

In this patch, we implement controller level error handling and recovery.
Upon an error discovered by the ULP or reset controller initiated by the
nvme-core (using reset_ctrl workqueue), the ULP will initiate a controller
recovery which includes teardown and re-connect of all queues.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 drivers/nvme/host/tcp-offload.c | 127 +++++++++++++++++++++++++++++++-
 drivers/nvme/host/tcp-offload.h |   1 +
 2 files changed, 126 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 806247722a35..3af89a938d1b 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -72,6 +72,23 @@ void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev)
 }
 EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
 
+/**
+ * nvme_tcp_ofld_error_recovery() - NVMeTCP Offload library error recovery.
+ * function.
+ * @nctrl:	NVMe controller instance to change to resetting.
+ *
+ * API function that change the controller state to resseting.
+ * Part of the overall controller reset sequence.
+ */
+void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl)
+{
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_RESETTING))
+		return;
+
+	queue_work(nvme_reset_wq, &to_tcp_ofld_ctrl(nctrl)->err_work);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_error_recovery);
+
 /**
  * nvme_tcp_ofld_report_queue_err() - NVMeTCP Offload report error event
  * callback function. Pointed to by nvme_tcp_ofld_queue->report_err.
@@ -82,7 +99,8 @@ EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
  */
 int nvme_tcp_ofld_report_queue_err(struct nvme_tcp_ofld_queue *queue)
 {
-	/* Placeholder - invoke error recovery flow */
+	pr_err("nvme-tcp-offload queue error\n");
+	nvme_tcp_ofld_error_recovery(&queue->ctrl->nctrl);
 
 	return 0;
 }
@@ -302,6 +320,28 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 	return rc;
 }
 
+static void nvme_tcp_ofld_reconnect_or_remove(struct nvme_ctrl *nctrl)
+{
+	/* If we are resetting/deleting then do nothing */
+	if (nctrl->state != NVME_CTRL_CONNECTING) {
+		WARN_ON_ONCE(nctrl->state == NVME_CTRL_NEW ||
+			     nctrl->state == NVME_CTRL_LIVE);
+
+		return;
+	}
+
+	if (nvmf_should_reconnect(nctrl)) {
+		dev_info(nctrl->device, "Reconnecting in %d seconds...\n",
+			 nctrl->opts->reconnect_delay);
+		queue_delayed_work(nvme_wq,
+				   &to_tcp_ofld_ctrl(nctrl)->connect_work,
+				   nctrl->opts->reconnect_delay * HZ);
+	} else {
+		dev_info(nctrl->device, "Removing controller...\n");
+		nvme_delete_ctrl(nctrl);
+	}
+}
+
 static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 {
 	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
@@ -426,10 +466,63 @@ nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
 	/* Placeholder - teardown_io_queues */
 }
 
+static void nvme_tcp_ofld_reconnect_ctrl_work(struct work_struct *work)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl =
+				container_of(to_delayed_work(work),
+					     struct nvme_tcp_ofld_ctrl,
+					     connect_work);
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+
+	++nctrl->nr_reconnects;
+
+	if (nvme_tcp_ofld_setup_ctrl(nctrl, false))
+		goto requeue;
+
+	dev_info(nctrl->device, "Successfully reconnected (%d attempt)\n",
+		 nctrl->nr_reconnects);
+
+	nctrl->nr_reconnects = 0;
+
+	return;
+
+requeue:
+	dev_info(nctrl->device, "Failed reconnect attempt %d\n",
+		 nctrl->nr_reconnects);
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
+static void nvme_tcp_ofld_error_recovery_work(struct work_struct *work)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl =
+		container_of(work, struct nvme_tcp_ofld_ctrl, err_work);
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+
+	nvme_stop_keep_alive(nctrl);
+	nvme_tcp_ofld_teardown_io_queues(nctrl, false);
+	/* unquiesce to fail fast pending requests */
+	nvme_start_queues(nctrl);
+	nvme_tcp_ofld_teardown_admin_queue(nctrl, false);
+	blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		/* state change failure is ok if we started nctrl delete */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+
+		return;
+	}
+
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
 static void
 nvme_tcp_ofld_teardown_ctrl(struct nvme_ctrl *nctrl, bool shutdown)
 {
-	/* Placeholder - err_work and connect_work */
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+
+	cancel_work_sync(&ctrl->err_work);
+	cancel_delayed_work_sync(&ctrl->connect_work);
 	nvme_tcp_ofld_teardown_io_queues(nctrl, shutdown);
 	blk_mq_quiesce_queue(nctrl->admin_q);
 	if (shutdown)
@@ -444,6 +537,32 @@ static void nvme_tcp_ofld_delete_ctrl(struct nvme_ctrl *nctrl)
 	nvme_tcp_ofld_teardown_ctrl(nctrl, true);
 }
 
+static void nvme_tcp_ofld_reset_ctrl_work(struct work_struct *work)
+{
+	struct nvme_ctrl *nctrl =
+		container_of(work, struct nvme_ctrl, reset_work);
+
+	nvme_stop_ctrl(nctrl);
+	nvme_tcp_ofld_teardown_ctrl(nctrl, false);
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		/* state change failure is ok if we started ctrl delete */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+
+		return;
+	}
+
+	if (nvme_tcp_ofld_setup_ctrl(nctrl, false))
+		goto out_fail;
+
+	return;
+
+out_fail:
+	++nctrl->nr_reconnects;
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
 static int
 nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 			   struct request *rq,
@@ -537,6 +656,10 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 			     opts->nr_poll_queues + 1;
 	nctrl->sqsize = opts->queue_size - 1;
 	nctrl->kato = opts->kato;
+	INIT_DELAYED_WORK(&ctrl->connect_work,
+			  nvme_tcp_ofld_reconnect_ctrl_work);
+	INIT_WORK(&ctrl->err_work, nvme_tcp_ofld_error_recovery_work);
+	INIT_WORK(&nctrl->reset_work, nvme_tcp_ofld_reset_ctrl_work);
 	if (!(opts->mask & NVMF_OPT_TRSVCID)) {
 		opts->trsvcid =
 			kstrdup(__stringify(NVME_TCP_DISC_PORT), GFP_KERNEL);
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index d1c2c6171897..51fec632b72b 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -198,3 +198,4 @@ struct nvme_tcp_ofld_ops {
 /* Exported functions for offload device specific drivers */
 int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
+void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl);
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:23:57 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:23:57 +0300
Subject: [PATCH v3 07/20] nvme-tcp-offload: Add queue level implementation
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-8-smalin@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

In this patch we implement queue level functionality.
The implementation is similar to the nvme-tcp module, the main
difference being that we call the offload device specific create_queue
op which creates the TCP connection, and NVMeTPC connection including
icreq+icresp negotiation.
Once create_queue returns successfully, we can move on to the fabrics
connect.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/tcp-offload.c | 420 +++++++++++++++++++++++++++++---
 drivers/nvme/host/tcp-offload.h |   4 +
 2 files changed, 396 insertions(+), 28 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 3af89a938d1b..26253b107db2 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -22,6 +22,11 @@ static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nc)
 	return container_of(nc, struct nvme_tcp_ofld_ctrl, nctrl);
 }
 
+static inline int nvme_tcp_ofld_qid(struct nvme_tcp_ofld_queue *queue)
+{
+	return queue - queue->ctrl->queues;
+}
+
 /**
  * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
  * function.
@@ -196,19 +201,127 @@ nvme_tcp_ofld_alloc_tagset(struct nvme_ctrl *nctrl, bool admin)
 	return set;
 }
 
+static void __nvme_tcp_ofld_stop_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	queue->dev->ops->drain_queue(queue);
+}
+
+static void nvme_tcp_ofld_stop_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+
+	mutex_lock(&queue->queue_lock);
+	if (test_and_clear_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags))
+		__nvme_tcp_ofld_stop_queue(queue);
+	mutex_unlock(&queue->queue_lock);
+}
+
+static void nvme_tcp_ofld_stop_io_queues(struct nvme_ctrl *ctrl)
+{
+	int i;
+
+	for (i = 1; i < ctrl->queue_count; i++)
+		nvme_tcp_ofld_stop_queue(ctrl, i);
+}
+
+static void __nvme_tcp_ofld_free_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	queue->dev->ops->destroy_queue(queue);
+}
+
+static void nvme_tcp_ofld_free_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+
+	if (test_and_clear_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags)) {
+		__nvme_tcp_ofld_free_queue(queue);
+		mutex_destroy(&queue->queue_lock);
+	}
+}
+
+static void
+nvme_tcp_ofld_free_io_queues(struct nvme_ctrl *nctrl)
+{
+	int i;
+
+	for (i = 1; i < nctrl->queue_count; i++)
+		nvme_tcp_ofld_free_queue(nctrl, i);
+}
+
+static void nvme_tcp_ofld_destroy_io_queues(struct nvme_ctrl *nctrl,
+					    bool remove)
+{
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+	if (remove) {
+		blk_cleanup_queue(nctrl->connect_q);
+		blk_mq_free_tag_set(nctrl->tagset);
+	}
+	nvme_tcp_ofld_free_io_queues(nctrl);
+}
+
+static void nvme_tcp_ofld_destroy_admin_queue(struct nvme_ctrl *nctrl,
+					      bool remove)
+{
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	if (remove) {
+		blk_cleanup_queue(nctrl->admin_q);
+		blk_cleanup_queue(nctrl->fabrics_q);
+		blk_mq_free_tag_set(nctrl->admin_tagset);
+	}
+	nvme_tcp_ofld_free_queue(nctrl, 0);
+}
+
+static int nvme_tcp_ofld_start_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+	int rc;
+
+	queue = &ctrl->queues[qid];
+	if (qid) {
+		queue->cmnd_capsule_len = nctrl->ioccsz * 16;
+		rc = nvmf_connect_io_queue(nctrl, qid, false);
+	} else {
+		queue->cmnd_capsule_len = sizeof(struct nvme_command) +
+						NVME_TCP_ADMIN_CCSZ;
+		rc = nvmf_connect_admin_queue(nctrl);
+	}
+
+	if (!rc) {
+		set_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags);
+	} else {
+		if (test_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags))
+			__nvme_tcp_ofld_stop_queue(queue);
+		dev_err(nctrl->device,
+			"failed to connect queue: %d ret=%d\n", qid, rc);
+	}
+
+	return rc;
+}
+
 static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 					       bool new)
 {
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[0];
 	int rc;
 
-	/* Placeholder - alloc_admin_queue */
+	mutex_init(&queue->queue_lock);
+
+	rc = ctrl->dev->ops->create_queue(queue, 0, NVME_AQ_DEPTH);
+	if (rc)
+		return rc;
+
+	set_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags);
 	if (new) {
 		nctrl->admin_tagset =
 				nvme_tcp_ofld_alloc_tagset(nctrl, true);
 		if (IS_ERR(nctrl->admin_tagset)) {
 			rc = PTR_ERR(nctrl->admin_tagset);
 			nctrl->admin_tagset = NULL;
-			goto out_destroy_queue;
+			goto out_free_queue;
 		}
 
 		nctrl->fabrics_q = blk_mq_init_queue(nctrl->admin_tagset);
@@ -226,7 +339,9 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 		}
 	}
 
-	/* Placeholder - nvme_tcp_ofld_start_queue */
+	rc = nvme_tcp_ofld_start_queue(nctrl, 0);
+	if (rc)
+		goto out_cleanup_queue;
 
 	rc = nvme_enable_ctrl(nctrl);
 	if (rc)
@@ -243,19 +358,143 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 out_quiesce_queue:
 	blk_mq_quiesce_queue(nctrl->admin_q);
 	blk_sync_queue(nctrl->admin_q);
-
 out_stop_queue:
-	/* Placeholder - stop offload queue */
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
 	nvme_cancel_admin_tagset(nctrl);
-
+out_cleanup_queue:
+	if (new)
+		blk_cleanup_queue(nctrl->admin_q);
 out_cleanup_fabrics_q:
 	if (new)
 		blk_cleanup_queue(nctrl->fabrics_q);
 out_free_tagset:
 	if (new)
 		blk_mq_free_tag_set(nctrl->admin_tagset);
-out_destroy_queue:
-	/* Placeholder - free admin queue */
+out_free_queue:
+	nvme_tcp_ofld_free_queue(nctrl, 0);
+
+	return rc;
+}
+
+static unsigned int nvme_tcp_ofld_nr_io_queues(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+	u32 hw_vectors = dev->num_hw_vectors;
+	u32 nr_write_queues, nr_poll_queues;
+	u32 nr_io_queues, nr_total_queues;
+
+	nr_io_queues = min3(nctrl->opts->nr_io_queues, num_online_cpus(),
+			    hw_vectors);
+	nr_write_queues = min3(nctrl->opts->nr_write_queues, num_online_cpus(),
+			       hw_vectors);
+	nr_poll_queues = min3(nctrl->opts->nr_poll_queues, num_online_cpus(),
+			      hw_vectors);
+
+	nr_total_queues = nr_io_queues + nr_write_queues + nr_poll_queues;
+
+	return nr_total_queues;
+}
+
+static void
+nvme_tcp_ofld_set_io_queues(struct nvme_ctrl *nctrl, unsigned int nr_io_queues)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvmf_ctrl_options *opts = nctrl->opts;
+
+	if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) {
+		/*
+		 * separate read/write queues
+		 * hand out dedicated default queues only after we have
+		 * sufficient read queues.
+		 */
+		ctrl->io_queues[HCTX_TYPE_READ] = opts->nr_io_queues;
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_READ];
+		ctrl->io_queues[HCTX_TYPE_DEFAULT] =
+			min(opts->nr_write_queues, nr_io_queues);
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	} else {
+		/*
+		 * shared read/write queues
+		 * either no write queues were requested, or we don't have
+		 * sufficient queue count to have dedicated default queues.
+		 */
+		ctrl->io_queues[HCTX_TYPE_DEFAULT] =
+			min(opts->nr_io_queues, nr_io_queues);
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	}
+
+	if (opts->nr_poll_queues && nr_io_queues) {
+		/* map dedicated poll queues only if we have queues left */
+		ctrl->io_queues[HCTX_TYPE_POLL] =
+			min(opts->nr_poll_queues, nr_io_queues);
+	}
+}
+
+static int nvme_tcp_ofld_create_io_queues(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	int i, rc;
+
+	for (i = 1; i < nctrl->queue_count; i++) {
+		mutex_init(&ctrl->queues[i].queue_lock);
+
+		rc = ctrl->dev->ops->create_queue(&ctrl->queues[i],
+						  i, nctrl->sqsize + 1);
+		if (rc)
+			goto out_free_queues;
+
+		set_bit(NVME_TCP_OFLD_Q_ALLOCATED, &ctrl->queues[i].flags);
+	}
+
+	return 0;
+
+out_free_queues:
+	for (i--; i >= 1; i--)
+		nvme_tcp_ofld_free_queue(nctrl, i);
+
+	return rc;
+}
+
+static int nvme_tcp_ofld_alloc_io_queues(struct nvme_ctrl *nctrl)
+{
+	unsigned int nr_io_queues;
+	int rc;
+
+	nr_io_queues = nvme_tcp_ofld_nr_io_queues(nctrl);
+	rc = nvme_set_queue_count(nctrl, &nr_io_queues);
+	if (rc)
+		return rc;
+
+	nctrl->queue_count = nr_io_queues + 1;
+	if (nctrl->queue_count < 2) {
+		dev_err(nctrl->device,
+			"unable to set any I/O queues\n");
+
+		return -ENOMEM;
+	}
+
+	dev_info(nctrl->device, "creating %d I/O queues.\n", nr_io_queues);
+	nvme_tcp_ofld_set_io_queues(nctrl, nr_io_queues);
+
+	return nvme_tcp_ofld_create_io_queues(nctrl);
+}
+
+static int nvme_tcp_ofld_start_io_queues(struct nvme_ctrl *nctrl)
+{
+	int i, rc = 0;
+
+	for (i = 1; i < nctrl->queue_count; i++) {
+		rc = nvme_tcp_ofld_start_queue(nctrl, i);
+		if (rc)
+			goto out_stop_queues;
+	}
+
+	return 0;
+
+out_stop_queues:
+	for (i--; i >= 1; i--)
+		nvme_tcp_ofld_stop_queue(nctrl, i);
 
 	return rc;
 }
@@ -263,9 +502,10 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 static int
 nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 {
-	int rc;
+	int rc = nvme_tcp_ofld_alloc_io_queues(nctrl);
 
-	/* Placeholder - alloc_io_queues */
+	if (rc)
+		return rc;
 
 	if (new) {
 		nctrl->tagset = nvme_tcp_ofld_alloc_tagset(nctrl, false);
@@ -283,7 +523,9 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 		}
 	}
 
-	/* Placeholder - start_io_queues */
+	rc = nvme_tcp_ofld_start_io_queues(nctrl);
+	if (rc)
+		goto out_cleanup_connect_q;
 
 	if (!new) {
 		nvme_start_queues(nctrl);
@@ -306,16 +548,16 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 out_wait_freeze_timed_out:
 	nvme_stop_queues(nctrl);
 	nvme_sync_io_queues(nctrl);
-
-	/* Placeholder - Stop IO queues */
-
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+out_cleanup_connect_q:
+	nvme_cancel_tagset(nctrl);
 	if (new)
 		blk_cleanup_queue(nctrl->connect_q);
 out_free_tag_set:
 	if (new)
 		blk_mq_free_tag_set(nctrl->tagset);
 out_free_io_queues:
-	/* Placeholder - free_io_queues */
+	nvme_tcp_ofld_free_io_queues(nctrl);
 
 	return rc;
 }
@@ -342,6 +584,17 @@ static void nvme_tcp_ofld_reconnect_or_remove(struct nvme_ctrl *nctrl)
 	}
 }
 
+static int
+nvme_tcp_ofld_init_admin_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			      unsigned int hctx_idx)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = data;
+
+	hctx->driver_data = &ctrl->queues[0];
+
+	return 0;
+}
+
 static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 {
 	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
@@ -404,9 +657,19 @@ static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 	return 0;
 
 destroy_io:
-	/* Placeholder - stop and destroy io queues*/
+	if (nctrl->queue_count > 1) {
+		nvme_stop_queues(nctrl);
+		nvme_sync_io_queues(nctrl);
+		nvme_tcp_ofld_stop_io_queues(nctrl);
+		nvme_cancel_tagset(nctrl);
+		nvme_tcp_ofld_destroy_io_queues(nctrl, new);
+	}
 destroy_admin:
-	/* Placeholder - stop and destroy admin queue*/
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	nvme_cancel_admin_tagset(nctrl);
+	nvme_tcp_ofld_destroy_admin_queue(nctrl, new);
 out_release_ctrl:
 	ctrl->dev->ops->release_ctrl(ctrl);
 
@@ -455,15 +718,37 @@ static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
 }
 
 static void
-nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove)
+nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *nctrl, bool remove)
 {
-	/* Placeholder - teardown_admin_queue */
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	nvme_cancel_admin_tagset(nctrl);
+
+	if (remove)
+		blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	nvme_tcp_ofld_destroy_admin_queue(nctrl, remove);
 }
 
 static void
 nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
 {
-	/* Placeholder - teardown_io_queues */
+	if (nctrl->queue_count <= 1)
+		return;
+
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	nvme_start_freeze(nctrl);
+	nvme_stop_queues(nctrl);
+	nvme_sync_io_queues(nctrl);
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+	nvme_cancel_tagset(nctrl);
+
+	if (remove)
+		nvme_start_queues(nctrl);
+
+	nvme_tcp_ofld_destroy_io_queues(nctrl, remove);
 }
 
 static void nvme_tcp_ofld_reconnect_ctrl_work(struct work_struct *work)
@@ -578,6 +863,12 @@ nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 	return 0;
 }
 
+inline size_t nvme_tcp_ofld_inline_data_size(struct nvme_tcp_ofld_queue *queue)
+{
+	return queue->cmnd_capsule_len - sizeof(struct nvme_command);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_inline_data_size);
+
 static blk_status_t
 nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 		       const struct blk_mq_queue_data *bd)
@@ -589,22 +880,95 @@ nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 	return BLK_STS_OK;
 }
 
+static void
+nvme_tcp_ofld_exit_request(struct blk_mq_tag_set *set,
+			   struct request *rq, unsigned int hctx_idx)
+{
+	/*
+	 * Nothing is allocated in nvme_tcp_ofld_init_request,
+	 * hence empty.
+	 */
+}
+
+static int
+nvme_tcp_ofld_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			unsigned int hctx_idx)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = data;
+
+	hctx->driver_data = &ctrl->queues[hctx_idx + 1];
+
+	return 0;
+}
+
+static int nvme_tcp_ofld_map_queues(struct blk_mq_tag_set *set)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = set->driver_data;
+	struct nvmf_ctrl_options *opts = ctrl->nctrl.opts;
+
+	if (opts->nr_write_queues && ctrl->io_queues[HCTX_TYPE_READ]) {
+		/* separate read/write queues */
+		set->map[HCTX_TYPE_DEFAULT].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_DEFAULT].queue_offset = 0;
+		set->map[HCTX_TYPE_READ].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_READ];
+		set->map[HCTX_TYPE_READ].queue_offset =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	} else {
+		/* shared read/write queues */
+		set->map[HCTX_TYPE_DEFAULT].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_DEFAULT].queue_offset = 0;
+		set->map[HCTX_TYPE_READ].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_READ].queue_offset = 0;
+	}
+	blk_mq_map_queues(&set->map[HCTX_TYPE_DEFAULT]);
+	blk_mq_map_queues(&set->map[HCTX_TYPE_READ]);
+
+	if (opts->nr_poll_queues && ctrl->io_queues[HCTX_TYPE_POLL]) {
+		/* map dedicated poll queues only if we have queues left */
+		set->map[HCTX_TYPE_POLL].nr_queues =
+				ctrl->io_queues[HCTX_TYPE_POLL];
+		set->map[HCTX_TYPE_POLL].queue_offset =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT] +
+			ctrl->io_queues[HCTX_TYPE_READ];
+		blk_mq_map_queues(&set->map[HCTX_TYPE_POLL]);
+	}
+
+	dev_info(ctrl->nctrl.device,
+		 "mapped %d/%d/%d default/read/poll queues.\n",
+		 ctrl->io_queues[HCTX_TYPE_DEFAULT],
+		 ctrl->io_queues[HCTX_TYPE_READ],
+		 ctrl->io_queues[HCTX_TYPE_POLL]);
+
+	return 0;
+}
+
+static int nvme_tcp_ofld_poll(struct blk_mq_hw_ctx *hctx)
+{
+	/* Placeholder - Implement polling mechanism */
+
+	return 0;
+}
+
 static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
 	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.complete	= nvme_complete_rq,
 	.init_request	= nvme_tcp_ofld_init_request,
-	/*
-	 * All additional ops will be also implemented and registered similar to
-	 * tcp.c
-	 */
+	.exit_request	= nvme_tcp_ofld_exit_request,
+	.init_hctx	= nvme_tcp_ofld_init_hctx,
+	.map_queues	= nvme_tcp_ofld_map_queues,
+	.poll		= nvme_tcp_ofld_poll,
 };
 
 static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
 	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.complete	= nvme_complete_rq,
 	.init_request	= nvme_tcp_ofld_init_request,
-	/*
-	 * All additional ops will be also implemented and registered similar to
-	 * tcp.c
-	 */
+	.exit_request	= nvme_tcp_ofld_exit_request,
+	.init_hctx	= nvme_tcp_ofld_init_admin_hctx,
 };
 
 static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index 51fec632b72b..b3502c01394e 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -65,6 +65,9 @@ struct nvme_tcp_ofld_queue {
 	unsigned long flags;
 	size_t cmnd_capsule_len;
 
+	/* mutex used during stop_queue */
+	struct mutex queue_lock;
+
 	u8 hdr_digest;
 	u8 data_digest;
 	u8 tos;
@@ -199,3 +202,4 @@ struct nvme_tcp_ofld_ops {
 int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl);
+inline size_t nvme_tcp_ofld_inline_data_size(struct nvme_tcp_ofld_queue *queue);
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:23:59 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:23:59 +0300
Subject: [PATCH v3 09/20] qedn: Add qedn - Marvell's NVMeTCP HW offload device
 driver
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-10-smalin@marvell.com>

This patch will present the skeleton of the qedn driver.
The new driver will be added under "drivers/nvme/hw/qedn" and will be
enabled by the Kconfig "Marvell NVM Express over Fabrics TCP offload".

The internal implementation:
- qedn.h:
  Includes all common structs to be used by the qedn offload device driver.

- qedn_main.c
  Includes the qedn_init and qedn_cleanup implementation.
  As part of the qedn init, the driver will register as a pci device and
  will work with the Marvell fastlinQ NICs.
  As part of the probe, the driver will register to the nvme_tcp_offload
  (ULP).

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 MAINTAINERS                      |  10 ++
 drivers/nvme/Kconfig             |   1 +
 drivers/nvme/Makefile            |   1 +
 drivers/nvme/hw/Kconfig          |   8 ++
 drivers/nvme/hw/Makefile         |   3 +
 drivers/nvme/hw/qedn/Makefile    |   5 +
 drivers/nvme/hw/qedn/qedn.h      |  19 +++
 drivers/nvme/hw/qedn/qedn_main.c | 200 +++++++++++++++++++++++++++++++
 8 files changed, 247 insertions(+)
 create mode 100644 drivers/nvme/hw/Kconfig
 create mode 100644 drivers/nvme/hw/Makefile
 create mode 100644 drivers/nvme/hw/qedn/Makefile
 create mode 100644 drivers/nvme/hw/qedn/qedn.h
 create mode 100644 drivers/nvme/hw/qedn/qedn_main.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 01fbebdc7722..207a62b768c5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -15006,6 +15006,16 @@ S:	Supported
 F:	drivers/infiniband/hw/qedr/
 F:	include/uapi/rdma/qedr-abi.h
 
+QLOGIC QL4xxx NVME-TCP-OFFLOAD DRIVER
+M:	Shai Malin <smalin at marvell.com>
+M:	Ariel Elior <aelior at marvell.com>
+L:	linux-nvme at lists.infradead.org
+S:	Supported
+W:	http://git.infradead.org/nvme.git
+T:	git://git.infradead.org/nvme.git
+F:	drivers/nvme/hw/qedn/
+F:	include/linux/qed/
+
 QLOGIC QLA1280 SCSI DRIVER
 M:	Michael Reed <mdr at sgi.com>
 L:	linux-scsi at vger.kernel.org
diff --git a/drivers/nvme/Kconfig b/drivers/nvme/Kconfig
index 87ae409a32b9..827c2c9f0ad1 100644
--- a/drivers/nvme/Kconfig
+++ b/drivers/nvme/Kconfig
@@ -3,5 +3,6 @@ menu "NVME Support"
 
 source "drivers/nvme/host/Kconfig"
 source "drivers/nvme/target/Kconfig"
+source "drivers/nvme/hw/Kconfig"
 
 endmenu
diff --git a/drivers/nvme/Makefile b/drivers/nvme/Makefile
index fb42c44609a8..14c569040ef2 100644
--- a/drivers/nvme/Makefile
+++ b/drivers/nvme/Makefile
@@ -2,3 +2,4 @@
 
 obj-y		+= host/
 obj-y		+= target/
+obj-y		+= hw/
\ No newline at end of file
diff --git a/drivers/nvme/hw/Kconfig b/drivers/nvme/hw/Kconfig
new file mode 100644
index 000000000000..374f1f9dbd3d
--- /dev/null
+++ b/drivers/nvme/hw/Kconfig
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NVME_QEDN
+	tristate "Marvell NVM Express over Fabrics TCP offload"
+	depends on NVME_TCP_OFFLOAD
+	help
+	  This enables the Marvell NVMe TCP offload support (qedn).
+
+	  If unsure, say N.
diff --git a/drivers/nvme/hw/Makefile b/drivers/nvme/hw/Makefile
new file mode 100644
index 000000000000..2f38e0520795
--- /dev/null
+++ b/drivers/nvme/hw/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_NVME_QEDN)		+= qedn/
diff --git a/drivers/nvme/hw/qedn/Makefile b/drivers/nvme/hw/qedn/Makefile
new file mode 100644
index 000000000000..1422cd878680
--- /dev/null
+++ b/drivers/nvme/hw/qedn/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_NVME_QEDN) := qedn.o
+
+qedn-y := qedn_main.o
diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
new file mode 100644
index 000000000000..bcd0748a10fd
--- /dev/null
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#ifndef _QEDN_H_
+#define _QEDN_H_
+
+/* Driver includes */
+#include "../../host/tcp-offload.h"
+
+#define QEDN_MODULE_NAME "qedn"
+
+struct qedn_ctx {
+	struct pci_dev *pdev;
+	struct nvme_tcp_ofld_dev qedn_ofld_dev;
+};
+
+#endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
new file mode 100644
index 000000000000..d81b7c65acdf
--- /dev/null
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -0,0 +1,200 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+ /* Kernel includes */
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+/* Driver includes */
+#include "qedn.h"
+
+#define CHIP_NUM_NVMETCP 0x8194
+
+static struct pci_device_id qedn_pci_tbl[] = {
+	{ PCI_VDEVICE(QLOGIC, CHIP_NUM_NVMETCP), 0 },
+	{0, 0},
+};
+
+static int
+qedn_claim_dev(struct nvme_tcp_ofld_dev *dev,
+	       struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	/* Placeholder - qedn_claim_dev */
+
+	return 0;
+}
+
+static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	/* Placeholder - qedn_setup_ctrl */
+
+	return 0;
+}
+
+static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	/* Placeholder - qedn_release_ctrl */
+
+	return 0;
+}
+
+static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
+			     size_t queue_size)
+{
+	/* Placeholder - qedn_create_queue */
+
+	return 0;
+}
+
+static void qedn_drain_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	/* Placeholder - qedn_drain_queue */
+}
+
+static void qedn_destroy_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	/* Placeholder - qedn_destroy_queue */
+}
+
+static int qedn_poll_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	/*
+	 * Poll queue support will be added as part of future
+	 * enhancements.
+	 */
+
+	return 0;
+}
+
+static int qedn_send_req(struct nvme_tcp_ofld_req *req)
+{
+	/* Placeholder - qedn_send_req */
+
+	return 0;
+}
+
+static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
+	.name = "qedn",
+	.module = THIS_MODULE,
+	.required_opts = NVMF_OPT_TRADDR,
+	.allowed_opts = NVMF_OPT_TRSVCID | NVMF_OPT_NR_WRITE_QUEUES |
+			NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO |
+			NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HOST_IFACE,
+		/* These flags will be as part of future enhancements
+		 *	NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST |
+		 *	NVMF_OPT_NR_POLL_QUEUES | NVMF_OPT_TOS
+		 */
+	.claim_dev = qedn_claim_dev,
+	.setup_ctrl = qedn_setup_ctrl,
+	.release_ctrl = qedn_release_ctrl,
+	.create_queue = qedn_create_queue,
+	.drain_queue = qedn_drain_queue,
+	.destroy_queue = qedn_destroy_queue,
+	.poll_queue = qedn_poll_queue,
+	.send_req = qedn_send_req,
+};
+
+static void __qedn_remove(struct pci_dev *pdev)
+{
+	struct qedn_ctx *qedn = pci_get_drvdata(pdev);
+
+	pr_notice("Starting qedn_remove\n");
+	nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev);
+	kfree(qedn);
+	pr_notice("Ending qedn_remove successfully\n");
+}
+
+static void qedn_remove(struct pci_dev *pdev)
+{
+	__qedn_remove(pdev);
+}
+
+static void qedn_shutdown(struct pci_dev *pdev)
+{
+	__qedn_remove(pdev);
+}
+
+static struct qedn_ctx *qedn_alloc_ctx(struct pci_dev *pdev)
+{
+	struct qedn_ctx *qedn = NULL;
+
+	qedn = kzalloc(sizeof(*qedn), GFP_KERNEL);
+	if (!qedn)
+		return NULL;
+
+	qedn->pdev = pdev;
+	pci_set_drvdata(pdev, qedn);
+
+	return qedn;
+}
+
+static int __qedn_probe(struct pci_dev *pdev)
+{
+	struct qedn_ctx *qedn;
+	int rc;
+
+	pr_notice("Starting qedn probe\n");
+
+	qedn = qedn_alloc_ctx(pdev);
+	if (!qedn)
+		return -ENODEV;
+
+	qedn->qedn_ofld_dev.ops = &qedn_ofld_ops;
+	INIT_LIST_HEAD(&qedn->qedn_ofld_dev.entry);
+	rc = nvme_tcp_ofld_register_dev(&qedn->qedn_ofld_dev);
+	if (rc)
+		goto release_qedn;
+
+	return 0;
+release_qedn:
+	kfree(qedn);
+
+	return rc;
+}
+
+static int qedn_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+	return __qedn_probe(pdev);
+}
+
+static struct pci_driver qedn_pci_driver = {
+	.name     = QEDN_MODULE_NAME,
+	.id_table = qedn_pci_tbl,
+	.probe    = qedn_probe,
+	.remove   = qedn_remove,
+	.shutdown = qedn_shutdown,
+};
+
+static int __init qedn_init(void)
+{
+	int rc;
+
+	rc = pci_register_driver(&qedn_pci_driver);
+	if (rc) {
+		pr_err("Failed to register pci driver\n");
+
+		return -EINVAL;
+	}
+
+	pr_notice("driver loaded successfully\n");
+
+	return 0;
+}
+
+static void __exit qedn_cleanup(void)
+{
+	pci_unregister_driver(&qedn_pci_driver);
+	pr_notice("Unloading qedn ended\n");
+}
+
+module_init(qedn_init);
+module_exit(qedn_cleanup);
+
+MODULE_LICENSE("GPL v2");
+MODULE_SOFTDEP("pre: qede nvme-fabrics nvme-tcp-offload");
+MODULE_DESCRIPTION("Marvell 25/50/100G NVMe-TCP Offload Host Driver");
+MODULE_AUTHOR("Marvell");
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:23:58 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:23:58 +0300
Subject: [PATCH v3 08/20] nvme-tcp-offload: Add IO level implementation
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-9-smalin@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

In this patch, we present the IO level functionality.
The nvme-tcp-offload shall work on the IO-level, meaning the
nvme-tcp-offload ULP module shall pass the request to the nvme-tcp-offload
driver and shall expect for the request completion.
No additional handling is needed in between, this design will reduce the
CPU utilization as we will describe below.

The nvme-tcp-offload driver shall register to nvme-tcp-offload ULP
with the following IO-path ops:
 - send_req - in order to pass the request to the handling of the offload
   driver that shall pass it to the offload specific device
 - poll_queue

The offload device driver will manage the context from which the request
will be executed and the request aggregations.
Once the IO completed, the nvme-tcp-offload driver shall call
command.done() that shall invoke the nvme-tcp-offload ULP layer for
completing the request.

This patch also add support for the nvme-tcp-offload timeout and
nvme-tcp-offload ASYNC flow.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 drivers/nvme/host/tcp-offload.c | 180 ++++++++++++++++++++++++++++++--
 drivers/nvme/host/tcp-offload.h |   2 +
 2 files changed, 175 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 26253b107db2..7ba56136f9b1 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -125,7 +125,30 @@ void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
 			    union nvme_result *result,
 			    __le16 status)
 {
-	/* Placeholder - complete request with/without error */
+	struct request *rq = blk_mq_rq_from_pdu(req);
+
+	if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), *result))
+		nvme_complete_rq(rq);
+}
+
+/**
+ * nvme_tcp_ofld_async_req_done() - NVMeTCP Offload request done callback
+ * function for async request. Pointed to by nvme_tcp_ofld_req->done.
+ * Handles both NVME_TCP_F_DATA_SUCCESS flag and NVMe CQ.
+ * @req:	NVMeTCP offload request to complete.
+ * @result:     The nvme_result.
+ * @status:     The completion status.
+ *
+ * API function that allows the offload device specific driver to report
+ * request completions to the common offload layer.
+ */
+void nvme_tcp_ofld_async_req_done(struct nvme_tcp_ofld_req *req,
+				  union nvme_result *result, __le16 status)
+{
+	struct nvme_tcp_ofld_queue *queue = req->queue;
+	struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl;
+
+	nvme_complete_async_event(&ctrl->nctrl, status, result);
 }
 
 static struct nvme_tcp_ofld_dev *
@@ -717,6 +740,56 @@ static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
 	kfree(ctrl);
 }
 
+static void nvme_tcp_ofld_set_sg_null(struct nvme_command *c)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = 0;
+	sg->length = 0;
+	sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) |
+			NVME_SGL_FMT_TRANSPORT_A;
+}
+
+inline void nvme_tcp_ofld_set_sg_inline(struct nvme_tcp_ofld_queue *queue,
+					struct nvme_command *c, u32 data_len)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = cpu_to_le64(queue->ctrl->nctrl.icdoff);
+	sg->length = cpu_to_le32(data_len);
+	sg->type = (NVME_SGL_FMT_DATA_DESC << 4) | NVME_SGL_FMT_OFFSET;
+}
+
+static void nvme_tcp_ofld_map_data(struct nvme_command *c, u32 data_len)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = 0;
+	sg->length = cpu_to_le32(data_len);
+	sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) |
+			NVME_SGL_FMT_TRANSPORT_A;
+}
+
+static void nvme_tcp_ofld_submit_async_event(struct nvme_ctrl *arg)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(arg);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[0];
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+
+	ctrl->async_req.nvme_cmd.common.opcode = nvme_admin_async_event;
+	ctrl->async_req.nvme_cmd.common.command_id = NVME_AQ_BLK_MQ_DEPTH;
+	ctrl->async_req.nvme_cmd.common.flags |= NVME_CMD_SGL_METABUF;
+
+	nvme_tcp_ofld_set_sg_null(&ctrl->async_req.nvme_cmd);
+
+	ctrl->async_req.async = true;
+	ctrl->async_req.queue = queue;
+	ctrl->async_req.done = nvme_tcp_ofld_async_req_done;
+
+	ops->send_req(&ctrl->async_req);
+}
+
 static void
 nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *nctrl, bool remove)
 {
@@ -855,9 +928,13 @@ nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 			   unsigned int numa_node)
 {
 	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_ofld_ctrl *ctrl = set->driver_data;
+	int qid;
 
-	/* Placeholder - init request */
-
+	qid = (set == &ctrl->tag_set) ? hctx_idx + 1 : 0;
+	req->queue = &ctrl->queues[qid];
+	nvme_req(rq)->ctrl = &ctrl->nctrl;
+	nvme_req(rq)->cmd = &req->nvme_cmd;
 	req->done = nvme_tcp_ofld_req_done;
 
 	return 0;
@@ -873,9 +950,46 @@ static blk_status_t
 nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 		       const struct blk_mq_queue_data *bd)
 {
-	/* Call nvme_setup_cmd(...) */
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(bd->rq);
+	struct nvme_tcp_ofld_queue *queue = hctx->driver_data;
+	struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl;
+	struct nvme_ns *ns = hctx->queue->queuedata;
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+	struct nvme_command *nvme_cmd;
+	struct request *rq = bd->rq;
+	bool queue_ready;
+	u32 data_len;
+	int rc;
+
+	queue_ready = test_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags);
+
+	req->async = false;
+
+	if (!nvme_check_ready(&ctrl->nctrl, rq, queue_ready))
+		return nvme_fail_nonready_command(&ctrl->nctrl, rq);
+
+	rc = nvme_setup_cmd(ns, rq);
+	if (unlikely(rc))
+		return rc;
 
-	/* Call ops->send_req(...) */
+	blk_mq_start_request(rq);
+
+	nvme_cmd = &req->nvme_cmd;
+	nvme_cmd->common.flags |= NVME_CMD_SGL_METABUF;
+
+	data_len = blk_rq_nr_phys_segments(rq) ? blk_rq_payload_bytes(rq) : 0;
+	if (!data_len)
+		nvme_tcp_ofld_set_sg_null(&req->nvme_cmd);
+	else if ((rq_data_dir(rq) == WRITE) &&
+		 data_len <= nvme_tcp_ofld_inline_data_size(queue))
+		nvme_tcp_ofld_set_sg_inline(queue, nvme_cmd, data_len);
+	else
+		nvme_tcp_ofld_map_data(nvme_cmd, data_len);
+
+	rc = ops->send_req(req);
+	if (unlikely(rc))
+		return rc;
 
 	return BLK_STS_OK;
 }
@@ -948,9 +1062,58 @@ static int nvme_tcp_ofld_map_queues(struct blk_mq_tag_set *set)
 
 static int nvme_tcp_ofld_poll(struct blk_mq_hw_ctx *hctx)
 {
-	/* Placeholder - Implement polling mechanism */
+	struct nvme_tcp_ofld_queue *queue = hctx->driver_data;
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
 
-	return 0;
+	return ops->poll_queue(queue);
+}
+
+static void nvme_tcp_ofld_complete_timed_out(struct request *rq)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_ctrl *nctrl = &req->queue->ctrl->nctrl;
+
+	nvme_tcp_ofld_stop_queue(nctrl, nvme_tcp_ofld_qid(req->queue));
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq)) {
+		nvme_req(rq)->status = NVME_SC_HOST_ABORTED_CMD;
+		blk_mq_complete_request(rq);
+	}
+}
+
+static enum blk_eh_timer_return nvme_tcp_ofld_timeout(struct request *rq,
+						      bool reserved)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_ofld_ctrl *ctrl = req->queue->ctrl;
+
+	dev_warn(ctrl->nctrl.device,
+		 "queue %d: timeout request %#x type %d\n",
+		 nvme_tcp_ofld_qid(req->queue), rq->tag,
+		 req->nvme_cmd.common.opcode);
+
+	if (ctrl->nctrl.state != NVME_CTRL_LIVE) {
+		/*
+		 * If we are resetting, connecting or deleting we should
+		 * complete immediately because we may block controller
+		 * teardown or setup sequence
+		 * - ctrl disable/shutdown fabrics requests
+		 * - connect requests
+		 * - initialization admin requests
+		 * - I/O requests that entered after unquiescing and
+		 *   the controller stopped responding
+		 *
+		 * All other requests should be cancelled by the error
+		 * recovery work, so it's fine that we fail it here.
+		 */
+		nvme_tcp_ofld_complete_timed_out(rq);
+
+		return BLK_EH_DONE;
+	}
+
+	nvme_tcp_ofld_error_recovery(&ctrl->nctrl);
+
+	return BLK_EH_RESET_TIMER;
 }
 
 static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
@@ -959,6 +1122,7 @@ static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
 	.init_request	= nvme_tcp_ofld_init_request,
 	.exit_request	= nvme_tcp_ofld_exit_request,
 	.init_hctx	= nvme_tcp_ofld_init_hctx,
+	.timeout	= nvme_tcp_ofld_timeout,
 	.map_queues	= nvme_tcp_ofld_map_queues,
 	.poll		= nvme_tcp_ofld_poll,
 };
@@ -969,6 +1133,7 @@ static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
 	.init_request	= nvme_tcp_ofld_init_request,
 	.exit_request	= nvme_tcp_ofld_exit_request,
 	.init_hctx	= nvme_tcp_ofld_init_admin_hctx,
+	.timeout	= nvme_tcp_ofld_timeout,
 };
 
 static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
@@ -979,6 +1144,7 @@ static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
 	.reg_read64		= nvmf_reg_read64,
 	.reg_write32		= nvmf_reg_write32,
 	.free_ctrl		= nvme_tcp_ofld_free_ctrl,
+	.submit_async_event     = nvme_tcp_ofld_submit_async_event,
 	.delete_ctrl		= nvme_tcp_ofld_delete_ctrl,
 	.get_address		= nvmf_get_address,
 };
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index b3502c01394e..a4c28ddaf3ab 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -115,6 +115,8 @@ struct nvme_tcp_ofld_ctrl {
 	/* Connectivity params */
 	struct nvme_tcp_ofld_ctrl_con_params conn_params;
 
+	struct nvme_tcp_ofld_req async_req;
+
 	/* Offload device driver context */
 	void *private_data;
 };
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:24:01 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:24:01 +0300
Subject: [PATCH v3 11/20] qedn: Add qedn_claim_dev API support
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-12-smalin@marvell.com>

From: Nikolay Assa <nassa at marvell.com>

This patch introduces the qedn_claim_dev() network service which the
offload device (qedn) is using through the paired net-device (qede).
qedn_claim_dev() returns true if the IP addr(IPv4 or IPv6) of the target
server is reachable via the net-device which is paired with the
offloaded device.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Nikolay Assa <nassa at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |  4 +++
 drivers/nvme/hw/qedn/qedn_main.c | 55 ++++++++++++++++++++++++++++++--
 2 files changed, 56 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 931efc3afbaa..0ce1e19d1ba8 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -8,6 +8,10 @@
 
 #include <linux/qed/qed_if.h>
 #include <linux/qed/qed_nvmetcp_if.h>
+#include <linux/qed/qed_nvmetcp_ip_services_if.h>
+#include <linux/qed/qed_chain.h>
+#include <linux/qed/storage_common.h>
+#include <linux/qed/nvmetcp_common.h>
 
 /* Driver includes */
 #include "../../host/tcp-offload.h"
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 97591797605e..78bc9fe17e7b 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -22,13 +22,62 @@ static struct pci_device_id qedn_pci_tbl[] = {
 	{0, 0},
 };
 
+static int
+qedn_find_dev(struct nvme_tcp_ofld_dev *dev,
+	      struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	struct nvme_tcp_ofld_ctrl_con_params *conn_params;
+	struct pci_dev *qede_pdev = NULL;
+	struct sockaddr remote_mac_addr;
+	struct net_device *ndev = NULL;
+	u16 vlan_id = 0;
+	int rc = 0;
+
+	conn_params = &ctrl->conn_params;
+
+	/* qedn utilizes host network stack through paired qede device for
+	 * non-offload traffic. First we verify there is valid route to remote
+	 * peer.
+	 */
+	if (conn_params->remote_ip_addr.ss_family == AF_INET) {
+		rc = qed_route_ipv4(&conn_params->local_ip_addr,
+				    &conn_params->remote_ip_addr,
+				    &remote_mac_addr, &ndev);
+	} else if (conn_params->remote_ip_addr.ss_family == AF_INET6) {
+		rc = qed_route_ipv6(&conn_params->local_ip_addr,
+				    &conn_params->remote_ip_addr,
+				    &remote_mac_addr, &ndev);
+	} else {
+		pr_err("address family %d not supported\n",
+		       conn_params->remote_ip_addr.ss_family);
+
+		return false;
+	}
+
+	if (rc)
+		return false;
+
+	if (!ctrl->private_data && ctrl->ndev &&
+	    strcmp(ctrl->ndev->name, ndev->name))
+		return false;
+
+	ctrl->ndev = ndev;
+
+	qed_vlan_get_ndev(&ctrl->ndev, &vlan_id);
+
+	/* route found through ndev - validate this is qede*/
+	qede_pdev = qed_validate_ndev(ctrl->ndev);
+	if (!qede_pdev)
+		return false;
+
+	return true;
+}
+
 static int
 qedn_claim_dev(struct nvme_tcp_ofld_dev *dev,
 	       struct nvme_tcp_ofld_ctrl *ctrl)
 {
-	/* Placeholder - qedn_claim_dev */
-
-	return 0;
+	return qedn_find_dev(dev, ctrl);
 }
 
 static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:24:02 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:24:02 +0300
Subject: [PATCH v3 12/20] qedn: Add IRQ and fast-path resources initializations
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-13-smalin@marvell.com>

This patch will present the adding of qedn_fp_queue - this is a per cpu
core element which handles all of the connections on that cpu core.
The qedn_fp_queue will handle a group of connections (NVMeoF QPs) which
are handled on the same cpu core, and will only use the same FW-driver
resources with no need to be related to the same NVMeoF controller.

The per qedn_fq_queue resources are the FW CQ and FW status block:
- The FW CQ will be used for the FW to notify the driver that the
  the exchange has ended and the FW will pass the incoming NVMeoF CQE
  (if exist) to the driver.
- FW status block - which is used for the FW to notify the driver with
  the producer update of the FW CQE chain.

The FW fast-path queues are based on qed_chain.h

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |  25 +++
 drivers/nvme/hw/qedn/qedn_main.c | 289 ++++++++++++++++++++++++++++++-
 2 files changed, 311 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 0ce1e19d1ba8..edb0836bca87 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -24,17 +24,39 @@
 #define QEDN_PROTO_CQ_PROD_IDX	0
 #define QEDN_NVMETCP_NUM_FW_CONN_QUEUE_PAGES 2
 
+#define QEDN_PAGE_SIZE	4096 /* FW page size - Configurable */
+#define QEDN_IRQ_NAME_LEN 24
+#define QEDN_IRQ_NO_FLAGS 0
+
+#define QEDN_TCP_RTO_DEFAULT 280
+
 enum qedn_state {
 	QEDN_STATE_CORE_PROBED = 0,
 	QEDN_STATE_CORE_OPEN,
 	QEDN_STATE_MFW_STATE,
+	QEDN_STATE_NVMETCP_OPEN,
+	QEDN_STATE_IRQ_SET,
+	QEDN_STATE_FP_WORK_THREAD_SET,
 	QEDN_STATE_REGISTERED_OFFLOAD_DEV,
 	QEDN_STATE_MODULE_REMOVE_ONGOING,
 };
 
+/* Per CPU core params */
+struct qedn_fp_queue {
+	struct qed_chain cq_chain;
+	u16 *cq_prod;
+	struct mutex cq_mutex; /* cq handler mutex */
+	struct qedn_ctx	*qedn;
+	struct qed_sb_info *sb_info;
+	unsigned int cpu;
+	u16 sb_id;
+	char irqname[QEDN_IRQ_NAME_LEN];
+};
+
 struct qedn_ctx {
 	struct pci_dev *pdev;
 	struct qed_dev *cdev;
+	struct qed_int_info int_info;
 	struct qed_dev_nvmetcp_info dev_info;
 	struct nvme_tcp_ofld_dev qedn_ofld_dev;
 	struct qed_pf_params pf_params;
@@ -44,6 +66,9 @@ struct qedn_ctx {
 
 	/* Fast path queues */
 	u8 num_fw_cqs;
+	struct qedn_fp_queue *fp_q_arr;
+	struct nvmetcp_glbl_queue_entry *fw_cq_array_virt;
+	dma_addr_t fw_cq_array_phy; /* Physical address of fw_cq_array_virt */
 };
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 78bc9fe17e7b..6c0f36f7d9d1 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -150,6 +150,104 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
 	.send_req = qedn_send_req,
 };
 
+/* Fastpath IRQ handler */
+static irqreturn_t qedn_irq_handler(int irq, void *dev_id)
+{
+	/* Placeholder */
+
+	return IRQ_HANDLED;
+}
+
+static void qedn_sync_free_irqs(struct qedn_ctx *qedn)
+{
+	u16 vector_idx;
+	int i;
+
+	for (i = 0; i < qedn->num_fw_cqs; i++) {
+		vector_idx = i * qedn->dev_info.common.num_hwfns +
+			     qed_ops->common->get_affin_hwfn_idx(qedn->cdev);
+		synchronize_irq(qedn->int_info.msix[vector_idx].vector);
+		irq_set_affinity_hint(qedn->int_info.msix[vector_idx].vector,
+				      NULL);
+		free_irq(qedn->int_info.msix[vector_idx].vector,
+			 &qedn->fp_q_arr[i]);
+	}
+
+	qedn->int_info.used_cnt = 0;
+	qed_ops->common->set_fp_int(qedn->cdev, 0);
+}
+
+static int qedn_request_msix_irq(struct qedn_ctx *qedn)
+{
+	struct pci_dev *pdev = qedn->pdev;
+	struct qedn_fp_queue *fp_q = NULL;
+	int i, rc, cpu;
+	u16 vector_idx;
+	u32 vector;
+
+	/* numa-awareness will be added in future enhancements */
+	cpu = cpumask_first(cpu_online_mask);
+	for (i = 0; i < qedn->num_fw_cqs; i++) {
+		fp_q = &qedn->fp_q_arr[i];
+		vector_idx = i * qedn->dev_info.common.num_hwfns +
+			     qed_ops->common->get_affin_hwfn_idx(qedn->cdev);
+		vector = qedn->int_info.msix[vector_idx].vector;
+		sprintf(fp_q->irqname, "qedn_queue_%x.%x.%x_%d",
+			pdev->bus->number, PCI_SLOT(pdev->devfn),
+			PCI_FUNC(pdev->devfn), i);
+		rc = request_irq(vector, qedn_irq_handler, QEDN_IRQ_NO_FLAGS,
+				 fp_q->irqname, fp_q);
+		if (rc) {
+			pr_err("request_irq failed.\n");
+			qedn_sync_free_irqs(qedn);
+
+			return rc;
+		}
+
+		fp_q->cpu = cpu;
+		qedn->int_info.used_cnt++;
+		rc = irq_set_affinity_hint(vector, get_cpu_mask(cpu));
+		cpu = cpumask_next_wrap(cpu, cpu_online_mask, -1, false);
+	}
+
+	return 0;
+}
+
+static int qedn_setup_irq(struct qedn_ctx *qedn)
+{
+	int rc = 0;
+	u8 rval;
+
+	rval = qed_ops->common->set_fp_int(qedn->cdev, qedn->num_fw_cqs);
+	if (rval < qedn->num_fw_cqs) {
+		qedn->num_fw_cqs = rval;
+		if (rval == 0) {
+			pr_err("set_fp_int return 0 IRQs\n");
+
+			return -ENODEV;
+		}
+	}
+
+	rc = qed_ops->common->get_fp_int(qedn->cdev, &qedn->int_info);
+	if (rc) {
+		pr_err("get_fp_int failed\n");
+		goto exit_setup_int;
+	}
+
+	if (qedn->int_info.msix_cnt) {
+		rc = qedn_request_msix_irq(qedn);
+		goto exit_setup_int;
+	} else {
+		pr_err("msix_cnt = 0\n");
+		rc = -EINVAL;
+		goto exit_setup_int;
+	}
+
+exit_setup_int:
+
+	return rc;
+}
+
 static inline void qedn_init_pf_struct(struct qedn_ctx *qedn)
 {
 	/* Placeholder - Initialize qedn fields */
@@ -180,21 +278,174 @@ static inline int qedn_core_probe(struct qedn_ctx *qedn)
 	return rc;
 }
 
+static void qedn_free_function_queues(struct qedn_ctx *qedn)
+{
+	struct qed_sb_info *sb_info = NULL;
+	struct qedn_fp_queue *fp_q;
+	int i;
+
+	/* Free workqueues */
+
+	/* Free the fast path queues*/
+	for (i = 0; i < qedn->num_fw_cqs; i++) {
+		fp_q = &qedn->fp_q_arr[i];
+
+		/* Free SB */
+		sb_info = fp_q->sb_info;
+		if (sb_info->sb_virt) {
+			qed_ops->common->sb_release(qedn->cdev, sb_info,
+						    fp_q->sb_id,
+						    QED_SB_TYPE_STORAGE);
+			dma_free_coherent(&qedn->pdev->dev,
+					  sizeof(*sb_info->sb_virt),
+					  (void *)sb_info->sb_virt,
+					  sb_info->sb_phys);
+			memset(sb_info, 0, sizeof(*sb_info));
+			kfree(sb_info);
+			fp_q->sb_info = NULL;
+		}
+
+		qed_ops->common->chain_free(qedn->cdev, &fp_q->cq_chain);
+	}
+
+	if (qedn->fw_cq_array_virt)
+		dma_free_coherent(&qedn->pdev->dev,
+				  qedn->num_fw_cqs * sizeof(u64),
+				  qedn->fw_cq_array_virt,
+				  qedn->fw_cq_array_phy);
+	kfree(qedn->fp_q_arr);
+	qedn->fp_q_arr = NULL;
+}
+
+static int qedn_alloc_and_init_sb(struct qedn_ctx *qedn,
+				  struct qed_sb_info *sb_info, u16 sb_id)
+{
+	int rc = 0;
+
+	sb_info->sb_virt = dma_alloc_coherent(&qedn->pdev->dev,
+					      sizeof(struct status_block_e4),
+					      &sb_info->sb_phys, GFP_KERNEL);
+	if (!sb_info->sb_virt) {
+		pr_err("Status block allocation failed\n");
+
+		return -ENOMEM;
+	}
+
+	rc = qed_ops->common->sb_init(qedn->cdev, sb_info, sb_info->sb_virt,
+				      sb_info->sb_phys, sb_id,
+				      QED_SB_TYPE_STORAGE);
+	if (rc) {
+		pr_err("Status block initialization failed\n");
+
+		return rc;
+	}
+
+	return 0;
+}
+
+static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
+{
+	struct qed_chain_init_params chain_params = {};
+	struct status_block_e4 *sb = NULL;
+	struct qedn_fp_queue *fp_q = NULL;
+	int rc = 0, arr_size;
+	u64 cq_phy_addr;
+	int i;
+
+	/* Place holder - IO-path workqueues */
+
+	qedn->fp_q_arr = kcalloc(qedn->num_fw_cqs,
+				 sizeof(struct qedn_fp_queue), GFP_KERNEL);
+	if (!qedn->fp_q_arr)
+		return -ENOMEM;
+
+	arr_size = qedn->num_fw_cqs * sizeof(struct nvmetcp_glbl_queue_entry);
+	qedn->fw_cq_array_virt = dma_alloc_coherent(&qedn->pdev->dev,
+						    arr_size,
+						    &qedn->fw_cq_array_phy,
+						    GFP_KERNEL);
+	if (!qedn->fw_cq_array_virt) {
+		rc = -ENOMEM;
+		goto mem_alloc_failure;
+	}
+
+	/* placeholder - create task pools */
+
+	for (i = 0; i < qedn->num_fw_cqs; i++) {
+		fp_q = &qedn->fp_q_arr[i];
+		mutex_init(&fp_q->cq_mutex);
+
+		/* FW CQ */
+		chain_params.intended_use = QED_CHAIN_USE_TO_CONSUME,
+		chain_params.mode = QED_CHAIN_MODE_PBL,
+		chain_params.cnt_type = QED_CHAIN_CNT_TYPE_U16,
+		chain_params.num_elems = QEDN_FW_CQ_SIZE;
+		/* Placeholder - sizeof(struct nvmetcp_fw_cqe)*/
+		chain_params.elem_size = 64;
+
+		rc = qed_ops->common->chain_alloc(qedn->cdev,
+						  &fp_q->cq_chain,
+						  &chain_params);
+		if (rc) {
+			pr_err("CQ chain pci_alloc_consistent fail\n");
+			goto mem_alloc_failure;
+		}
+
+		cq_phy_addr = qed_chain_get_pbl_phys(&fp_q->cq_chain);
+		qedn->fw_cq_array_virt[i].cq_pbl_addr.hi = PTR_HI(cq_phy_addr);
+		qedn->fw_cq_array_virt[i].cq_pbl_addr.lo = PTR_LO(cq_phy_addr);
+
+		/* SB */
+		fp_q->sb_info = kzalloc(sizeof(*fp_q->sb_info), GFP_KERNEL);
+		if (!fp_q->sb_info)
+			goto mem_alloc_failure;
+
+		fp_q->sb_id = i;
+		rc = qedn_alloc_and_init_sb(qedn, fp_q->sb_info, fp_q->sb_id);
+		if (rc) {
+			pr_err("SB allocation and initialization failed.\n");
+			goto mem_alloc_failure;
+		}
+
+		sb = fp_q->sb_info->sb_virt;
+		fp_q->cq_prod = (u16 *)&sb->pi_array[QEDN_PROTO_CQ_PROD_IDX];
+		fp_q->qedn = qedn;
+
+		/* Placeholder - Init IO-path workqueue */
+
+		/* Placeholder - Init IO-path resources */
+	}
+
+	return 0;
+
+mem_alloc_failure:
+	pr_err("Function allocation failed\n");
+	qedn_free_function_queues(qedn);
+
+	return rc;
+}
+
 static int qedn_set_nvmetcp_pf_param(struct qedn_ctx *qedn)
 {
 	u32 fw_conn_queue_pages = QEDN_NVMETCP_NUM_FW_CONN_QUEUE_PAGES;
 	struct qed_nvmetcp_pf_params *pf_params;
+	int rc;
 
 	pf_params = &qedn->pf_params.nvmetcp_pf_params;
 	memset(pf_params, 0, sizeof(*pf_params));
 	qedn->num_fw_cqs = min_t(u8, qedn->dev_info.num_cqs, num_online_cpus());
+	pr_info("Num qedn FW CQs %u\n", qedn->num_fw_cqs);
 
 	pf_params->num_cons = QEDN_MAX_CONNS_PER_PF;
 	pf_params->num_tasks = QEDN_MAX_TASKS_PER_PF;
 
-	/* Placeholder - Initialize function level queues */
+	rc = qedn_alloc_function_queues(qedn);
+	if (rc) {
+		pr_err("Global queue allocation failed.\n");
+		goto err_alloc_mem;
+	}
 
-	/* Placeholder - Initialize TCP params */
+	set_bit(QEDN_STATE_FP_WORK_THREAD_SET, &qedn->state);
 
 	/* Queues */
 	pf_params->num_sq_pages_in_ring = fw_conn_queue_pages;
@@ -202,11 +453,14 @@ static int qedn_set_nvmetcp_pf_param(struct qedn_ctx *qedn)
 	pf_params->num_uhq_pages_in_ring = fw_conn_queue_pages;
 	pf_params->num_queues = qedn->num_fw_cqs;
 	pf_params->cq_num_entries = QEDN_FW_CQ_SIZE;
+	pf_params->glbl_q_params_addr = qedn->fw_cq_array_phy;
 
 	/* the CQ SB pi */
 	pf_params->gl_rq_pi = QEDN_PROTO_CQ_PROD_IDX;
 
-	return 0;
+err_alloc_mem:
+
+	return rc;
 }
 
 static inline int qedn_slowpath_start(struct qedn_ctx *qedn)
@@ -241,6 +495,12 @@ static void __qedn_remove(struct pci_dev *pdev)
 	if (test_and_clear_bit(QEDN_STATE_REGISTERED_OFFLOAD_DEV, &qedn->state))
 		nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev);
 
+	if (test_and_clear_bit(QEDN_STATE_IRQ_SET, &qedn->state))
+		qedn_sync_free_irqs(qedn);
+
+	if (test_and_clear_bit(QEDN_STATE_NVMETCP_OPEN, &qedn->state))
+		qed_ops->stop(qedn->cdev);
+
 	if (test_and_clear_bit(QEDN_STATE_MFW_STATE, &qedn->state)) {
 		rc = qed_ops->common->update_drv_state(qedn->cdev, false);
 		if (rc)
@@ -250,6 +510,9 @@ static void __qedn_remove(struct pci_dev *pdev)
 	if (test_and_clear_bit(QEDN_STATE_CORE_OPEN, &qedn->state))
 		qed_ops->common->slowpath_stop(qedn->cdev);
 
+	if (test_and_clear_bit(QEDN_STATE_FP_WORK_THREAD_SET, &qedn->state))
+		qedn_free_function_queues(qedn);
+
 	if (test_and_clear_bit(QEDN_STATE_CORE_PROBED, &qedn->state))
 		qed_ops->common->remove(qedn->cdev);
 
@@ -318,6 +581,25 @@ static int __qedn_probe(struct pci_dev *pdev)
 
 	set_bit(QEDN_STATE_CORE_OPEN, &qedn->state);
 
+	rc = qedn_setup_irq(qedn);
+	if (rc)
+		goto exit_probe_and_release_mem;
+
+	set_bit(QEDN_STATE_IRQ_SET, &qedn->state);
+
+	/* NVMeTCP start HW PF */
+	rc = qed_ops->start(qedn->cdev,
+			    NULL /* Placeholder for FW IO-path resources */,
+			    qedn,
+			    NULL /* Placeholder for FW Event callback */);
+	if (rc) {
+		rc = -ENODEV;
+		pr_err("Cannot start NVMeTCP Function\n");
+		goto exit_probe_and_release_mem;
+	}
+
+	set_bit(QEDN_STATE_NVMETCP_OPEN, &qedn->state);
+
 	rc = qed_ops->common->update_drv_state(qedn->cdev, true);
 	if (rc) {
 		pr_err("Failed to send drv state to MFW\n");
@@ -326,6 +608,7 @@ static int __qedn_probe(struct pci_dev *pdev)
 
 	set_bit(QEDN_STATE_MFW_STATE, &qedn->state);
 
+	qedn->qedn_ofld_dev.num_hw_vectors = qedn->num_fw_cqs;
 	qedn->qedn_ofld_dev.ops = &qedn_ofld_ops;
 	INIT_LIST_HEAD(&qedn->qedn_ofld_dev.entry);
 	rc = nvme_tcp_ofld_register_dev(&qedn->qedn_ofld_dev);
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:24:00 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:24:00 +0300
Subject: [PATCH v3 10/20] qedn: Add qedn probe
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-11-smalin@marvell.com>

This patch introduces the functionality of loading and unloading
physical function.
qedn_probe() loads the offload device PF(physical function), and
initialize the HW and the FW with the PF parameters using the
HW ops->qed_nvmetcp_ops, which are similar to other "qed_*_ops" which
are used by the qede, qedr, qedf and qedi device drivers.
qedn_remove() unloads the offload device PF, re-initialize the HW and
the FW with the PF parameters.

The struct qedn_ctx is per PF container for PF-specific attributes and
resources.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/Kconfig          |   1 +
 drivers/nvme/hw/qedn/qedn.h      |  26 ++++++
 drivers/nvme/hw/qedn/qedn_main.c | 155 ++++++++++++++++++++++++++++++-
 3 files changed, 177 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/hw/Kconfig b/drivers/nvme/hw/Kconfig
index 374f1f9dbd3d..91b1bd6f07d8 100644
--- a/drivers/nvme/hw/Kconfig
+++ b/drivers/nvme/hw/Kconfig
@@ -2,6 +2,7 @@
 config NVME_QEDN
 	tristate "Marvell NVM Express over Fabrics TCP offload"
 	depends on NVME_TCP_OFFLOAD
+	select QED_NVMETCP
 	help
 	  This enables the Marvell NVMe TCP offload support (qedn).
 
diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index bcd0748a10fd..931efc3afbaa 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -6,14 +6,40 @@
 #ifndef _QEDN_H_
 #define _QEDN_H_
 
+#include <linux/qed/qed_if.h>
+#include <linux/qed/qed_nvmetcp_if.h>
+
 /* Driver includes */
 #include "../../host/tcp-offload.h"
 
 #define QEDN_MODULE_NAME "qedn"
 
+#define QEDN_MAX_TASKS_PER_PF (16 * 1024)
+#define QEDN_MAX_CONNS_PER_PF (4 * 1024)
+#define QEDN_FW_CQ_SIZE (4 * 1024)
+#define QEDN_PROTO_CQ_PROD_IDX	0
+#define QEDN_NVMETCP_NUM_FW_CONN_QUEUE_PAGES 2
+
+enum qedn_state {
+	QEDN_STATE_CORE_PROBED = 0,
+	QEDN_STATE_CORE_OPEN,
+	QEDN_STATE_MFW_STATE,
+	QEDN_STATE_REGISTERED_OFFLOAD_DEV,
+	QEDN_STATE_MODULE_REMOVE_ONGOING,
+};
+
 struct qedn_ctx {
 	struct pci_dev *pdev;
+	struct qed_dev *cdev;
+	struct qed_dev_nvmetcp_info dev_info;
 	struct nvme_tcp_ofld_dev qedn_ofld_dev;
+	struct qed_pf_params pf_params;
+
+	/* Accessed with atomic bit ops, used with enum qedn_state */
+	unsigned long state;
+
+	/* Fast path queues */
+	u8 num_fw_cqs;
 };
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index d81b7c65acdf..97591797605e 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -14,6 +14,9 @@
 
 #define CHIP_NUM_NVMETCP 0x8194
 
+const struct qed_nvmetcp_ops *qed_ops;
+
+/* Global context instance */
 static struct pci_device_id qedn_pci_tbl[] = {
 	{ PCI_VDEVICE(QLOGIC, CHIP_NUM_NVMETCP), 0 },
 	{0, 0},
@@ -98,12 +101,109 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
 	.send_req = qedn_send_req,
 };
 
+static inline void qedn_init_pf_struct(struct qedn_ctx *qedn)
+{
+	/* Placeholder - Initialize qedn fields */
+}
+
+static inline void
+qedn_init_core_probe_params(struct qed_probe_params *probe_params)
+{
+	memset(probe_params, 0, sizeof(*probe_params));
+	probe_params->protocol = QED_PROTOCOL_NVMETCP;
+	probe_params->is_vf = false;
+	probe_params->recov_in_prog = 0;
+}
+
+static inline int qedn_core_probe(struct qedn_ctx *qedn)
+{
+	struct qed_probe_params probe_params;
+	int rc = 0;
+
+	qedn_init_core_probe_params(&probe_params);
+	pr_info("Starting QED probe\n");
+	qedn->cdev = qed_ops->common->probe(qedn->pdev, &probe_params);
+	if (!qedn->cdev) {
+		rc = -ENODEV;
+		pr_err("QED probe failed\n");
+	}
+
+	return rc;
+}
+
+static int qedn_set_nvmetcp_pf_param(struct qedn_ctx *qedn)
+{
+	u32 fw_conn_queue_pages = QEDN_NVMETCP_NUM_FW_CONN_QUEUE_PAGES;
+	struct qed_nvmetcp_pf_params *pf_params;
+
+	pf_params = &qedn->pf_params.nvmetcp_pf_params;
+	memset(pf_params, 0, sizeof(*pf_params));
+	qedn->num_fw_cqs = min_t(u8, qedn->dev_info.num_cqs, num_online_cpus());
+
+	pf_params->num_cons = QEDN_MAX_CONNS_PER_PF;
+	pf_params->num_tasks = QEDN_MAX_TASKS_PER_PF;
+
+	/* Placeholder - Initialize function level queues */
+
+	/* Placeholder - Initialize TCP params */
+
+	/* Queues */
+	pf_params->num_sq_pages_in_ring = fw_conn_queue_pages;
+	pf_params->num_r2tq_pages_in_ring = fw_conn_queue_pages;
+	pf_params->num_uhq_pages_in_ring = fw_conn_queue_pages;
+	pf_params->num_queues = qedn->num_fw_cqs;
+	pf_params->cq_num_entries = QEDN_FW_CQ_SIZE;
+
+	/* the CQ SB pi */
+	pf_params->gl_rq_pi = QEDN_PROTO_CQ_PROD_IDX;
+
+	return 0;
+}
+
+static inline int qedn_slowpath_start(struct qedn_ctx *qedn)
+{
+	struct qed_slowpath_params sp_params = {};
+	int rc = 0;
+
+	/* Start the Slowpath-process */
+	sp_params.int_mode = QED_INT_MODE_MSIX;
+	strscpy(sp_params.name, "qedn NVMeTCP", QED_DRV_VER_STR_SIZE);
+	rc = qed_ops->common->slowpath_start(qedn->cdev, &sp_params);
+	if (rc)
+		pr_err("Cannot start slowpath\n");
+
+	return rc;
+}
+
 static void __qedn_remove(struct pci_dev *pdev)
 {
 	struct qedn_ctx *qedn = pci_get_drvdata(pdev);
+	int rc;
+
+	pr_notice("Starting qedn_remove: abs PF id=%u\n",
+		  qedn->dev_info.common.abs_pf_id);
+
+	if (test_and_set_bit(QEDN_STATE_MODULE_REMOVE_ONGOING, &qedn->state)) {
+		pr_err("Remove already ongoing\n");
+
+		return;
+	}
+
+	if (test_and_clear_bit(QEDN_STATE_REGISTERED_OFFLOAD_DEV, &qedn->state))
+		nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev);
+
+	if (test_and_clear_bit(QEDN_STATE_MFW_STATE, &qedn->state)) {
+		rc = qed_ops->common->update_drv_state(qedn->cdev, false);
+		if (rc)
+			pr_err("Failed to send drv state to MFW\n");
+	}
+
+	if (test_and_clear_bit(QEDN_STATE_CORE_OPEN, &qedn->state))
+		qed_ops->common->slowpath_stop(qedn->cdev);
+
+	if (test_and_clear_bit(QEDN_STATE_CORE_PROBED, &qedn->state))
+		qed_ops->common->remove(qedn->cdev);
 
-	pr_notice("Starting qedn_remove\n");
-	nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev);
 	kfree(qedn);
 	pr_notice("Ending qedn_remove successfully\n");
 }
@@ -143,15 +243,52 @@ static int __qedn_probe(struct pci_dev *pdev)
 	if (!qedn)
 		return -ENODEV;
 
+	qedn_init_pf_struct(qedn);
+
+	/* QED probe */
+	rc = qedn_core_probe(qedn);
+	if (rc)
+		goto exit_probe_and_release_mem;
+
+	set_bit(QEDN_STATE_CORE_PROBED, &qedn->state);
+
+	rc = qed_ops->fill_dev_info(qedn->cdev, &qedn->dev_info);
+	if (rc) {
+		pr_err("fill_dev_info failed\n");
+		goto exit_probe_and_release_mem;
+	}
+
+	rc = qedn_set_nvmetcp_pf_param(qedn);
+	if (rc)
+		goto exit_probe_and_release_mem;
+
+	qed_ops->common->update_pf_params(qedn->cdev, &qedn->pf_params);
+	rc = qedn_slowpath_start(qedn);
+	if (rc)
+		goto exit_probe_and_release_mem;
+
+	set_bit(QEDN_STATE_CORE_OPEN, &qedn->state);
+
+	rc = qed_ops->common->update_drv_state(qedn->cdev, true);
+	if (rc) {
+		pr_err("Failed to send drv state to MFW\n");
+		goto exit_probe_and_release_mem;
+	}
+
+	set_bit(QEDN_STATE_MFW_STATE, &qedn->state);
+
 	qedn->qedn_ofld_dev.ops = &qedn_ofld_ops;
 	INIT_LIST_HEAD(&qedn->qedn_ofld_dev.entry);
 	rc = nvme_tcp_ofld_register_dev(&qedn->qedn_ofld_dev);
 	if (rc)
-		goto release_qedn;
+		goto exit_probe_and_release_mem;
+
+	set_bit(QEDN_STATE_REGISTERED_OFFLOAD_DEV, &qedn->state);
 
 	return 0;
-release_qedn:
-	kfree(qedn);
+exit_probe_and_release_mem:
+	__qedn_remove(pdev);
+	pr_err("probe ended with error\n");
 
 	return rc;
 }
@@ -173,6 +310,13 @@ static int __init qedn_init(void)
 {
 	int rc;
 
+	qed_ops = qed_get_nvmetcp_ops();
+	if (!qed_ops) {
+		pr_err("Failed to get QED NVMeTCP ops\n");
+
+		return -EINVAL;
+	}
+
 	rc = pci_register_driver(&qedn_pci_driver);
 	if (rc) {
 		pr_err("Failed to register pci driver\n");
@@ -188,6 +332,7 @@ static int __init qedn_init(void)
 static void __exit qedn_cleanup(void)
 {
 	pci_unregister_driver(&qedn_pci_driver);
+	qed_put_nvmetcp_ops();
 	pr_notice("Unloading qedn ended\n");
 }
 
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:24:06 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:24:06 +0300
Subject: [PATCH v3 16/20] qedn: Add support of Task and SGL
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-17-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

This patch will add support of Task and SGL which is used
for slowpath and fast path IO. here Task is IO granule used
by firmware to perform tasks

The internal implementation:
- Create task/sgl resources used by all connection
- Provide APIs to allocate and free task.
- Add task support during connection establishment i.e. slowpath

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
---
 drivers/nvme/hw/qedn/qedn.h      |  61 ++++++
 drivers/nvme/hw/qedn/qedn_conn.c |  46 ++++-
 drivers/nvme/hw/qedn/qedn_main.c |  34 +++-
 drivers/nvme/hw/qedn/qedn_task.c | 326 +++++++++++++++++++++++++++++++
 4 files changed, 463 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 13d63f420a23..78efa8c02810 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -40,6 +40,16 @@
 
 #define QEDN_FW_CQ_FP_WQ_WORKQUEUE "qedn_fw_cq_fp_wq"
 
+/* Protocol defines */
+#define QEDN_MAX_IO_SIZE QED_NVMETCP_MAX_IO_SIZE
+
+#define QEDN_FW_SLOW_IO_MIN_SGE_LIMIT (9700 / 6)
+
+#define QEDN_MAX_HW_SECTORS (QEDN_MAX_IO_SIZE / 512)
+#define QEDN_MAX_SEGMENTS 2048
+
+#define QEDN_INVALID_ITID 0xFFFF
+
 /*
  * TCP offload stack default configurations and defines.
  * Future enhancements will allow controlling the configurable
@@ -84,6 +94,15 @@ enum qedn_state {
 	QEDN_STATE_MODULE_REMOVE_ONGOING,
 };
 
+struct qedn_io_resources {
+	/* Lock for IO resources */
+	spinlock_t resources_lock;
+	struct list_head task_free_list;
+	u32 num_alloc_tasks;
+	u32 num_free_tasks;
+	u32 no_avail_resrc_cnt;
+};
+
 /* Per CPU core params */
 struct qedn_fp_queue {
 	struct qed_chain cq_chain;
@@ -93,6 +112,10 @@ struct qedn_fp_queue {
 	struct qed_sb_info *sb_info;
 	unsigned int cpu;
 	struct work_struct fw_cq_fp_wq_entry;
+
+	/* IO related resources for host */
+	struct qedn_io_resources host_resrc;
+
 	u16 sb_id;
 	char irqname[QEDN_IRQ_NAME_LEN];
 };
@@ -116,12 +139,35 @@ struct qedn_ctx {
 	/* Connections */
 	DECLARE_HASHTABLE(conn_ctx_hash, 16);
 
+	u32 num_tasks_per_pool;
+
 	/* Fast path queues */
 	u8 num_fw_cqs;
 	struct qedn_fp_queue *fp_q_arr;
 	struct nvmetcp_glbl_queue_entry *fw_cq_array_virt;
 	dma_addr_t fw_cq_array_phy; /* Physical address of fw_cq_array_virt */
 	struct workqueue_struct *fw_cq_fp_wq;
+
+	/* Fast Path Tasks */
+	struct qed_nvmetcp_tid	tasks;
+};
+
+struct qedn_task_ctx {
+	struct qedn_conn_ctx *qedn_conn;
+	struct qedn_ctx *qedn;
+	void *fw_task_ctx;
+	struct qedn_fp_queue *fp_q;
+	struct scatterlist *nvme_sg;
+	struct nvme_tcp_ofld_req *req; /* currently proccessed request */
+	struct list_head entry;
+	spinlock_t lock; /* To protect task resources */
+	bool valid;
+	unsigned long flags; /* Used by qedn_task_flags */
+	u32 task_size;
+	u16 itid;
+	u16 cccid;
+	int req_direction;
+	struct storage_sgl_task_params sgl_task_params;
 };
 
 struct qedn_endpoint {
@@ -219,6 +265,7 @@ struct qedn_conn_ctx {
 	struct nvme_tcp_ofld_ctrl *ctrl;
 	u32 conn_handle;
 	u32 fw_cid;
+	u8 default_cq;
 
 	atomic_t est_conn_indicator;
 	atomic_t destroy_conn_indicator;
@@ -236,6 +283,11 @@ struct qedn_conn_ctx {
 	dma_addr_t host_cccid_itid_phy_addr;
 	struct qedn_endpoint ep;
 	int abrt_flag;
+	/* Spinlock for accessing active_task_list */
+	spinlock_t task_list_lock;
+	struct list_head active_task_list;
+	atomic_t num_active_tasks;
+	atomic_t num_active_fw_tasks;
 
 	/* Connection resources - turned on to indicate what resource was
 	 * allocated, to that it can later be released.
@@ -255,6 +307,7 @@ struct qedn_conn_ctx {
 enum qedn_conn_resources_state {
 	QEDN_CONN_RESRC_FW_SQ,
 	QEDN_CONN_RESRC_ACQUIRE_CONN,
+	QEDN_CONN_RESRC_TASKS,
 	QEDN_CONN_RESRC_CCCID_ITID_MAP,
 	QEDN_CONN_RESRC_TCP_PORT,
 	QEDN_CONN_RESRC_DB_ADD,
@@ -278,5 +331,13 @@ int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 		       struct nvme_tcp_ofld_req *req);
 void qedn_nvme_req_fp_wq_handler(struct work_struct *work);
 void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe);
+int qedn_alloc_tasks(struct qedn_conn_ctx *conn_ctx);
+inline int qedn_qid(struct nvme_tcp_ofld_queue *queue);
+void qedn_common_clear_fw_sgl(struct storage_sgl_task_params *sgl_task_params);
+void qedn_return_active_tasks(struct qedn_conn_ctx *conn_ctx);
+struct qedn_task_ctx *
+qedn_get_free_task_from_pool(struct qedn_conn_ctx *conn_ctx, u16 cccid);
+void qedn_destroy_free_tasks(struct qedn_fp_queue *fp_q,
+			     struct qedn_io_resources *io_resrc);
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
index 962c88a4f345..5c5b365df522 100644
--- a/drivers/nvme/hw/qedn/qedn_conn.c
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -29,6 +29,11 @@ static const char * const qedn_conn_state_str[] = {
 	NULL
 };
 
+inline int qedn_qid(struct nvme_tcp_ofld_queue *queue)
+{
+	return queue - queue->ctrl->queues;
+}
+
 int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
 		       enum qedn_conn_state new_state)
 {
@@ -170,6 +175,11 @@ static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx)
 		clear_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state);
 	}
 
+	if (test_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state)) {
+		clear_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state);
+			qedn_return_active_tasks(conn_ctx);
+	}
+
 	if (test_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state)) {
 		dma_free_coherent(&qedn->pdev->dev,
 				  conn_ctx->sq_depth *
@@ -272,6 +282,7 @@ static int qedn_nvmetcp_offload_conn(struct qedn_conn_ctx *conn_ctx)
 	offld_prms.max_rt_time = QEDN_TCP_MAX_RT_TIME;
 	offld_prms.sq_pbl_addr =
 		(u64)qed_chain_get_pbl_phys(&qedn_ep->fw_sq_chain);
+	offld_prms.default_cq = conn_ctx->default_cq;
 
 	rc = qed_ops->offload_conn(qedn->cdev,
 				   conn_ctx->conn_handle,
@@ -420,7 +431,10 @@ void qedn_prep_db_data(struct qedn_conn_ctx *conn_ctx)
 static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 {
 	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct qedn_io_resources *io_resrc;
+	struct qedn_fp_queue *fp_q;
 	size_t dma_size;
+	u8 qid;
 	int rc;
 
 	rc = qedn_alloc_fw_sq(qedn, &conn_ctx->ep);
@@ -431,6 +445,9 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 
 	set_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state);
 
+	atomic_set(&conn_ctx->num_active_tasks, 0);
+	atomic_set(&conn_ctx->num_active_fw_tasks, 0);
+
 	rc = qed_ops->acquire_conn(qedn->cdev,
 				   &conn_ctx->conn_handle,
 				   &conn_ctx->fw_cid,
@@ -444,7 +461,34 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 		 conn_ctx->conn_handle);
 	set_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state);
 
-	/* Placeholder - Allocate task resources and initialize fields */
+	qid = qedn_qid(conn_ctx->queue);
+
+	/* default_cq is mapped 1:1 with qid (with cpu core) which are
+	 * assigned for the driver
+	 */
+	conn_ctx->default_cq = qid ? qid - 1 : 0;
+	fp_q = &qedn->fp_q_arr[conn_ctx->default_cq];
+	conn_ctx->fp_q = fp_q;
+	io_resrc = &fp_q->host_resrc;
+
+	/* The first connection on each fp_q will fill task
+	 * resources
+	 */
+	spin_lock(&io_resrc->resources_lock);
+	if (io_resrc->num_alloc_tasks == 0) {
+		rc = qedn_alloc_tasks(conn_ctx);
+		if (rc) {
+			pr_err("Failed allocating tasks: CID=0x%x\n",
+			       conn_ctx->fw_cid);
+			spin_unlock(&io_resrc->resources_lock);
+			goto rel_conn;
+		}
+	}
+	spin_unlock(&io_resrc->resources_lock);
+
+	spin_lock_init(&conn_ctx->task_list_lock);
+	INIT_LIST_HEAD(&conn_ctx->active_task_list);
+	set_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state);
 
 	rc = qedn_fetch_tcp_port(conn_ctx);
 	if (rc)
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 975949ce6fb0..42c8ad6ac2d6 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -29,6 +29,12 @@ __be16 qedn_get_in_port(struct sockaddr_storage *sa)
 		: ((struct sockaddr_in6 *)sa)->sin6_port;
 }
 
+static void qedn_init_io_resc(struct qedn_io_resources *io_resrc)
+{
+	spin_lock_init(&io_resrc->resources_lock);
+	INIT_LIST_HEAD(&io_resrc->task_free_list);
+}
+
 struct qedn_llh_filter *qedn_add_llh_filter(struct qedn_ctx *qedn, u16 tcp_port)
 {
 	struct qedn_llh_filter *llh_filter = NULL;
@@ -436,6 +442,8 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
 		 *	NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST |
 		 *	NVMF_OPT_NR_POLL_QUEUES | NVMF_OPT_TOS
 		 */
+	.max_hw_sectors = QEDN_MAX_HW_SECTORS,
+	.max_segments = QEDN_MAX_SEGMENTS,
 	.claim_dev = qedn_claim_dev,
 	.setup_ctrl = qedn_setup_ctrl,
 	.release_ctrl = qedn_release_ctrl,
@@ -640,8 +648,24 @@ static inline int qedn_core_probe(struct qedn_ctx *qedn)
 	return rc;
 }
 
+static void qedn_call_destroy_free_tasks(struct qedn_fp_queue *fp_q,
+					 struct qedn_io_resources *io_resrc)
+{
+	if (list_empty(&io_resrc->task_free_list))
+		return;
+
+	if (io_resrc->num_alloc_tasks != io_resrc->num_free_tasks)
+		pr_err("Task Pool:Not all returned allocated=0x%x, free=0x%x\n",
+		       io_resrc->num_alloc_tasks, io_resrc->num_free_tasks);
+
+	qedn_destroy_free_tasks(fp_q, io_resrc);
+	if (io_resrc->num_free_tasks)
+		pr_err("Expected num_free_tasks to be 0\n");
+}
+
 static void qedn_free_function_queues(struct qedn_ctx *qedn)
 {
+	struct qedn_io_resources *host_resrc;
 	struct qed_sb_info *sb_info = NULL;
 	struct qedn_fp_queue *fp_q;
 	int i;
@@ -653,6 +677,9 @@ static void qedn_free_function_queues(struct qedn_ctx *qedn)
 	/* Free the fast path queues*/
 	for (i = 0; i < qedn->num_fw_cqs; i++) {
 		fp_q = &qedn->fp_q_arr[i];
+		host_resrc = &fp_q->host_resrc;
+
+		qedn_call_destroy_free_tasks(fp_q, host_resrc);
 
 		/* Free SB */
 		sb_info = fp_q->sb_info;
@@ -740,7 +767,8 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
 		goto mem_alloc_failure;
 	}
 
-	/* placeholder - create task pools */
+	qedn->num_tasks_per_pool =
+		qedn->pf_params.nvmetcp_pf_params.num_tasks / qedn->num_fw_cqs;
 
 	for (i = 0; i < qedn->num_fw_cqs; i++) {
 		fp_q = &qedn->fp_q_arr[i];
@@ -782,7 +810,7 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
 		fp_q->qedn = qedn;
 		INIT_WORK(&fp_q->fw_cq_fp_wq_entry, qedn_fw_cq_fq_wq_handler);
 
-		/* Placeholder - Init IO-path resources */
+		qedn_init_io_resc(&fp_q->host_resrc);
 	}
 
 	return 0;
@@ -964,7 +992,7 @@ static int __qedn_probe(struct pci_dev *pdev)
 
 	/* NVMeTCP start HW PF */
 	rc = qed_ops->start(qedn->cdev,
-			    NULL /* Placeholder for FW IO-path resources */,
+			    &qedn->tasks,
 			    qedn,
 			    qedn_event_cb);
 	if (rc) {
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
index a3af55fba95c..7b228b5c5169 100644
--- a/drivers/nvme/hw/qedn/qedn_task.c
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -11,6 +11,332 @@
 /* Driver includes */
 #include "qedn.h"
 
+static void qedn_free_nvme_sg(struct qedn_task_ctx *qedn_task)
+{
+	kfree(qedn_task->nvme_sg);
+	qedn_task->nvme_sg = NULL;
+}
+
+static void qedn_free_fw_sgl(struct qedn_task_ctx *qedn_task)
+{
+	struct qedn_ctx *qedn = qedn_task->qedn;
+	dma_addr_t sgl_pa;
+
+	sgl_pa = HILO_DMA_REGPAIR(qedn_task->sgl_task_params.sgl_phys_addr);
+	dma_free_coherent(&qedn->pdev->dev,
+			  QEDN_MAX_SEGMENTS * sizeof(struct nvmetcp_sge),
+			  qedn_task->sgl_task_params.sgl,
+			  sgl_pa);
+	qedn_task->sgl_task_params.sgl = NULL;
+}
+
+static void qedn_destroy_single_task(struct qedn_task_ctx *qedn_task)
+{
+	u16 itid;
+
+	itid = qedn_task->itid;
+	list_del(&qedn_task->entry);
+	qedn_free_nvme_sg(qedn_task);
+	qedn_free_fw_sgl(qedn_task);
+	kfree(qedn_task);
+	qedn_task = NULL;
+}
+
+void qedn_destroy_free_tasks(struct qedn_fp_queue *fp_q,
+			     struct qedn_io_resources *io_resrc)
+{
+	struct qedn_task_ctx *qedn_task, *task_tmp;
+
+	/* Destroy tasks from the free task list */
+	list_for_each_entry_safe(qedn_task, task_tmp,
+				 &io_resrc->task_free_list, entry) {
+		qedn_destroy_single_task(qedn_task);
+		io_resrc->num_free_tasks -= 1;
+	}
+}
+
+static int qedn_alloc_nvme_sg(struct qedn_task_ctx *qedn_task)
+{
+	int rc;
+
+	qedn_task->nvme_sg = kcalloc(QEDN_MAX_SEGMENTS,
+				     sizeof(*qedn_task->nvme_sg), GFP_KERNEL);
+	if (!qedn_task->nvme_sg) {
+		rc = -ENOMEM;
+
+		return rc;
+	}
+
+	return 0;
+}
+
+static int qedn_alloc_fw_sgl(struct qedn_task_ctx *qedn_task)
+{
+	struct nvmetcp_sge **sgl = &qedn_task->sgl_task_params.sgl;
+	struct qedn_ctx *qedn = qedn_task->qedn_conn->qedn;
+	dma_addr_t sgl_phys;
+	u32 sz;
+
+	sz = QEDN_MAX_SEGMENTS * sizeof(struct nvmetcp_sge);
+	*sgl = dma_alloc_coherent(&qedn->pdev->dev, sz, &sgl_phys, GFP_KERNEL);
+	if (!*sgl) {
+		pr_err("Couldn't allocate FW sgl\n");
+
+		return -ENOMEM;
+	}
+
+	DMA_REGPAIR_LE(qedn_task->sgl_task_params.sgl_phys_addr, sgl_phys);
+
+	return 0;
+}
+
+static inline void *qedn_get_fw_task(struct qed_nvmetcp_tid *info, u16 itid)
+{
+	return (void *)(info->blocks[itid / info->num_tids_per_block] +
+			(itid % info->num_tids_per_block) * info->size);
+}
+
+static struct qedn_task_ctx *qedn_alloc_task(struct qedn_conn_ctx *conn_ctx,
+					     u16 itid)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct qedn_task_ctx *qedn_task;
+	void *fw_task_ctx;
+	int rc = 0;
+
+	qedn_task = kzalloc(sizeof(*qedn_task), GFP_KERNEL);
+	if (!qedn_task)
+		return NULL;
+
+	spin_lock_init(&qedn_task->lock);
+	fw_task_ctx = qedn_get_fw_task(&qedn->tasks, itid);
+	if (!fw_task_ctx) {
+		pr_err("iTID: 0x%x; Failed getting fw_task_ctx memory\n", itid);
+		goto release_task;
+	}
+
+	/* No need to memset fw_task_ctx - its done in the HSI func */
+	qedn_task->qedn_conn = conn_ctx;
+	qedn_task->qedn = qedn;
+	qedn_task->fw_task_ctx = fw_task_ctx;
+	qedn_task->valid = 0;
+	qedn_task->flags = 0;
+	qedn_task->itid = itid;
+	rc = qedn_alloc_fw_sgl(qedn_task);
+	if (rc) {
+		pr_err("iTID: 0x%x; Failed allocating FW sgl\n", itid);
+		goto release_task;
+	}
+
+	rc = qedn_alloc_nvme_sg(qedn_task);
+	if (rc) {
+		pr_err("iTID: 0x%x; Failed allocating FW sgl\n", itid);
+		goto release_fw_sgl;
+	}
+
+	return qedn_task;
+
+release_fw_sgl:
+	qedn_free_fw_sgl(qedn_task);
+release_task:
+	kfree(qedn_task);
+
+	return NULL;
+}
+
+int qedn_alloc_tasks(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct qedn_task_ctx *qedn_task = NULL;
+	struct qedn_io_resources *io_resrc;
+	u16 itid, start_itid, offset;
+	struct qedn_fp_queue *fp_q;
+	int i, rc;
+
+	fp_q = conn_ctx->fp_q;
+
+	offset = fp_q->sb_id;
+	io_resrc = &fp_q->host_resrc;
+
+	start_itid = qedn->num_tasks_per_pool * offset;
+	for (i = 0; i < qedn->num_tasks_per_pool; ++i) {
+		itid = start_itid + i;
+		qedn_task = qedn_alloc_task(conn_ctx, itid);
+		if (!qedn_task) {
+			pr_err("Failed allocating task\n");
+			rc = -ENOMEM;
+			goto release_tasks;
+		}
+
+		qedn_task->fp_q = fp_q;
+		io_resrc->num_free_tasks += 1;
+		list_add_tail(&qedn_task->entry, &io_resrc->task_free_list);
+	}
+
+	io_resrc->num_alloc_tasks = io_resrc->num_free_tasks;
+
+	return 0;
+
+release_tasks:
+	qedn_destroy_free_tasks(fp_q, io_resrc);
+
+	return rc;
+}
+
+void qedn_common_clear_fw_sgl(struct storage_sgl_task_params *sgl_task_params)
+{
+	u16 sge_cnt = sgl_task_params->num_sges;
+
+	memset(&sgl_task_params->sgl[(sge_cnt - 1)], 0,
+	       sizeof(struct nvmetcp_sge));
+	sgl_task_params->total_buffer_size = 0;
+	sgl_task_params->small_mid_sge = false;
+	sgl_task_params->num_sges = 0;
+}
+
+inline void qedn_host_reset_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx,
+					     u16 cccid)
+{
+	conn_ctx->host_cccid_itid[cccid].itid = cpu_to_le16(QEDN_INVALID_ITID);
+}
+
+inline void qedn_host_set_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx,
+					   u16 cccid, u16 itid)
+{
+	conn_ctx->host_cccid_itid[cccid].itid = cpu_to_le16(itid);
+}
+
+static void qedn_clear_sgl(struct qedn_ctx *qedn,
+			   struct qedn_task_ctx *qedn_task)
+{
+	struct storage_sgl_task_params *sgl_task_params;
+	enum dma_data_direction dma_dir;
+	u32 sge_cnt;
+
+	sgl_task_params = &qedn_task->sgl_task_params;
+	sge_cnt = sgl_task_params->num_sges;
+
+	/* Nothing to do if no SGEs were used */
+	if (!qedn_task->task_size || !sge_cnt)
+		return;
+
+	dma_dir = (qedn_task->req_direction == WRITE ?
+			DMA_TO_DEVICE : DMA_FROM_DEVICE);
+	dma_unmap_sg(&qedn->pdev->dev, qedn_task->nvme_sg, sge_cnt, dma_dir);
+	memset(&qedn_task->nvme_sg[(sge_cnt - 1)], 0,
+	       sizeof(struct scatterlist));
+	qedn_common_clear_fw_sgl(sgl_task_params);
+	qedn_task->task_size = 0;
+}
+
+static void qedn_clear_task(struct qedn_conn_ctx *conn_ctx,
+			    struct qedn_task_ctx *qedn_task)
+{
+	/* Task lock isn't needed since it is no longer in use */
+	qedn_clear_sgl(conn_ctx->qedn, qedn_task);
+	qedn_task->valid = 0;
+	qedn_task->flags = 0;
+
+	atomic_dec(&conn_ctx->num_active_tasks);
+}
+
+void qedn_return_active_tasks(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_fp_queue *fp_q = conn_ctx->fp_q;
+	struct qedn_task_ctx *qedn_task, *task_tmp;
+	struct qedn_io_resources *io_resrc;
+	int num_returned_tasks = 0;
+	int num_active_tasks;
+
+	io_resrc = &fp_q->host_resrc;
+
+	/* Return tasks that aren't "Used by FW" to the pool */
+	list_for_each_entry_safe(qedn_task, task_tmp,
+				 &conn_ctx->active_task_list, entry) {
+		qedn_clear_task(conn_ctx, qedn_task);
+		num_returned_tasks++;
+	}
+
+	if (num_returned_tasks) {
+		spin_lock(&io_resrc->resources_lock);
+		/* Return tasks to FP_Q pool in one shot */
+
+		list_splice_tail_init(&conn_ctx->active_task_list,
+				      &io_resrc->task_free_list);
+		io_resrc->num_free_tasks += num_returned_tasks;
+		spin_unlock(&io_resrc->resources_lock);
+	}
+
+	num_active_tasks = atomic_read(&conn_ctx->num_active_tasks);
+	if (num_active_tasks)
+		pr_err("num_active_tasks is %u after cleanup.\n",
+		       num_active_tasks);
+}
+
+void qedn_return_task_to_pool(struct qedn_conn_ctx *conn_ctx,
+			      struct qedn_task_ctx *qedn_task)
+{
+	struct qedn_fp_queue *fp_q = conn_ctx->fp_q;
+	struct qedn_io_resources *io_resrc;
+	unsigned long lock_flags;
+
+	io_resrc = &fp_q->host_resrc;
+
+	spin_lock_irqsave(&qedn_task->lock, lock_flags);
+	qedn_task->valid = 0;
+	qedn_task->flags = 0;
+	qedn_clear_sgl(conn_ctx->qedn, qedn_task);
+	spin_unlock_irqrestore(&qedn_task->lock, lock_flags);
+
+	spin_lock(&conn_ctx->task_list_lock);
+	list_del(&qedn_task->entry);
+	qedn_host_reset_cccid_itid_entry(conn_ctx, qedn_task->cccid);
+	spin_unlock(&conn_ctx->task_list_lock);
+
+	atomic_dec(&conn_ctx->num_active_tasks);
+	atomic_dec(&conn_ctx->num_active_fw_tasks);
+
+	spin_lock(&io_resrc->resources_lock);
+	list_add_tail(&qedn_task->entry, &io_resrc->task_free_list);
+	io_resrc->num_free_tasks += 1;
+	spin_unlock(&io_resrc->resources_lock);
+}
+
+struct qedn_task_ctx *
+qedn_get_free_task_from_pool(struct qedn_conn_ctx *conn_ctx, u16 cccid)
+{
+	struct qedn_task_ctx *qedn_task = NULL;
+	struct qedn_io_resources *io_resrc;
+	struct qedn_fp_queue *fp_q;
+
+	fp_q = conn_ctx->fp_q;
+	io_resrc = &fp_q->host_resrc;
+
+	spin_lock(&io_resrc->resources_lock);
+	qedn_task = list_first_entry_or_null(&io_resrc->task_free_list,
+					     struct qedn_task_ctx, entry);
+	if (unlikely(!qedn_task)) {
+		spin_unlock(&io_resrc->resources_lock);
+
+		return NULL;
+	}
+	list_del(&qedn_task->entry);
+	io_resrc->num_free_tasks -= 1;
+	spin_unlock(&io_resrc->resources_lock);
+
+	spin_lock(&conn_ctx->task_list_lock);
+	list_add_tail(&qedn_task->entry, &conn_ctx->active_task_list);
+	qedn_host_set_cccid_itid_entry(conn_ctx, cccid, qedn_task->itid);
+	spin_unlock(&conn_ctx->task_list_lock);
+
+	atomic_inc(&conn_ctx->num_active_tasks);
+	qedn_task->cccid = cccid;
+	qedn_task->qedn_conn = conn_ctx;
+	qedn_task->valid = 1;
+
+	return qedn_task;
+}
+
 int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 		       struct nvme_tcp_ofld_req *req)
 {
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:24:05 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:24:05 +0300
Subject: [PATCH v3 15/20] qedn: Add IO level qedn_send_req and fw_cq workqueue
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-16-smalin@marvell.com>

This patch will present the IO level skeleton flows:

- qedn_send_req(): process new requests, similar to nvme_tcp_queue_rq().

- qedn_fw_cq_fp_wq():   process new FW completions, the flow starts from
			the IRQ handler and for a single interrupt it will
			process all the pending NVMeoF Completions under
			polling mode.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
---
 drivers/nvme/hw/qedn/Makefile    |  2 +-
 drivers/nvme/hw/qedn/qedn.h      | 14 ++++++
 drivers/nvme/hw/qedn/qedn_conn.c |  2 +
 drivers/nvme/hw/qedn/qedn_main.c | 84 +++++++++++++++++++++++++++++---
 drivers/nvme/hw/qedn/qedn_task.c | 79 ++++++++++++++++++++++++++++++
 5 files changed, 172 insertions(+), 9 deletions(-)
 create mode 100644 drivers/nvme/hw/qedn/qedn_task.c

diff --git a/drivers/nvme/hw/qedn/Makefile b/drivers/nvme/hw/qedn/Makefile
index ece84772d317..888d466fa5ed 100644
--- a/drivers/nvme/hw/qedn/Makefile
+++ b/drivers/nvme/hw/qedn/Makefile
@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0
 
 obj-$(CONFIG_NVME_QEDN) += qedn.o
-qedn-y := qedn_main.o qedn_conn.o
+qedn-y := qedn_main.o qedn_conn.o qedn_task.o
\ No newline at end of file
diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 9e16297fa323..13d63f420a23 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -38,6 +38,8 @@
 #define QEDN_NON_ABORTIVE_TERMINATION 0
 #define QEDN_ABORTIVE_TERMINATION 1
 
+#define QEDN_FW_CQ_FP_WQ_WORKQUEUE "qedn_fw_cq_fp_wq"
+
 /*
  * TCP offload stack default configurations and defines.
  * Future enhancements will allow controlling the configurable
@@ -90,6 +92,7 @@ struct qedn_fp_queue {
 	struct qedn_ctx	*qedn;
 	struct qed_sb_info *sb_info;
 	unsigned int cpu;
+	struct work_struct fw_cq_fp_wq_entry;
 	u16 sb_id;
 	char irqname[QEDN_IRQ_NAME_LEN];
 };
@@ -118,6 +121,7 @@ struct qedn_ctx {
 	struct qedn_fp_queue *fp_q_arr;
 	struct nvmetcp_glbl_queue_entry *fw_cq_array_virt;
 	dma_addr_t fw_cq_array_phy; /* Physical address of fw_cq_array_virt */
+	struct workqueue_struct *fw_cq_fp_wq;
 };
 
 struct qedn_endpoint {
@@ -204,6 +208,12 @@ struct qedn_ctrl {
 
 /* Connection level struct */
 struct qedn_conn_ctx {
+	/* IO path */
+	struct qedn_fp_queue *fp_q;
+	/* mutex for queueing request */
+	struct mutex send_mutex;
+	int qid;
+
 	struct qedn_ctx *qedn;
 	struct nvme_tcp_ofld_queue *queue;
 	struct nvme_tcp_ofld_ctrl *ctrl;
@@ -264,5 +274,9 @@ int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
 void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx);
 void qedn_cleanp_fw(struct qedn_conn_ctx *conn_ctx);
 __be16 qedn_get_in_port(struct sockaddr_storage *sa);
+int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
+		       struct nvme_tcp_ofld_req *req);
+void qedn_nvme_req_fp_wq_handler(struct work_struct *work);
+void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe);
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
index 04c45a6fa14b..962c88a4f345 100644
--- a/drivers/nvme/hw/qedn/qedn_conn.c
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -190,6 +190,7 @@ static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx)
 		pr_err("Conn resources state isn't 0 as expected 0x%lx\n",
 		       conn_ctx->resrc_state);
 
+	mutex_destroy(&conn_ctx->send_mutex);
 	atomic_inc(&conn_ctx->destroy_conn_indicator);
 	qedn_set_con_state(conn_ctx, CONN_STATE_DESTROY_COMPLETE);
 	wake_up_interruptible(&conn_ctx->conn_waitq);
@@ -429,6 +430,7 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 	}
 
 	set_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state);
+
 	rc = qed_ops->acquire_conn(qedn->cdev,
 				   &conn_ctx->conn_handle,
 				   &conn_ctx->fw_cid,
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 3cadec6d6f5d..975949ce6fb0 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -310,6 +310,7 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 	conn_ctx->queue = queue;
 	conn_ctx->ctrl = ctrl;
 	conn_ctx->sq_depth = queue_size;
+	mutex_init(&conn_ctx->send_mutex);
 
 	init_waitqueue_head(&conn_ctx->conn_waitq);
 	atomic_set(&conn_ctx->est_conn_indicator, 0);
@@ -317,6 +318,8 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 
 	spin_lock_init(&conn_ctx->conn_state_lock);
 
+	conn_ctx->qid = qid;
+
 	qedn_initialize_endpoint(&conn_ctx->ep, qedn->local_mac_addr, ctrl);
 
 	atomic_inc(&qctrl->host_num_active_conns);
@@ -408,9 +411,18 @@ static int qedn_poll_queue(struct nvme_tcp_ofld_queue *queue)
 
 static int qedn_send_req(struct nvme_tcp_ofld_req *req)
 {
-	/* Placeholder - qedn_send_req */
+	struct qedn_conn_ctx *qedn_conn;
+	int rc = 0;
 
-	return 0;
+	qedn_conn = (struct qedn_conn_ctx *)req->queue->private_data;
+	if (unlikely(!qedn_conn))
+		return -ENXIO;
+
+	mutex_lock(&qedn_conn->send_mutex);
+	rc = qedn_queue_request(qedn_conn, req);
+	mutex_unlock(&qedn_conn->send_mutex);
+
+	return rc;
 }
 
 static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
@@ -450,9 +462,58 @@ struct qedn_conn_ctx *qedn_get_conn_hash(struct qedn_ctx *qedn, u16 icid)
 }
 
 /* Fastpath IRQ handler */
+void qedn_fw_cq_fp_handler(struct qedn_fp_queue *fp_q)
+{
+	u16 sb_id, cq_prod_idx, cq_cons_idx;
+	struct qedn_ctx *qedn = fp_q->qedn;
+	struct nvmetcp_fw_cqe *cqe = NULL;
+
+	sb_id = fp_q->sb_id;
+	qed_sb_update_sb_idx(fp_q->sb_info);
+
+	/* rmb - to prevent missing new cqes */
+	rmb();
+
+	/* Read the latest cq_prod from the SB */
+	cq_prod_idx = *fp_q->cq_prod;
+	cq_cons_idx = qed_chain_get_cons_idx(&fp_q->cq_chain);
+
+	while (cq_cons_idx != cq_prod_idx) {
+		cqe = qed_chain_consume(&fp_q->cq_chain);
+		if (likely(cqe))
+			qedn_io_work_cq(qedn, cqe);
+		else
+			pr_err("Failed consuming cqe\n");
+
+		cq_cons_idx = qed_chain_get_cons_idx(&fp_q->cq_chain);
+
+		/* Check if new completions were posted */
+		if (unlikely(cq_prod_idx == cq_cons_idx)) {
+			/* rmb - to prevent missing new cqes */
+			rmb();
+
+			/* Update the latest cq_prod from the SB */
+			cq_prod_idx = *fp_q->cq_prod;
+		}
+	}
+}
+
+static void qedn_fw_cq_fq_wq_handler(struct work_struct *work)
+{
+	struct qedn_fp_queue *fp_q = container_of(work, struct qedn_fp_queue,
+						  fw_cq_fp_wq_entry);
+
+	qedn_fw_cq_fp_handler(fp_q);
+	qed_sb_ack(fp_q->sb_info, IGU_INT_ENABLE, 1);
+}
+
 static irqreturn_t qedn_irq_handler(int irq, void *dev_id)
 {
-	/* Placeholder */
+	struct qedn_fp_queue *fp_q = dev_id;
+	struct qedn_ctx *qedn = fp_q->qedn;
+
+	qed_sb_ack(fp_q->sb_info, IGU_INT_DISABLE, 0);
+	queue_work_on(fp_q->cpu, qedn->fw_cq_fp_wq, &fp_q->fw_cq_fp_wq_entry);
 
 	return IRQ_HANDLED;
 }
@@ -586,6 +647,8 @@ static void qedn_free_function_queues(struct qedn_ctx *qedn)
 	int i;
 
 	/* Free workqueues */
+	destroy_workqueue(qedn->fw_cq_fp_wq);
+	qedn->fw_cq_fp_wq = NULL;
 
 	/* Free the fast path queues*/
 	for (i = 0; i < qedn->num_fw_cqs; i++) {
@@ -653,7 +716,14 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
 	u64 cq_phy_addr;
 	int i;
 
-	/* Place holder - IO-path workqueues */
+	qedn->fw_cq_fp_wq = alloc_workqueue(QEDN_FW_CQ_FP_WQ_WORKQUEUE,
+					    WQ_HIGHPRI | WQ_MEM_RECLAIM, 0);
+	if (!qedn->fw_cq_fp_wq) {
+		rc = -ENODEV;
+		pr_err("Unable to create fastpath FW CQ workqueue!\n");
+
+		return rc;
+	}
 
 	qedn->fp_q_arr = kcalloc(qedn->num_fw_cqs,
 				 sizeof(struct qedn_fp_queue), GFP_KERNEL);
@@ -681,8 +751,7 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
 		chain_params.mode = QED_CHAIN_MODE_PBL,
 		chain_params.cnt_type = QED_CHAIN_CNT_TYPE_U16,
 		chain_params.num_elems = QEDN_FW_CQ_SIZE;
-		/* Placeholder - sizeof(struct nvmetcp_fw_cqe)*/
-		chain_params.elem_size = 64;
+		chain_params.elem_size = sizeof(struct nvmetcp_fw_cqe);
 
 		rc = qed_ops->common->chain_alloc(qedn->cdev,
 						  &fp_q->cq_chain,
@@ -711,8 +780,7 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
 		sb = fp_q->sb_info->sb_virt;
 		fp_q->cq_prod = (u16 *)&sb->pi_array[QEDN_PROTO_CQ_PROD_IDX];
 		fp_q->qedn = qedn;
-
-		/* Placeholder - Init IO-path workqueue */
+		INIT_WORK(&fp_q->fw_cq_fp_wq_entry, qedn_fw_cq_fq_wq_handler);
 
 		/* Placeholder - Init IO-path resources */
 	}
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
new file mode 100644
index 000000000000..a3af55fba95c
--- /dev/null
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -0,0 +1,79 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+ /* Kernel includes */
+#include <linux/kernel.h>
+
+/* Driver includes */
+#include "qedn.h"
+
+int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
+		       struct nvme_tcp_ofld_req *req)
+{
+	/* Process the request */
+
+	return 0;
+}
+
+struct qedn_task_ctx *qedn_cqe_get_active_task(struct nvmetcp_fw_cqe *cqe)
+{
+	struct regpair *p = &cqe->task_opaque;
+
+	return (struct qedn_task_ctx *)((((u64)(le32_to_cpu(p->hi)) << 32)
+					+ le32_to_cpu(p->lo)));
+}
+
+void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
+{
+	struct qedn_task_ctx *qedn_task = NULL;
+	struct qedn_conn_ctx *conn_ctx = NULL;
+	u16 itid;
+	u32 cid;
+
+	conn_ctx = qedn_get_conn_hash(qedn, le16_to_cpu(cqe->conn_id));
+	if (unlikely(!conn_ctx)) {
+		pr_err("CID 0x%x: Failed to fetch conn_ctx from hash\n",
+		       le16_to_cpu(cqe->conn_id));
+
+		return;
+	}
+
+	cid = conn_ctx->fw_cid;
+	itid = le16_to_cpu(cqe->itid);
+	qedn_task = qedn_cqe_get_active_task(cqe);
+	if (unlikely(!qedn_task))
+		return;
+
+	if (likely(cqe->cqe_type == NVMETCP_FW_CQE_TYPE_NORMAL)) {
+		/* Placeholder - verify the connection was established */
+
+		switch (cqe->task_type) {
+		case NVMETCP_TASK_TYPE_HOST_WRITE:
+		case NVMETCP_TASK_TYPE_HOST_READ:
+
+			/* Placeholder - IO flow */
+
+			break;
+
+		case NVMETCP_TASK_TYPE_HOST_READ_NO_CQE:
+
+			/* Placeholder - IO flow */
+
+			break;
+
+		case NVMETCP_TASK_TYPE_INIT_CONN_REQUEST:
+
+			/* Placeholder - ICReq flow */
+
+			break;
+		default:
+			pr_info("Could not identify task type\n");
+		}
+	} else {
+		/* Placeholder - Recovery flows */
+	}
+}
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:24:04 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:24:04 +0300
Subject: [PATCH v3 14/20] qedn: Add support of configuring HW filter block
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-15-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

HW filter can be configured to filter TCP packets based on either
source or target TCP port. QEDN leverage this feature to route
NVMeTCP traffic.

This patch configures HW filter block based on source port for all
receiving packets to deliver correct QEDN PF.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |  15 ++++
 drivers/nvme/hw/qedn/qedn_main.c | 113 ++++++++++++++++++++++++++++++-
 2 files changed, 127 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 38ab4ff88999..9e16297fa323 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -29,6 +29,11 @@
 #define QEDN_IRQ_NAME_LEN 24
 #define QEDN_IRQ_NO_FLAGS 0
 
+/* HW defines */
+
+/* QEDN_MAX_LLH_PORTS will be extended in future */
+#define QEDN_MAX_LLH_PORTS 16
+
 /* Destroy connection defines */
 #define QEDN_NON_ABORTIVE_TERMINATION 0
 #define QEDN_ABORTIVE_TERMINATION 1
@@ -68,6 +73,7 @@
 enum qedn_state {
 	QEDN_STATE_CORE_PROBED = 0,
 	QEDN_STATE_CORE_OPEN,
+	QEDN_STATE_LLH_PORT_FILTER_SET,
 	QEDN_STATE_MFW_STATE,
 	QEDN_STATE_NVMETCP_OPEN,
 	QEDN_STATE_IRQ_SET,
@@ -99,6 +105,8 @@ struct qedn_ctx {
 	/* Accessed with atomic bit ops, used with enum qedn_state */
 	unsigned long state;
 
+	u8 num_llh_filters;
+	struct list_head llh_filter_list;
 	u8 local_mac_addr[ETH_ALEN];
 	u16 mtu;
 
@@ -165,6 +173,12 @@ enum qedn_conn_state {
 	CONN_STATE_DESTROY_COMPLETE
 };
 
+struct qedn_llh_filter {
+	struct list_head entry;
+	u16 port;
+	u16 ref_cnt;
+};
+
 struct qedn_ctrl {
 	struct list_head glb_entry;
 	struct list_head pf_entry;
@@ -249,5 +263,6 @@ int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
 		       enum qedn_conn_state new_state);
 void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx);
 void qedn_cleanp_fw(struct qedn_conn_ctx *conn_ctx);
+__be16 qedn_get_in_port(struct sockaddr_storage *sa);
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index bd5618f65c70..3cadec6d6f5d 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -22,6 +22,86 @@ static struct pci_device_id qedn_pci_tbl[] = {
 	{0, 0},
 };
 
+__be16 qedn_get_in_port(struct sockaddr_storage *sa)
+{
+	return sa->ss_family == AF_INET
+		? ((struct sockaddr_in *)sa)->sin_port
+		: ((struct sockaddr_in6 *)sa)->sin6_port;
+}
+
+struct qedn_llh_filter *qedn_add_llh_filter(struct qedn_ctx *qedn, u16 tcp_port)
+{
+	struct qedn_llh_filter *llh_filter = NULL;
+	struct qedn_llh_filter *llh_tmp = NULL;
+	bool new_filter = 1;
+	int rc = 0;
+
+	/* Check if LLH filter already defined */
+	list_for_each_entry_safe(llh_filter, llh_tmp, &qedn->llh_filter_list,
+				 entry) {
+		if (llh_filter->port == tcp_port) {
+			new_filter = 0;
+			llh_filter->ref_cnt++;
+			break;
+		}
+	}
+
+	if (new_filter) {
+		if (qedn->num_llh_filters >= QEDN_MAX_LLH_PORTS) {
+			pr_err("PF reached the max target ports limit %u. %u\n",
+			       qedn->dev_info.common.abs_pf_id,
+			       qedn->num_llh_filters);
+
+			return NULL;
+		}
+
+		rc = qed_ops->add_src_tcp_port_filter(qedn->cdev, tcp_port);
+		if (rc) {
+			pr_err("LLH port config failed. port:%u; rc:%u\n",
+			       tcp_port, rc);
+
+			return NULL;
+		}
+
+		llh_filter = kzalloc(sizeof(*llh_filter), GFP_KERNEL);
+		if (!llh_filter) {
+			qed_ops->remove_src_tcp_port_filter(qedn->cdev,
+							    tcp_port);
+
+			return NULL;
+		}
+
+		llh_filter->port = tcp_port;
+		llh_filter->ref_cnt = 1;
+		++qedn->num_llh_filters;
+		list_add_tail(&llh_filter->entry, &qedn->llh_filter_list);
+		set_bit(QEDN_STATE_LLH_PORT_FILTER_SET, &qedn->state);
+	}
+
+	return llh_filter;
+}
+
+void qedn_dec_llh_filter(struct qedn_ctx *qedn,
+			 struct qedn_llh_filter *llh_filter)
+{
+	if (!llh_filter)
+		return;
+
+	llh_filter->ref_cnt--;
+	if (!llh_filter->ref_cnt) {
+		list_del(&llh_filter->entry);
+
+		/* Remove LLH protocol port filter */
+		qed_ops->remove_src_tcp_port_filter(qedn->cdev,
+						    llh_filter->port);
+
+		--qedn->num_llh_filters;
+		kfree(llh_filter);
+		if (!qedn->num_llh_filters)
+			clear_bit(QEDN_STATE_LLH_PORT_FILTER_SET, &qedn->state);
+	}
+}
+
 static bool qedn_matches_qede(struct qedn_ctx *qedn, struct pci_dev *qede_pdev)
 {
 	struct pci_dev *qedn_pdev = qedn->pdev;
@@ -107,8 +187,10 @@ qedn_claim_dev(struct nvme_tcp_ofld_dev *dev,
 static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 {
 	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+	struct qedn_llh_filter *llh_filter = NULL;
 	struct qedn_ctrl *qctrl = NULL;
 	struct qedn_ctx *qedn = NULL;
+	__be16 remote_port;
 	bool new = true;
 	int rc = 0;
 
@@ -147,7 +229,22 @@ static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 	qedn = container_of(dev, struct qedn_ctx, qedn_ofld_dev);
 	qctrl->qedn = qedn;
 
-	/* Placeholder - setup LLH filter */
+	if (qedn->num_llh_filters == 0) {
+		qedn->mtu = ctrl->ndev->mtu;
+		memcpy(qedn->local_mac_addr, ctrl->ndev->dev_addr, ETH_ALEN);
+	}
+
+	remote_port = qedn_get_in_port(&ctrl->conn_params.remote_ip_addr);
+	if (new) {
+		llh_filter = qedn_add_llh_filter(qedn, ntohs(remote_port));
+		if (!llh_filter) {
+			rc = -EFAULT;
+			goto err_out;
+		}
+
+		qctrl->llh_filter = llh_filter;
+		set_bit(LLH_FILTER, &qctrl->agg_state);
+	}
 
 	return 0;
 err_out:
@@ -165,6 +262,12 @@ static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 	if (!qctrl)
 		return -ENODEV;
 
+	if (test_and_clear_bit(LLH_FILTER, &qctrl->agg_state) &&
+	    qctrl->llh_filter) {
+		qedn_dec_llh_filter(qctrl->qedn, qctrl->llh_filter);
+		qctrl->llh_filter = NULL;
+	}
+
 	if (test_and_clear_bit(QEDN_STATE_SP_WORK_THREAD_SET,
 			       &qctrl->agg_state))
 		flush_workqueue(qctrl->sp_wq);
@@ -446,6 +549,8 @@ static int qedn_setup_irq(struct qedn_ctx *qedn)
 
 static inline void qedn_init_pf_struct(struct qedn_ctx *qedn)
 {
+	INIT_LIST_HEAD(&qedn->llh_filter_list);
+	qedn->num_llh_filters = 0;
 	hash_init(qedn->conn_ctx_hash);
 }
 
@@ -688,6 +793,12 @@ static void __qedn_remove(struct pci_dev *pdev)
 		return;
 	}
 
+	if (test_and_clear_bit(QEDN_STATE_LLH_PORT_FILTER_SET, &qedn->state)) {
+		pr_err("LLH port configuration removal. %d filters still set\n",
+		       qedn->num_llh_filters);
+		qed_ops->clear_all_filters(qedn->cdev);
+	}
+
 	if (test_and_clear_bit(QEDN_STATE_REGISTERED_OFFLOAD_DEV, &qedn->state))
 		nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev);
 
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:24:08 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:24:08 +0300
Subject: [PATCH v3 18/20] qedn: Add IO level fastpath functionality
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-19-smalin@marvell.com>

This patch will present the IO level functionality of qedn
nvme-tcp-offload host mode. The qedn_task_ctx structure is containing
various params and state of the current IO, and is mapped 1x1 to the
fw_task_ctx which is a HW and FW IO context.
A qedn_task is mapped directly to its parent connection.
For every new IO a qedn_task structure will be assigned and they will be
linked for the entire IO's life span.

The patch will include 2 flows:
  1. Send new command to the FW:
	 The flow is: nvme_tcp_ofld_queue_rq() which invokes qedn_send_req()
	 which invokes qedn_queue_request() which will:
         - Assign fw_task_ctx.
	 - Prepare the Read/Write SG buffer.
	 - Initialize the HW and FW context.
	 - Pass the IO to the FW.

  2. Process the IO completion:
     The flow is: qedn_irq_handler() which invokes qedn_fw_cq_fp_handler()
	 which invokes qedn_io_work_cq() which will:
	 - process the FW completion.
	 - Return the fw_task_ctx to the task pool.
	 - complete the nvme req.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |   5 +
 drivers/nvme/hw/qedn/qedn_conn.c |   1 +
 drivers/nvme/hw/qedn/qedn_main.c |   8 +
 drivers/nvme/hw/qedn/qedn_task.c | 317 ++++++++++++++++++++++++++++++-
 4 files changed, 327 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 829d474b3ab1..b36994be65cb 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -172,6 +172,10 @@ struct qedn_ctx {
 	struct qed_nvmetcp_tid	tasks;
 };
 
+enum qedn_task_flags {
+	QEDN_TASK_USED_BY_FW,
+};
+
 struct qedn_task_ctx {
 	struct qedn_conn_ctx *qedn_conn;
 	struct qedn_ctx *qedn;
@@ -376,6 +380,7 @@ void qedn_destroy_free_tasks(struct qedn_fp_queue *fp_q,
 			     struct qedn_io_resources *io_resrc);
 void qedn_prep_icresp(struct qedn_conn_ctx *conn_ctx,
 		      struct nvmetcp_fw_cqe *cqe);
+void qedn_swap_bytes(u32 *p, int size);
 void qedn_ring_doorbell(struct qedn_conn_ctx *conn_ctx);
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
index b4c0a1a3e890..ea072eff34a6 100644
--- a/drivers/nvme/hw/qedn/qedn_conn.c
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -528,6 +528,7 @@ static int qedn_send_icreq(struct qedn_conn_ctx *conn_ctx)
 				     sgl_task_params,  NULL);
 
 	qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_IC_COMP);
+	set_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags);
 	atomic_inc(&conn_ctx->num_active_fw_tasks);
 
 	/* spin_lock - doorbell is accessed  both Rx flow and response flow */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 3cf913d527c0..fb47e315ab03 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -1047,6 +1047,14 @@ static int qedn_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	return __qedn_probe(pdev);
 }
 
+void qedn_swap_bytes(u32 *p, int size)
+{
+	int i;
+
+	for (i = 0; i < size; ++i, ++p)
+		*p = __swab32(*p);
+}
+
 static struct pci_driver qedn_pci_driver = {
 	.name     = QEDN_MODULE_NAME,
 	.id_table = qedn_pci_tbl,
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
index e52460f9650e..dd0b5f31c052 100644
--- a/drivers/nvme/hw/qedn/qedn_task.c
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -11,6 +11,77 @@
 /* Driver includes */
 #include "qedn.h"
 
+extern const struct qed_nvmetcp_ops *qed_ops;
+
+static bool qedn_sgl_has_small_mid_sge(struct nvmetcp_sge *sgl, u16 sge_count)
+{
+	u16 sge_num;
+
+	if (sge_count > 8) {
+		for (sge_num = 0; sge_num < sge_count; sge_num++) {
+			if (le32_to_cpu(sgl[sge_num].sge_len) <
+			    QEDN_FW_SLOW_IO_MIN_SGE_LIMIT)
+				return true; /* small middle SGE found */
+		}
+	}
+
+	return false; /* no small middle SGEs */
+}
+
+static int qedn_init_sgl(struct qedn_ctx *qedn, struct qedn_task_ctx *qedn_task)
+{
+	struct storage_sgl_task_params *sgl_task_params;
+	enum dma_data_direction dma_dir;
+	struct scatterlist *sg;
+	struct request *rq;
+	u16 num_sges;
+	int index;
+	u32 len;
+	int rc;
+
+	sgl_task_params = &qedn_task->sgl_task_params;
+	rq = blk_mq_rq_from_pdu(qedn_task->req);
+	if (qedn_task->task_size == 0) {
+		sgl_task_params->num_sges = 0;
+
+		return 0;
+	}
+
+	/* Convert BIO to scatterlist */
+	num_sges = blk_rq_map_sg(rq->q, rq, qedn_task->nvme_sg);
+	if (qedn_task->req_direction == WRITE)
+		dma_dir = DMA_TO_DEVICE;
+	else
+		dma_dir = DMA_FROM_DEVICE;
+
+	/* DMA map the scatterlist */
+	if (dma_map_sg(&qedn->pdev->dev, qedn_task->nvme_sg,
+		       num_sges, dma_dir) != num_sges) {
+		pr_err("Couldn't map sgl\n");
+		rc = -EPERM;
+
+		return rc;
+	}
+
+	sgl_task_params->total_buffer_size = qedn_task->task_size;
+	sgl_task_params->num_sges = num_sges;
+
+	for_each_sg(qedn_task->nvme_sg, sg, num_sges, index) {
+		DMA_REGPAIR_LE(sgl_task_params->sgl[index].sge_addr,
+			       sg_dma_address(sg));
+		len = sg_dma_len(sg);
+		sgl_task_params->sgl[index].sge_len = cpu_to_le32(len);
+	}
+
+	/* Relevant for Host Write Only */
+	sgl_task_params->small_mid_sge = (qedn_task->req_direction == READ) ?
+		false :
+		qedn_sgl_has_small_mid_sge(sgl_task_params->sgl,
+					   sgl_task_params->num_sges);
+
+	return 0;
+}
+
 static void qedn_free_nvme_sg(struct qedn_task_ctx *qedn_task)
 {
 	kfree(qedn_task->nvme_sg);
@@ -337,12 +408,168 @@ qedn_get_free_task_from_pool(struct qedn_conn_ctx *conn_ctx, u16 cccid)
 	return qedn_task;
 }
 
+int qedn_send_read_cmd(struct qedn_task_ctx *qedn_task,
+		       struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_command *nvme_cmd = &qedn_task->req->nvme_cmd;
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct nvmetcp_task_params task_params;
+	struct nvme_tcp_cmd_pdu cmd_hdr;
+	struct nvmetcp_wqe *chain_sqe;
+	struct nvmetcp_wqe local_sqe;
+	int rc;
+
+	rc = qedn_init_sgl(qedn, qedn_task);
+	if (rc)
+		return rc;
+
+	task_params.opq.lo = cpu_to_le32(((u64)(qedn_task)) & 0xffffffff);
+	task_params.opq.hi = cpu_to_le32(((u64)(qedn_task)) >> 32);
+
+	/* Initialize task params */
+	task_params.context = qedn_task->fw_task_ctx;
+	task_params.sqe = &local_sqe;
+	task_params.tx_io_size = 0;
+	task_params.rx_io_size = qedn_task->task_size;
+	task_params.conn_icid = (u16)conn_ctx->conn_handle;
+	task_params.itid = qedn_task->itid;
+	task_params.cq_rss_number = conn_ctx->default_cq;
+	task_params.send_write_incapsule = 0;
+
+	cmd_hdr.hdr.type = nvme_tcp_cmd;
+	cmd_hdr.hdr.flags = 0;
+	cmd_hdr.hdr.hlen = sizeof(cmd_hdr);
+	cmd_hdr.hdr.pdo = 0x0;
+	cmd_hdr.hdr.plen = cpu_to_le32(cmd_hdr.hdr.hlen);
+
+	qed_ops->init_read_io(&task_params, &cmd_hdr, nvme_cmd,
+			      &qedn_task->sgl_task_params);
+
+	set_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags);
+	atomic_inc(&conn_ctx->num_active_fw_tasks);
+
+	spin_lock(&conn_ctx->ep.doorbell_lock);
+	chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain);
+	memcpy(chain_sqe, &local_sqe, sizeof(local_sqe));
+	qedn_ring_doorbell(conn_ctx);
+	spin_unlock(&conn_ctx->ep.doorbell_lock);
+
+	return 0;
+}
+
+int qedn_send_write_cmd(struct qedn_task_ctx *qedn_task,
+			struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_command *nvme_cmd = &qedn_task->req->nvme_cmd;
+	struct nvmetcp_task_params task_params;
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct nvme_tcp_cmd_pdu cmd_hdr;
+	u32 pdu_len = sizeof(cmd_hdr);
+	struct nvmetcp_wqe *chain_sqe;
+	struct nvmetcp_wqe local_sqe;
+	u8 send_write_incapsule;
+	int rc;
+
+	if (qedn_task->task_size <=
+	    nvme_tcp_ofld_inline_data_size(conn_ctx->queue) &&
+	    qedn_task->task_size) {
+		send_write_incapsule = 1;
+		pdu_len += qedn_task->task_size;
+
+		/* Add digest length once supported */
+		cmd_hdr.hdr.pdo = sizeof(cmd_hdr);
+	} else {
+		send_write_incapsule = 0;
+
+		cmd_hdr.hdr.pdo = 0x0;
+	}
+
+	rc = qedn_init_sgl(qedn, qedn_task);
+	if (rc)
+		return rc;
+
+	task_params.host_cccid = cpu_to_le16(qedn_task->cccid);
+	task_params.opq.lo = cpu_to_le32(((u64)(qedn_task)) & 0xffffffff);
+	task_params.opq.hi = cpu_to_le32(((u64)(qedn_task)) >> 32);
+
+	/* Initialize task params */
+	task_params.context = qedn_task->fw_task_ctx;
+	task_params.sqe = &local_sqe;
+	task_params.tx_io_size = qedn_task->task_size;
+	task_params.rx_io_size = 0;
+	task_params.conn_icid = (u16)conn_ctx->conn_handle;
+	task_params.itid = qedn_task->itid;
+	task_params.cq_rss_number = conn_ctx->default_cq;
+	task_params.send_write_incapsule = send_write_incapsule;
+
+	cmd_hdr.hdr.type = nvme_tcp_cmd;
+	cmd_hdr.hdr.flags = 0;
+	cmd_hdr.hdr.hlen = sizeof(cmd_hdr);
+	cmd_hdr.hdr.plen = cpu_to_le32(pdu_len);
+
+	qed_ops->init_write_io(&task_params, &cmd_hdr, nvme_cmd,
+			       &qedn_task->sgl_task_params);
+
+	set_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags);
+	atomic_inc(&conn_ctx->num_active_fw_tasks);
+
+	spin_lock(&conn_ctx->ep.doorbell_lock);
+	chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain);
+	memcpy(chain_sqe, &local_sqe, sizeof(local_sqe));
+	qedn_ring_doorbell(conn_ctx);
+	spin_unlock(&conn_ctx->ep.doorbell_lock);
+
+	return 0;
+}
+
 int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 		       struct nvme_tcp_ofld_req *req)
 {
-	/* Process the request */
+	struct qedn_task_ctx *qedn_task;
+	struct request *rq;
+	int rc = 0;
+	u16 cccid;
 
-	return 0;
+	rq = blk_mq_rq_from_pdu(req);
+
+	/* Placeholder - async */
+
+	cccid = rq->tag;
+	qedn_task = qedn_get_free_task_from_pool(qedn_conn, cccid);
+	if (unlikely(!qedn_task)) {
+		pr_err("Not able to allocate task context resource\n");
+
+		return BLK_STS_NOTSUPP;
+	}
+
+	req->private_data = qedn_task;
+	qedn_task->req = req;
+
+	/* Placeholder - handle (req->async) */
+
+	/* Check if there are physical segments in request to determine the
+	 * task size. The logic of nvme_tcp_set_sg_null() will be implemented
+	 * as a part of qedn_set_sg_host_data().
+	 */
+	qedn_task->task_size = blk_rq_nr_phys_segments(rq) ?
+				blk_rq_payload_bytes(rq) : 0;
+	qedn_task->req_direction = rq_data_dir(rq);
+	if (qedn_task->req_direction == WRITE)
+		rc = qedn_send_write_cmd(qedn_task, qedn_conn);
+	else
+		rc = qedn_send_read_cmd(qedn_task, qedn_conn);
+
+	if (unlikely(rc)) {
+		pr_err("Read/Write command failure\n");
+
+		return BLK_STS_TRANSPORT;
+	}
+
+	spin_lock(&qedn_conn->ep.doorbell_lock);
+	qedn_ring_doorbell(qedn_conn);
+	spin_unlock(&qedn_conn->ep.doorbell_lock);
+
+	return BLK_STS_OK;
 }
 
 struct qedn_task_ctx *qedn_cqe_get_active_task(struct nvmetcp_fw_cqe *cqe)
@@ -353,8 +580,75 @@ struct qedn_task_ctx *qedn_cqe_get_active_task(struct nvmetcp_fw_cqe *cqe)
 					+ le32_to_cpu(p->lo)));
 }
 
+static struct nvme_tcp_ofld_req *qedn_decouple_req_task(struct qedn_task_ctx
+							*qedn_task)
+{
+	struct nvme_tcp_ofld_req *ulp_req = qedn_task->req;
+
+	qedn_task->req = NULL;
+	if (ulp_req)
+		ulp_req->private_data = NULL;
+
+	return ulp_req;
+}
+
+static inline int qedn_comp_valid_task(struct qedn_task_ctx *qedn_task,
+				       union nvme_result *result, __le16 status)
+{
+	struct qedn_conn_ctx *conn_ctx = qedn_task->qedn_conn;
+	struct nvme_tcp_ofld_req *req;
+
+	req = qedn_decouple_req_task(qedn_task);
+	qedn_return_task_to_pool(conn_ctx, qedn_task);
+	if (!req) {
+		pr_err("req not found\n");
+
+		return -EINVAL;
+	}
+
+	/* Call request done to complete the request */
+	if (req->done)
+		req->done(req, result, status);
+	else
+		pr_err("request done not Set !!!\n");
+
+	return 0;
+}
+
+int qedn_process_nvme_cqe(struct qedn_task_ctx *qedn_task,
+			  struct nvme_completion *cqe)
+{
+	int rc = 0;
+
+	/* CQE arrives swapped
+	 * Swapping requirement will be removed in future FW versions
+	 */
+	qedn_swap_bytes((u32 *)cqe, (sizeof(*cqe) / sizeof(u32)));
+
+	/* Placeholder - async */
+
+	rc = qedn_comp_valid_task(qedn_task, &cqe->result, cqe->status);
+
+	return rc;
+}
+
+int qedn_complete_c2h(struct qedn_task_ctx *qedn_task)
+{
+	int rc = 0;
+
+	__le16 status = cpu_to_le16(NVME_SC_SUCCESS << 1);
+	union nvme_result result = {};
+
+	rc = qedn_comp_valid_task(qedn_task, &result, status);
+
+	return rc;
+}
+
 void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
 {
+	int rc = 0;
+
+	struct nvme_completion *nvme_cqe = NULL;
 	struct qedn_task_ctx *qedn_task = NULL;
 	struct qedn_conn_ctx *conn_ctx = NULL;
 	u16 itid;
@@ -381,13 +675,28 @@ void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
 		case NVMETCP_TASK_TYPE_HOST_WRITE:
 		case NVMETCP_TASK_TYPE_HOST_READ:
 
-			/* Placeholder - IO flow */
+			/* Verify data digest once supported */
 
+			nvme_cqe = (struct nvme_completion *)
+						&cqe->cqe_data.nvme_cqe;
+			rc = qedn_process_nvme_cqe(qedn_task, nvme_cqe);
+			if (rc) {
+				pr_err("Read/Write completion error\n");
+
+				return;
+			}
 			break;
 
 		case NVMETCP_TASK_TYPE_HOST_READ_NO_CQE:
 
-			/* Placeholder - IO flow */
+			/* Verify data digest once supported */
+
+			rc = qedn_complete_c2h(qedn_task);
+			if (rc) {
+				pr_err("Controller To Host Data Transfer error error\n");
+
+				return;
+			}
 
 			break;
 
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:24:03 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:24:03 +0300
Subject: [PATCH v3 13/20] qedn: Add connection-level slowpath functionality
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-14-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

This patch will present the connection (queue) level slowpath
implementation relevant for create_queue flow.

The internal implementation:
- Add per controller slowpath workqeueue via pre_setup_ctrl

- qedn_main.c:
  Includes qedn's implementation of the create_queue op.

- qedn_conn.c will include main slowpath connection level functions,
  including:
    1. Per-queue resources allocation.
    2. Creating a new connection.
    3. Offloading the connection to the FW for TCP handshake.
    4. Destroy of a connection.
    5. Support of delete and free controller.
    6. TCP port management via qed_fetch_tcp_port, qed_return_tcp_port

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/Makefile    |   5 +-
 drivers/nvme/hw/qedn/qedn.h      | 179 ++++++++++
 drivers/nvme/hw/qedn/qedn_conn.c | 564 +++++++++++++++++++++++++++++++
 drivers/nvme/hw/qedn/qedn_main.c | 210 +++++++++++-
 4 files changed, 948 insertions(+), 10 deletions(-)
 create mode 100644 drivers/nvme/hw/qedn/qedn_conn.c

diff --git a/drivers/nvme/hw/qedn/Makefile b/drivers/nvme/hw/qedn/Makefile
index 1422cd878680..ece84772d317 100644
--- a/drivers/nvme/hw/qedn/Makefile
+++ b/drivers/nvme/hw/qedn/Makefile
@@ -1,5 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0
 
-obj-$(CONFIG_NVME_QEDN) := qedn.o
-
-qedn-y := qedn_main.o
+obj-$(CONFIG_NVME_QEDN) += qedn.o
+qedn-y := qedn_main.o qedn_conn.o
diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index edb0836bca87..38ab4ff88999 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -6,6 +6,7 @@
 #ifndef _QEDN_H_
 #define _QEDN_H_
 
+#include <linux/qed/common_hsi.h>
 #include <linux/qed/qed_if.h>
 #include <linux/qed/qed_nvmetcp_if.h>
 #include <linux/qed/qed_nvmetcp_ip_services_if.h>
@@ -28,7 +29,41 @@
 #define QEDN_IRQ_NAME_LEN 24
 #define QEDN_IRQ_NO_FLAGS 0
 
+/* Destroy connection defines */
+#define QEDN_NON_ABORTIVE_TERMINATION 0
+#define QEDN_ABORTIVE_TERMINATION 1
+
+/*
+ * TCP offload stack default configurations and defines.
+ * Future enhancements will allow controlling the configurable
+ * parameters via devlink.
+ */
 #define QEDN_TCP_RTO_DEFAULT 280
+#define QEDN_TCP_ECN_EN 0
+#define QEDN_TCP_TS_EN 0
+#define QEDN_TCP_DA_EN 0
+#define QEDN_TCP_KA_EN 0
+#define QEDN_TCP_TOS 0
+#define QEDN_TCP_TTL 0xfe
+#define QEDN_TCP_FLOW_LABEL 0
+#define QEDN_TCP_KA_TIMEOUT 7200000
+#define QEDN_TCP_KA_INTERVAL 10000
+#define QEDN_TCP_KA_MAX_PROBE_COUNT 10
+#define QEDN_TCP_MAX_RT_TIME 30000
+#define QEDN_TCP_MAX_CWND 4
+#define QEDN_TCP_RCV_WND_SCALE 2
+#define QEDN_TCP_TS_OPTION_LEN 12
+
+/* SP Work queue defines */
+#define QEDN_SP_WORKQUEUE "qedn_sp_wq"
+#define QEDN_SP_WORKQUEUE_MAX_ACTIVE 1
+
+#define QEDN_HOST_MAX_SQ_SIZE (512)
+#define QEDN_SQ_SIZE (2 * QEDN_HOST_MAX_SQ_SIZE)
+
+/* Timeouts and delay constants */
+#define QEDN_WAIT_CON_ESTABLSH_TMO 10000 /* 10 seconds */
+#define QEDN_RLS_CONS_TMO 5000 /* 5 sec */
 
 enum qedn_state {
 	QEDN_STATE_CORE_PROBED = 0,
@@ -64,6 +99,12 @@ struct qedn_ctx {
 	/* Accessed with atomic bit ops, used with enum qedn_state */
 	unsigned long state;
 
+	u8 local_mac_addr[ETH_ALEN];
+	u16 mtu;
+
+	/* Connections */
+	DECLARE_HASHTABLE(conn_ctx_hash, 16);
+
 	/* Fast path queues */
 	u8 num_fw_cqs;
 	struct qedn_fp_queue *fp_q_arr;
@@ -71,4 +112,142 @@ struct qedn_ctx {
 	dma_addr_t fw_cq_array_phy; /* Physical address of fw_cq_array_virt */
 };
 
+struct qedn_endpoint {
+	/* FW Params */
+	struct qed_chain fw_sq_chain;
+	struct nvmetcp_db_data db_data;
+	void __iomem *p_doorbell;
+
+	/* TCP Params */
+	__be32 dst_addr[4]; /* In network order */
+	__be32 src_addr[4]; /* In network order */
+	u16 src_port;
+	u16 dst_port;
+	u16 vlan_id;
+	u8 src_mac[ETH_ALEN];
+	u8 dst_mac[ETH_ALEN];
+	u8 ip_type;
+};
+
+enum sp_work_agg_action {
+	CREATE_CONNECTION = 0,
+	SEND_ICREQ,
+	HANDLE_ICRESP,
+	DESTROY_CONNECTION,
+};
+
+enum qedn_ctrl_agg_state {
+	QEDN_CTRL_SET_TO_OFLD_CTRL = 0, /* CTRL set to OFLD_CTRL */
+	QEDN_STATE_SP_WORK_THREAD_SET, /* slow patch WQ was created*/
+	LLH_FILTER, /* LLH filter added */
+	QEDN_RECOVERY,
+	ADMINQ_CONNECTED, /* At least one connection has attempted offload */
+	ERR_FLOW,
+};
+
+enum qedn_ctrl_sp_wq_state {
+	QEDN_CTRL_STATE_UNINITIALIZED = 0,
+	QEDN_CTRL_STATE_FREE_CTRL,
+	QEDN_CTRL_STATE_CTRL_ERR,
+};
+
+/* Any change to this enum requires an update of qedn_conn_state_str */
+enum qedn_conn_state {
+	CONN_STATE_CONN_IDLE = 0,
+	CONN_STATE_CREATE_CONNECTION,
+	CONN_STATE_WAIT_FOR_CONNECT_DONE,
+	CONN_STATE_OFFLOAD_COMPLETE,
+	CONN_STATE_WAIT_FOR_UPDATE_EQE,
+	CONN_STATE_WAIT_FOR_IC_COMP,
+	CONN_STATE_NVMETCP_CONN_ESTABLISHED,
+	CONN_STATE_DESTROY_CONNECTION,
+	CONN_STATE_WAIT_FOR_DESTROY_DONE,
+	CONN_STATE_DESTROY_COMPLETE
+};
+
+struct qedn_ctrl {
+	struct list_head glb_entry;
+	struct list_head pf_entry;
+
+	struct qedn_ctx *qedn;
+	struct nvme_tcp_ofld_queue *queue;
+	struct nvme_tcp_ofld_ctrl *ctrl;
+
+	struct sockaddr remote_mac_addr;
+	u16 vlan_id;
+
+	struct workqueue_struct *sp_wq;
+	enum qedn_ctrl_sp_wq_state sp_wq_state;
+
+	struct work_struct sp_wq_entry;
+
+	struct qedn_llh_filter *llh_filter;
+
+	unsigned long agg_state;
+
+	atomic_t host_num_active_conns;
+};
+
+/* Connection level struct */
+struct qedn_conn_ctx {
+	struct qedn_ctx *qedn;
+	struct nvme_tcp_ofld_queue *queue;
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	u32 conn_handle;
+	u32 fw_cid;
+
+	atomic_t est_conn_indicator;
+	atomic_t destroy_conn_indicator;
+	wait_queue_head_t conn_waitq;
+
+	struct work_struct sp_wq_entry;
+
+	/* Connection aggregative state.
+	 * Can have different states independently.
+	 */
+	unsigned long agg_work_action;
+
+	struct hlist_node hash_node;
+	struct nvmetcp_host_cccid_itid_entry *host_cccid_itid;
+	dma_addr_t host_cccid_itid_phy_addr;
+	struct qedn_endpoint ep;
+	int abrt_flag;
+
+	/* Connection resources - turned on to indicate what resource was
+	 * allocated, to that it can later be released.
+	 */
+	unsigned long resrc_state;
+
+	/* Connection state */
+	spinlock_t conn_state_lock;
+	enum qedn_conn_state state;
+
+	size_t sq_depth;
+
+	/* "dummy" socket */
+	struct socket *sock;
+};
+
+enum qedn_conn_resources_state {
+	QEDN_CONN_RESRC_FW_SQ,
+	QEDN_CONN_RESRC_ACQUIRE_CONN,
+	QEDN_CONN_RESRC_CCCID_ITID_MAP,
+	QEDN_CONN_RESRC_TCP_PORT,
+	QEDN_CONN_RESRC_DB_ADD,
+	QEDN_CONN_RESRC_MAX = 64
+};
+
+struct qedn_conn_ctx *qedn_get_conn_hash(struct qedn_ctx *qedn, u16 icid);
+int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data);
+void qedn_sp_wq_handler(struct work_struct *work);
+void qedn_set_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit);
+void qedn_clr_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit);
+int qedn_initialize_endpoint(struct qedn_endpoint *ep, u8 *local_mac_addr,
+			     struct nvme_tcp_ofld_ctrl *ctrl);
+int qedn_wait_for_conn_est(struct qedn_conn_ctx *conn_ctx);
+int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
+		       enum qedn_conn_state new_state);
+void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx);
+void qedn_cleanp_fw(struct qedn_conn_ctx *conn_ctx);
+
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
new file mode 100644
index 000000000000..04c45a6fa14b
--- /dev/null
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -0,0 +1,564 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+ /* Kernel includes */
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <net/tcp.h>
+
+/* Driver includes */
+#include "qedn.h"
+
+extern const struct qed_nvmetcp_ops *qed_ops;
+
+static const char * const qedn_conn_state_str[] = {
+	"CONN_IDLE",
+	"CREATE_CONNECTION",
+	"WAIT_FOR_CONNECT_DONE",
+	"OFFLOAD_COMPLETE",
+	"WAIT_FOR_UPDATE_EQE",
+	"WAIT_FOR_IC_COMP",
+	"NVMETCP_CONN_ESTABLISHED",
+	"DESTROY_CONNECTION",
+	"WAIT_FOR_DESTROY_DONE",
+	"DESTROY_COMPLETE",
+	NULL
+};
+
+int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
+		       enum qedn_conn_state new_state)
+{
+	spin_lock_bh(&conn_ctx->conn_state_lock);
+	conn_ctx->state = new_state;
+	spin_unlock_bh(&conn_ctx->conn_state_lock);
+
+	return 0;
+}
+
+static void qedn_return_tcp_port(struct qedn_conn_ctx *conn_ctx)
+{
+	if (conn_ctx->sock && conn_ctx->sock->sk) {
+		qed_return_tcp_port(conn_ctx->sock);
+		conn_ctx->sock = NULL;
+	}
+
+	conn_ctx->ep.src_port = 0;
+}
+
+int qedn_wait_for_conn_est(struct qedn_conn_ctx *conn_ctx)
+{
+	u32 est_timeout = msecs_to_jiffies(QEDN_WAIT_CON_ESTABLSH_TMO);
+	atomic_t *conn_est_ind = &conn_ctx->est_conn_indicator;
+	int wrc, rc;
+
+	wrc = wait_event_interruptible_timeout(conn_ctx->conn_waitq,
+					       atomic_read(conn_est_ind) > 0,
+					       est_timeout);
+	atomic_set(conn_est_ind, 0);
+	if (!wrc ||
+	    conn_ctx->state != CONN_STATE_NVMETCP_CONN_ESTABLISHED) {
+		rc = -ETIMEDOUT;
+
+		/* If error was prior or during offload, conn_ctx was released.
+		 * If the error was after offload sync has completed, we need to
+		 * terminate the connection ourselves.
+		 */
+		if (conn_ctx &&
+		    conn_ctx->state >= CONN_STATE_WAIT_FOR_CONNECT_DONE &&
+		    conn_ctx->state <= CONN_STATE_NVMETCP_CONN_ESTABLISHED)
+			qedn_terminate_connection(conn_ctx);
+	} else {
+		rc = 0;
+	}
+
+	return rc;
+}
+
+int qedn_fill_ep_addr4(struct qedn_endpoint *ep,
+		       struct nvme_tcp_ofld_ctrl_con_params *conn_params)
+{
+	struct sockaddr_in *raddr, *laddr;
+
+	raddr = (struct sockaddr_in *)&conn_params->remote_ip_addr;
+	laddr = (struct sockaddr_in *)&conn_params->local_ip_addr;
+
+	ep->ip_type = TCP_IPV4;
+	ep->src_port = laddr->sin_port;
+	ep->dst_port = ntohs(raddr->sin_port);
+
+	ep->src_addr[0] = laddr->sin_addr.s_addr;
+	ep->dst_addr[0] = raddr->sin_addr.s_addr;
+
+	return 0;
+}
+
+int qedn_fill_ep_addr6(struct qedn_endpoint *ep,
+		       struct nvme_tcp_ofld_ctrl_con_params *conn_params)
+{
+	struct sockaddr_in6 *raddr6, *laddr6;
+	int i;
+
+	raddr6 = (struct sockaddr_in6 *)&conn_params->remote_ip_addr;
+	laddr6 = (struct sockaddr_in6 *)&conn_params->local_ip_addr;
+
+	ep->ip_type = TCP_IPV6;
+	ep->src_port = laddr6->sin6_port;
+	ep->dst_port = ntohs(raddr6->sin6_port);
+
+	for (i = 0; i < 4; i++) {
+		ep->src_addr[i] = laddr6->sin6_addr.in6_u.u6_addr32[i];
+		ep->dst_addr[i] = raddr6->sin6_addr.in6_u.u6_addr32[i];
+	}
+
+	return 0;
+}
+
+int qedn_initialize_endpoint(struct qedn_endpoint *ep, u8 *local_mac_addr,
+			     struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	struct nvme_tcp_ofld_ctrl_con_params *conn_params = &ctrl->conn_params;
+	struct qedn_ctrl *qctrl;
+
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return -ENODEV;
+
+	ether_addr_copy(ep->dst_mac, qctrl->remote_mac_addr.sa_data);
+	ether_addr_copy(ep->src_mac, local_mac_addr);
+	ep->vlan_id = qctrl->vlan_id;
+	if (conn_params->remote_ip_addr.ss_family == AF_INET)
+		qedn_fill_ep_addr4(ep, conn_params);
+	else
+		qedn_fill_ep_addr6(ep, conn_params);
+
+	return -1;
+}
+
+static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int rc = 0;
+
+	if (test_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state)) {
+		qed_ops->common->chain_free(qedn->cdev,
+					    &conn_ctx->ep.fw_sq_chain);
+		clear_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state);
+	}
+
+	if (test_bit(QEDN_CONN_RESRC_DB_ADD, &conn_ctx->resrc_state)) {
+		rc = qed_ops->common->db_recovery_del(qedn->cdev,
+						      conn_ctx->ep.p_doorbell,
+						      &conn_ctx->ep.db_data);
+		if (rc)
+			pr_warn("Doorbell recovery del returned error %u\n",
+				rc);
+
+		clear_bit(QEDN_CONN_RESRC_DB_ADD, &conn_ctx->resrc_state);
+	}
+
+	if (test_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state)) {
+		hash_del(&conn_ctx->hash_node);
+		rc = qed_ops->release_conn(qedn->cdev, conn_ctx->conn_handle);
+		if (rc)
+			pr_warn("Release_conn returned with an error %u\n",
+				rc);
+
+		clear_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state);
+	}
+
+	if (test_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state)) {
+		dma_free_coherent(&qedn->pdev->dev,
+				  conn_ctx->sq_depth *
+				  sizeof(struct nvmetcp_host_cccid_itid_entry),
+				  conn_ctx->host_cccid_itid,
+				  conn_ctx->host_cccid_itid_phy_addr);
+		clear_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP,
+			  &conn_ctx->resrc_state);
+	}
+
+	if (test_bit(QEDN_CONN_RESRC_TCP_PORT, &conn_ctx->resrc_state)) {
+		qedn_return_tcp_port(conn_ctx);
+		clear_bit(QEDN_CONN_RESRC_TCP_PORT,
+			  &conn_ctx->resrc_state);
+	}
+
+	if (conn_ctx->resrc_state)
+		pr_err("Conn resources state isn't 0 as expected 0x%lx\n",
+		       conn_ctx->resrc_state);
+
+	atomic_inc(&conn_ctx->destroy_conn_indicator);
+	qedn_set_con_state(conn_ctx, CONN_STATE_DESTROY_COMPLETE);
+	wake_up_interruptible(&conn_ctx->conn_waitq);
+}
+
+static int qedn_alloc_fw_sq(struct qedn_ctx *qedn,
+			    struct qedn_endpoint *ep)
+{
+	struct qed_chain_init_params params = {
+		.mode           = QED_CHAIN_MODE_PBL,
+		.intended_use   = QED_CHAIN_USE_TO_PRODUCE,
+		.cnt_type       = QED_CHAIN_CNT_TYPE_U16,
+		.num_elems      = QEDN_SQ_SIZE,
+		.elem_size      = sizeof(struct nvmetcp_wqe),
+	};
+	int rc;
+
+	rc = qed_ops->common->chain_alloc(qedn->cdev,
+					   &ep->fw_sq_chain,
+					   &params);
+	if (rc) {
+		pr_err("Failed to allocate SQ chain\n");
+
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
+static int qedn_nvmetcp_offload_conn(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qed_nvmetcp_params_offload offld_prms = { 0 };
+	struct qedn_endpoint *qedn_ep = &conn_ctx->ep;
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	u8 ts_hdr_size = 0;
+	u32 hdr_size;
+	int rc, i;
+
+	ether_addr_copy(offld_prms.src.mac, qedn_ep->src_mac);
+	ether_addr_copy(offld_prms.dst.mac, qedn_ep->dst_mac);
+	offld_prms.vlan_id = qedn_ep->vlan_id;
+	offld_prms.ecn_en = QEDN_TCP_ECN_EN;
+	offld_prms.timestamp_en =  QEDN_TCP_TS_EN;
+	offld_prms.delayed_ack_en = QEDN_TCP_DA_EN;
+	offld_prms.tcp_keep_alive_en = QEDN_TCP_KA_EN;
+	offld_prms.ip_version = qedn_ep->ip_type;
+
+	offld_prms.src.ip[0] = ntohl(qedn_ep->src_addr[0]);
+	offld_prms.dst.ip[0] = ntohl(qedn_ep->dst_addr[0]);
+	if (qedn_ep->ip_type == TCP_IPV6) {
+		for (i = 1; i < 4; i++) {
+			offld_prms.src.ip[i] = ntohl(qedn_ep->src_addr[i]);
+			offld_prms.dst.ip[i] = ntohl(qedn_ep->dst_addr[i]);
+		}
+	}
+
+	offld_prms.ttl = QEDN_TCP_TTL;
+	offld_prms.tos_or_tc = QEDN_TCP_TOS;
+	offld_prms.dst.port = qedn_ep->dst_port;
+	offld_prms.src.port = qedn_ep->src_port;
+	offld_prms.nvmetcp_cccid_itid_table_addr =
+		conn_ctx->host_cccid_itid_phy_addr;
+	offld_prms.nvmetcp_cccid_max_range = conn_ctx->sq_depth;
+
+	/* Calculate MSS */
+	if (offld_prms.timestamp_en)
+		ts_hdr_size = QEDN_TCP_TS_OPTION_LEN;
+
+	hdr_size = qedn_ep->ip_type == TCP_IPV4 ?
+		   sizeof(struct iphdr) : sizeof(struct ipv6hdr);
+	hdr_size += sizeof(struct tcphdr) + ts_hdr_size;
+
+	offld_prms.mss = qedn->mtu - hdr_size;
+	offld_prms.rcv_wnd_scale = QEDN_TCP_RCV_WND_SCALE;
+	offld_prms.cwnd = QEDN_TCP_MAX_CWND * offld_prms.mss;
+	offld_prms.ka_max_probe_cnt = QEDN_TCP_KA_MAX_PROBE_COUNT;
+	offld_prms.ka_timeout = QEDN_TCP_KA_TIMEOUT;
+	offld_prms.ka_interval = QEDN_TCP_KA_INTERVAL;
+	offld_prms.max_rt_time = QEDN_TCP_MAX_RT_TIME;
+	offld_prms.sq_pbl_addr =
+		(u64)qed_chain_get_pbl_phys(&qedn_ep->fw_sq_chain);
+
+	rc = qed_ops->offload_conn(qedn->cdev,
+				   conn_ctx->conn_handle,
+				   &offld_prms);
+	if (rc)
+		pr_err("offload_conn returned with an error\n");
+
+	return rc;
+}
+
+static int qedn_fetch_tcp_port(struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	struct qedn_ctrl *qctrl;
+	int rc = 0;
+
+	ctrl = conn_ctx->ctrl;
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return -ENODEV;
+
+	rc = qed_fetch_tcp_port(ctrl->conn_params.local_ip_addr,
+				&conn_ctx->sock, &conn_ctx->ep.src_port);
+
+	return rc;
+}
+
+static void qedn_decouple_conn(struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_tcp_ofld_queue *queue;
+
+	queue = conn_ctx->queue;
+	queue->private_data = NULL;
+}
+
+void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctrl *qctrl;
+
+	if (!conn_ctx)
+		return;
+
+	qctrl = (struct qedn_ctrl *)conn_ctx->ctrl->private_data;
+	if (!qctrl)
+		return;
+
+	if (test_and_set_bit(DESTROY_CONNECTION, &conn_ctx->agg_work_action))
+		return;
+
+	qedn_set_con_state(conn_ctx, CONN_STATE_DESTROY_CONNECTION);
+	queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
+}
+
+/* Slowpath EQ Callback */
+int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
+{
+	struct nvmetcp_connect_done_results *eqe_connect_done;
+	struct nvmetcp_eqe_data *eqe_data;
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	struct qedn_conn_ctx *conn_ctx;
+	struct qedn_ctrl *qctrl;
+	struct qedn_ctx *qedn;
+	u16 icid;
+	int rc;
+
+	if (!context || !event_ring_data) {
+		pr_err("Recv event with ctx NULL\n");
+
+		return -EINVAL;
+	}
+
+	qedn = (struct qedn_ctx *)context;
+
+	if (fw_event_code != NVMETCP_EVENT_TYPE_ASYN_CONNECT_COMPLETE) {
+		eqe_data = (struct nvmetcp_eqe_data *)event_ring_data;
+		icid = le16_to_cpu(eqe_data->icid);
+		pr_err("EQE Type=0x%x icid=0x%x, conn_id=0x%x err-code=0x%x\n",
+		       fw_event_code, eqe_data->icid, eqe_data->conn_id,
+		       eqe_data->error_code);
+	} else {
+		eqe_connect_done =
+			(struct nvmetcp_connect_done_results *)event_ring_data;
+		icid = le16_to_cpu(eqe_connect_done->icid);
+	}
+
+	conn_ctx = qedn_get_conn_hash(qedn, icid);
+	if (!conn_ctx) {
+		pr_err("Connection with icid=0x%x doesn't exist in conn list\n",
+		       icid);
+
+		return -EINVAL;
+	}
+
+	ctrl = conn_ctx->ctrl;
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return -ENODEV;
+
+	switch (fw_event_code) {
+	case NVMETCP_EVENT_TYPE_ASYN_CONNECT_COMPLETE:
+		if (conn_ctx->state != CONN_STATE_WAIT_FOR_CONNECT_DONE) {
+			pr_err("CID=0x%x:ASYN_CONNECT_COMPL:Wrong state %u\n",
+			       conn_ctx->fw_cid, conn_ctx->state);
+		} else {
+			rc = qedn_set_con_state(conn_ctx,
+						CONN_STATE_OFFLOAD_COMPLETE);
+
+			if (rc)
+				return rc;
+
+			/* Placeholder - for ICReq flow */
+		}
+
+		break;
+	case NVMETCP_EVENT_TYPE_ASYN_TERMINATE_DONE:
+		if (conn_ctx->state != CONN_STATE_WAIT_FOR_DESTROY_DONE)
+			pr_err("CID=0x%x:ASYN_TERMINATE_DONE:Wrong state %u\n",
+			       conn_ctx->fw_cid, conn_ctx->state);
+		else
+			queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
+
+		break;
+	default:
+		pr_err("CID=0x%x - Recv Unknown Event %u\n",
+		       conn_ctx->fw_cid, fw_event_code);
+		break;
+	}
+
+	return 0;
+}
+
+void qedn_prep_db_data(struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvmetcp_db_data *db_data = &conn_ctx->ep.db_data;
+	u8 params = 0;
+
+	params |= DB_DEST_XCM << NVMETCP_DB_DATA_DEST_SHIFT;
+	params |= DB_AGG_CMD_SET << NVMETCP_DB_DATA_AGG_CMD_SHIFT;
+	params |= DQ_XCM_ISCSI_SQ_PROD_CMD << NVMETCP_DB_DATA_AGG_VAL_SEL_SHIFT;
+	params |= 1 << NVMETCP_DB_DATA_BYPASS_EN_SHIFT;
+
+	db_data->params = params;
+	db_data->agg_flags = 0;
+}
+
+static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	size_t dma_size;
+	int rc;
+
+	rc = qedn_alloc_fw_sq(qedn, &conn_ctx->ep);
+	if (rc) {
+		pr_err("Failed to allocate FW SQ\n");
+		goto rel_conn;
+	}
+
+	set_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state);
+	rc = qed_ops->acquire_conn(qedn->cdev,
+				   &conn_ctx->conn_handle,
+				   &conn_ctx->fw_cid,
+				   &conn_ctx->ep.p_doorbell);
+	if (rc) {
+		pr_err("Couldn't acquire connection\n");
+		goto rel_conn;
+	}
+
+	hash_add(qedn->conn_ctx_hash, &conn_ctx->hash_node,
+		 conn_ctx->conn_handle);
+	set_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state);
+
+	/* Placeholder - Allocate task resources and initialize fields */
+
+	rc = qedn_fetch_tcp_port(conn_ctx);
+	if (rc)
+		goto rel_conn;
+
+	set_bit(QEDN_CONN_RESRC_TCP_PORT, &conn_ctx->resrc_state);
+	dma_size = conn_ctx->sq_depth *
+			   sizeof(struct nvmetcp_host_cccid_itid_entry);
+	conn_ctx->host_cccid_itid =
+			dma_alloc_coherent(&qedn->pdev->dev,
+					   dma_size,
+					   &conn_ctx->host_cccid_itid_phy_addr,
+					   GFP_ATOMIC);
+	if (!conn_ctx->host_cccid_itid) {
+		pr_err("CCCID-iTID Map allocation failed\n");
+		goto rel_conn;
+	}
+
+	memset(conn_ctx->host_cccid_itid, 0xFF, dma_size);
+	set_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state);
+	rc = qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_CONNECT_DONE);
+	if (rc)
+		goto rel_conn;
+
+	qedn_prep_db_data(conn_ctx);
+	rc = qed_ops->common->db_recovery_add(qedn->cdev,
+					      conn_ctx->ep.p_doorbell,
+					      &conn_ctx->ep.db_data,
+					      DB_REC_WIDTH_32B, DB_REC_KERNEL);
+	if (rc)
+		goto rel_conn;
+	set_bit(QEDN_CONN_RESRC_DB_ADD, &conn_ctx->resrc_state);
+
+	rc = qedn_nvmetcp_offload_conn(conn_ctx);
+	if (rc) {
+		pr_err("Offload error: CID=0x%x\n", conn_ctx->fw_cid);
+		goto rel_conn;
+	}
+
+	return 0;
+
+rel_conn:
+	pr_err("qedn create queue ended with ERROR\n");
+	qedn_release_conn_ctx(conn_ctx);
+
+	return -EINVAL;
+}
+
+void qedn_cleanp_fw(struct qedn_conn_ctx *conn_ctx)
+{
+	/* Placeholder - task cleanup */
+}
+
+void qedn_destroy_connection(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int rc;
+
+	qedn_decouple_conn(conn_ctx);
+
+	if (qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_DESTROY_DONE))
+		return;
+
+	rc = qed_ops->destroy_conn(qedn->cdev, conn_ctx->conn_handle,
+				   conn_ctx->abrt_flag);
+	if (rc)
+		pr_warn("destroy_conn failed - rc %u\n", rc);
+}
+
+void qedn_sp_wq_handler(struct work_struct *work)
+{
+	struct qedn_conn_ctx *conn_ctx;
+	struct qedn_ctx *qedn;
+	int rc;
+
+	conn_ctx = container_of(work, struct qedn_conn_ctx, sp_wq_entry);
+	qedn = conn_ctx->qedn;
+
+	if (conn_ctx->state == CONN_STATE_DESTROY_COMPLETE) {
+		pr_err("Connection already released!\n");
+
+		return;
+	}
+
+	if (conn_ctx->state == CONN_STATE_WAIT_FOR_DESTROY_DONE) {
+		qedn_release_conn_ctx(conn_ctx);
+
+		return;
+	}
+
+	qedn = conn_ctx->qedn;
+	if (test_bit(DESTROY_CONNECTION, &conn_ctx->agg_work_action)) {
+		qedn_destroy_connection(conn_ctx);
+
+		return;
+	}
+
+	if (test_bit(CREATE_CONNECTION, &conn_ctx->agg_work_action)) {
+		qedn_clr_sp_wa(conn_ctx, CREATE_CONNECTION);
+		rc = qedn_prep_and_offload_queue(conn_ctx);
+		if (rc) {
+			pr_err("Error in queue prepare & firmware offload\n");
+
+			return;
+		}
+	}
+}
+
+/* Clear connection aggregative slowpath work action */
+void qedn_clr_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit)
+{
+	clear_bit(bit, &conn_ctx->agg_work_action);
+}
+
+/* Set connection aggregative slowpath work action */
+void qedn_set_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit)
+{
+	set_bit(bit, &conn_ctx->agg_work_action);
+}
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 6c0f36f7d9d1..bd5618f65c70 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -22,6 +22,15 @@ static struct pci_device_id qedn_pci_tbl[] = {
 	{0, 0},
 };
 
+static bool qedn_matches_qede(struct qedn_ctx *qedn, struct pci_dev *qede_pdev)
+{
+	struct pci_dev *qedn_pdev = qedn->pdev;
+
+	return (qede_pdev->bus->number == qedn_pdev->bus->number &&
+		PCI_SLOT(qede_pdev->devfn) == PCI_SLOT(qedn_pdev->devfn) &&
+		PCI_FUNC(qede_pdev->devfn) == qedn->dev_info.port_id);
+}
+
 static int
 qedn_find_dev(struct nvme_tcp_ofld_dev *dev,
 	      struct nvme_tcp_ofld_ctrl *ctrl)
@@ -29,7 +38,9 @@ qedn_find_dev(struct nvme_tcp_ofld_dev *dev,
 	struct nvme_tcp_ofld_ctrl_con_params *conn_params;
 	struct pci_dev *qede_pdev = NULL;
 	struct sockaddr remote_mac_addr;
+	struct qedn_ctrl *qctrl = NULL;
 	struct net_device *ndev = NULL;
+	struct qedn_ctx *qedn = NULL;
 	u16 vlan_id = 0;
 	int rc = 0;
 
@@ -65,11 +76,24 @@ qedn_find_dev(struct nvme_tcp_ofld_dev *dev,
 
 	qed_vlan_get_ndev(&ctrl->ndev, &vlan_id);
 
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (qctrl) {
+		qctrl->remote_mac_addr = remote_mac_addr;
+		qctrl->vlan_id = vlan_id;
+	}
+
 	/* route found through ndev - validate this is qede*/
 	qede_pdev = qed_validate_ndev(ctrl->ndev);
 	if (!qede_pdev)
 		return false;
 
+	qedn = container_of(dev, struct qedn_ctx, qedn_ofld_dev);
+	if (!qedn)
+		return false;
+
+	if (!qedn_matches_qede(qedn, qede_pdev))
+		return false;
+
 	return true;
 }
 
@@ -82,14 +106,73 @@ qedn_claim_dev(struct nvme_tcp_ofld_dev *dev,
 
 static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 {
-	/* Placeholder - qedn_setup_ctrl */
+	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+	struct qedn_ctrl *qctrl = NULL;
+	struct qedn_ctx *qedn = NULL;
+	bool new = true;
+	int rc = 0;
+
+	if (ctrl->private_data) {
+		qctrl = (struct qedn_ctrl *)ctrl->private_data;
+		new = false;
+	}
+
+	if (new) {
+		qctrl = kzalloc(sizeof(*qctrl), GFP_KERNEL);
+		if (!qctrl)
+			return -ENOMEM;
+
+		ctrl->private_data = (void *)qctrl;
+		set_bit(QEDN_CTRL_SET_TO_OFLD_CTRL, &qctrl->agg_state);
+
+		qctrl->sp_wq = alloc_workqueue(QEDN_SP_WORKQUEUE,
+					       WQ_MEM_RECLAIM,
+					       QEDN_SP_WORKQUEUE_MAX_ACTIVE);
+		if (!qctrl->sp_wq) {
+			rc = -ENODEV;
+			pr_err("Unable to create slowpath work queue!\n");
+			kfree(qctrl);
+
+			return rc;
+		}
+
+		set_bit(QEDN_STATE_SP_WORK_THREAD_SET, &qctrl->agg_state);
+	}
+
+	if (!qedn_find_dev(dev, ctrl)) {
+		rc = -ENODEV;
+		goto err_out;
+	}
+
+	qedn = container_of(dev, struct qedn_ctx, qedn_ofld_dev);
+	qctrl->qedn = qedn;
+
+	/* Placeholder - setup LLH filter */
 
 	return 0;
+err_out:
+	flush_workqueue(qctrl->sp_wq);
+	kfree(qctrl);
+
+	return rc;
 }
 
 static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 {
-	/* Placeholder - qedn_release_ctrl */
+	struct qedn_ctrl *qctrl;
+
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return -ENODEV;
+
+	if (test_and_clear_bit(QEDN_STATE_SP_WORK_THREAD_SET,
+			       &qctrl->agg_state))
+		flush_workqueue(qctrl->sp_wq);
+
+	if (test_and_clear_bit(QEDN_CTRL_SET_TO_OFLD_CTRL, &qctrl->agg_state)) {
+		kfree(qctrl);
+		ctrl->private_data = NULL;
+	}
 
 	return 0;
 }
@@ -97,19 +180,117 @@ static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 			     size_t queue_size)
 {
-	/* Placeholder - qedn_create_queue */
+	struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl;
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+	struct qedn_conn_ctx *conn_ctx;
+	struct qedn_ctrl *qctrl;
+	struct qedn_ctx *qedn;
+	int rc;
+
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return -ENODEV;
+
+	qedn = qctrl->qedn;
+
+	/* Allocate qedn connection context */
+	conn_ctx = kzalloc(sizeof(*conn_ctx), GFP_KERNEL);
+	if (!conn_ctx)
+		return -ENOMEM;
+
+	queue->private_data = conn_ctx;
+	queue->hdr_digest = nctrl->opts->hdr_digest;
+	queue->data_digest = nctrl->opts->data_digest;
+	queue->tos = nctrl->opts->tos;
+
+	conn_ctx->qedn = qedn;
+	conn_ctx->queue = queue;
+	conn_ctx->ctrl = ctrl;
+	conn_ctx->sq_depth = queue_size;
+
+	init_waitqueue_head(&conn_ctx->conn_waitq);
+	atomic_set(&conn_ctx->est_conn_indicator, 0);
+	atomic_set(&conn_ctx->destroy_conn_indicator, 0);
+
+	spin_lock_init(&conn_ctx->conn_state_lock);
+
+	qedn_initialize_endpoint(&conn_ctx->ep, qedn->local_mac_addr, ctrl);
+
+	atomic_inc(&qctrl->host_num_active_conns);
+
+	qedn_set_sp_wa(conn_ctx, CREATE_CONNECTION);
+	qedn_set_con_state(conn_ctx, CONN_STATE_CREATE_CONNECTION);
+	INIT_WORK(&conn_ctx->sp_wq_entry, qedn_sp_wq_handler);
+	queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
+
+	/* Wait for the connection establishment to complete - this includes the
+	 * FW TCP connection establishment and the NVMeTCP ICReq & ICResp
+	 */
+	rc = qedn_wait_for_conn_est(conn_ctx);
+	if (rc)
+		return -ENXIO;
 
 	return 0;
 }
 
 static void qedn_drain_queue(struct nvme_tcp_ofld_queue *queue)
 {
-	/* Placeholder - qedn_drain_queue */
+	struct qedn_conn_ctx *conn_ctx;
+
+	if (!queue) {
+		pr_err("ctrl has no queues\n");
+
+		return;
+	}
+
+	conn_ctx = (struct qedn_conn_ctx *)queue->private_data;
+	if (!conn_ctx)
+		return;
+
+	qedn_cleanp_fw(conn_ctx);
+}
+
+static inline void
+qedn_queue_wait_for_terminate_complete(struct qedn_conn_ctx *conn_ctx)
+{
+	/* Returns valid non-0 */
+	atomic_t *conn_dest_ind = &conn_ctx->destroy_conn_indicator;
+	u32 dest_timeout = msecs_to_jiffies(QEDN_RLS_CONS_TMO);
+	int wrc, state;
+
+	wrc = wait_event_interruptible_timeout(conn_ctx->conn_waitq,
+					       atomic_read(conn_dest_ind) > 0,
+					       dest_timeout);
+
+	atomic_set(conn_dest_ind, 0);
+
+	spin_lock_bh(&conn_ctx->conn_state_lock);
+	state = conn_ctx->state;
+	spin_unlock_bh(&conn_ctx->conn_state_lock);
+
+	if (!wrc  || state != CONN_STATE_DESTROY_COMPLETE)
+		pr_warn("Timed out waiting for clear-SQ on FW conns");
 }
 
 static void qedn_destroy_queue(struct nvme_tcp_ofld_queue *queue)
 {
-	/* Placeholder - qedn_destroy_queue */
+	struct qedn_conn_ctx *conn_ctx;
+
+	if (!queue) {
+		pr_err("ctrl has no queues\n");
+
+		return;
+	}
+
+	conn_ctx = (struct qedn_conn_ctx *)queue->private_data;
+	if (!conn_ctx)
+		return;
+
+	qedn_terminate_connection(conn_ctx);
+
+	qedn_queue_wait_for_terminate_complete(conn_ctx);
+
+	kfree(conn_ctx);
 }
 
 static int qedn_poll_queue(struct nvme_tcp_ofld_queue *queue)
@@ -150,6 +331,21 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
 	.send_req = qedn_send_req,
 };
 
+struct qedn_conn_ctx *qedn_get_conn_hash(struct qedn_ctx *qedn, u16 icid)
+{
+	struct qedn_conn_ctx *conn = NULL;
+
+	hash_for_each_possible(qedn->conn_ctx_hash, conn, hash_node, icid) {
+		if (conn->conn_handle == icid)
+			break;
+	}
+
+	if (!conn || conn->conn_handle != icid)
+		return NULL;
+
+	return conn;
+}
+
 /* Fastpath IRQ handler */
 static irqreturn_t qedn_irq_handler(int irq, void *dev_id)
 {
@@ -250,7 +446,7 @@ static int qedn_setup_irq(struct qedn_ctx *qedn)
 
 static inline void qedn_init_pf_struct(struct qedn_ctx *qedn)
 {
-	/* Placeholder - Initialize qedn fields */
+	hash_init(qedn->conn_ctx_hash);
 }
 
 static inline void
@@ -591,7 +787,7 @@ static int __qedn_probe(struct pci_dev *pdev)
 	rc = qed_ops->start(qedn->cdev,
 			    NULL /* Placeholder for FW IO-path resources */,
 			    qedn,
-			    NULL /* Placeholder for FW Event callback */);
+			    qedn_event_cb);
 	if (rc) {
 		rc = -ENODEV;
 		pr_err("Cannot start NVMeTCP Function\n");
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:24:09 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:24:09 +0300
Subject: [PATCH v3 19/20] qedn: Add Connection and IO level recovery flows
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-20-smalin@marvell.com>

This patch will present the connection level functionalities:
 - conn clear-sq: will release the FW restrictions in order to flush all
   the pending IOs.
 - drain: in case clear-sq is stuck, will release all the device FW
   restrictions in order to flush all the pending IOs.
 - task cleanup - will flush the IO level resources.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |   8 ++
 drivers/nvme/hw/qedn/qedn_conn.c | 135 ++++++++++++++++++++++++++++++-
 drivers/nvme/hw/qedn/qedn_main.c |   6 ++
 drivers/nvme/hw/qedn/qedn_task.c |  30 ++++++-
 4 files changed, 176 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index b36994be65cb..5539791fc49b 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -41,6 +41,8 @@
 
 #define QEDN_FW_CQ_FP_WQ_WORKQUEUE "qedn_fw_cq_fp_wq"
 
+#define QEDN_DRAIN_MAX_ATTEMPTS 3
+
 /* Protocol defines */
 #define QEDN_MAX_IO_SIZE QED_NVMETCP_MAX_IO_SIZE
 #define QEDN_MAX_PDU_SIZE 0x80000 /* 512KB */
@@ -91,6 +93,8 @@
 /* Timeouts and delay constants */
 #define QEDN_WAIT_CON_ESTABLSH_TMO 10000 /* 10 seconds */
 #define QEDN_RLS_CONS_TMO 5000 /* 5 sec */
+#define QEDN_TASK_CLEANUP_TMO 3000 /* 3 sec */
+#define QEDN_DRAIN_TMO 1000 /* 1 sec */
 
 enum qedn_state {
 	QEDN_STATE_CORE_PROBED = 0,
@@ -173,7 +177,9 @@ struct qedn_ctx {
 };
 
 enum qedn_task_flags {
+	QEDN_TASK_IS_ICREQ,
 	QEDN_TASK_USED_BY_FW,
+	QEDN_TASK_WAIT_FOR_CLEANUP,
 };
 
 struct qedn_task_ctx {
@@ -321,6 +327,8 @@ struct qedn_conn_ctx {
 	struct list_head active_task_list;
 	atomic_t num_active_tasks;
 	atomic_t num_active_fw_tasks;
+	atomic_t task_cleanups_cnt;
+	wait_queue_head_t cleanup_waitq;
 
 	/* Connection resources - turned on to indicate what resource was
 	 * allocated, to that it can later be released.
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
index ea072eff34a6..a61e77bf8bd0 100644
--- a/drivers/nvme/hw/qedn/qedn_conn.c
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -626,6 +626,11 @@ static int qedn_handle_icresp(struct qedn_conn_ctx *conn_ctx)
 	return rc;
 }
 
+void qedn_error_recovery(struct nvme_ctrl *nctrl)
+{
+	nvme_tcp_ofld_error_recovery(nctrl);
+}
+
 /* Slowpath EQ Callback */
 int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
 {
@@ -688,6 +693,7 @@ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
 		}
 
 		break;
+
 	case NVMETCP_EVENT_TYPE_ASYN_TERMINATE_DONE:
 		if (conn_ctx->state != CONN_STATE_WAIT_FOR_DESTROY_DONE)
 			pr_err("CID=0x%x:ASYN_TERMINATE_DONE:Wrong state %u\n",
@@ -696,6 +702,19 @@ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
 			queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
 
 		break;
+
+	case NVMETCP_EVENT_TYPE_ASYN_CLOSE_RCVD:
+	case NVMETCP_EVENT_TYPE_ASYN_ABORT_RCVD:
+	case NVMETCP_EVENT_TYPE_ASYN_MAX_RT_TIME:
+	case NVMETCP_EVENT_TYPE_ASYN_MAX_RT_CNT:
+	case NVMETCP_EVENT_TYPE_ASYN_SYN_RCVD:
+	case NVMETCP_EVENT_TYPE_ASYN_MAX_KA_PROBES_CNT:
+	case NVMETCP_EVENT_TYPE_NVMETCP_CONN_ERROR:
+	case NVMETCP_EVENT_TYPE_TCP_CONN_ERROR:
+		qedn_error_recovery(&conn_ctx->ctrl->nctrl);
+
+		break;
+
 	default:
 		pr_err("CID=0x%x - Recv Unknown Event %u\n",
 		       conn_ctx->fw_cid, fw_event_code);
@@ -835,9 +854,123 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 	return -EINVAL;
 }
 
+static void qedn_cleanup_fw_task(struct qedn_ctx *qedn,
+				 struct qedn_task_ctx *qedn_task)
+{
+	struct qedn_conn_ctx *conn_ctx = qedn_task->qedn_conn;
+	struct nvmetcp_task_params task_params;
+	struct nvmetcp_wqe *chain_sqe;
+	struct nvmetcp_wqe local_sqe;
+	unsigned long lock_flags;
+
+	/* Take lock to prevent race with fastpath, we don't want to
+	 * invoke cleanup flows on tasks that already returned.
+	 */
+	spin_lock_irqsave(&qedn_task->lock, lock_flags);
+	if (!qedn_task->valid) {
+		spin_unlock_irqrestore(&qedn_task->lock, lock_flags);
+
+		return;
+	}
+	/* Skip tasks not used by FW */
+	if (!test_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags)) {
+		spin_unlock_irqrestore(&qedn_task->lock, lock_flags);
+
+		return;
+	}
+	/* Skip tasks that were already invoked for cleanup */
+	if (unlikely(test_bit(QEDN_TASK_WAIT_FOR_CLEANUP, &qedn_task->flags))) {
+		spin_unlock_irqrestore(&qedn_task->lock, lock_flags);
+
+		return;
+	}
+	set_bit(QEDN_TASK_WAIT_FOR_CLEANUP, &qedn_task->flags);
+	spin_unlock_irqrestore(&qedn_task->lock, lock_flags);
+
+	atomic_inc(&conn_ctx->task_cleanups_cnt);
+
+	task_params.sqe = &local_sqe;
+	task_params.itid = qedn_task->itid;
+	qed_ops->init_task_cleanup(&task_params);
+
+	/* spin_lock - doorbell is accessed  both Rx flow and response flow */
+	spin_lock(&conn_ctx->ep.doorbell_lock);
+	chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain);
+	memcpy(chain_sqe, &local_sqe, sizeof(local_sqe));
+	qedn_ring_doorbell(conn_ctx);
+	spin_unlock(&conn_ctx->ep.doorbell_lock);
+}
+
+inline int qedn_drain(struct qedn_conn_ctx *conn_ctx)
+{
+	u32 drain_timeout = msecs_to_jiffies(QEDN_DRAIN_TMO);
+	atomic_t *clnup_cnt = &conn_ctx->task_cleanups_cnt;
+	int drain_iter = QEDN_DRAIN_MAX_ATTEMPTS;
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int wrc;
+
+	while (drain_iter) {
+		qed_ops->common->drain(qedn->cdev);
+		msleep(100);
+
+		wrc = wait_event_interruptible_timeout(conn_ctx->cleanup_waitq,
+						       !atomic_read(clnup_cnt),
+						       drain_timeout);
+		if (!wrc) {
+			drain_iter--;
+			continue;
+		}
+
+		return 0;
+	}
+
+	pr_err("CID 0x%x: cleanup after drain failed - need hard reset.\n",
+	       conn_ctx->fw_cid);
+
+	return -EINVAL;
+}
+
+void qedn_cleanup_all_fw_tasks(struct qedn_conn_ctx *conn_ctx)
+{
+	u32 clnup_timeout = msecs_to_jiffies(QEDN_TASK_CLEANUP_TMO);
+	atomic_t *clnup_cnt = &conn_ctx->task_cleanups_cnt;
+	struct qedn_task_ctx *qedn_task, *task_tmp;
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int wrc;
+
+	list_for_each_entry_safe_reverse(qedn_task, task_tmp,
+					 &conn_ctx->active_task_list, entry) {
+		qedn_cleanup_fw_task(qedn, qedn_task);
+	}
+
+	wrc = wait_event_interruptible_timeout(conn_ctx->cleanup_waitq,
+					       atomic_read(clnup_cnt) == 0,
+					       clnup_timeout);
+	if (!wrc) {
+		if (qedn_drain(conn_ctx))
+			return;
+	}
+}
+
+static void qedn_clear_fw_sq(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int rc;
+
+	rc = qed_ops->clear_sq(qedn->cdev, conn_ctx->conn_handle);
+	if (rc)
+		pr_warn("clear_sq failed - rc %u\n", rc);
+}
+
 void qedn_cleanp_fw(struct qedn_conn_ctx *conn_ctx)
 {
-	/* Placeholder - task cleanup */
+	if (atomic_read(&conn_ctx->num_active_fw_tasks)) {
+		conn_ctx->abrt_flag = QEDN_ABORTIVE_TERMINATION;
+		qedn_clear_fw_sq(conn_ctx);
+		qedn_cleanup_all_fw_tasks(conn_ctx);
+	} else {
+		conn_ctx->abrt_flag = QEDN_NON_ABORTIVE_TERMINATION;
+	}
 }
 
 void qedn_destroy_connection(struct qedn_conn_ctx *conn_ctx)
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index fb47e315ab03..d18cc3624764 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -262,12 +262,17 @@ static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 
 static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 {
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
 	struct qedn_ctrl *qctrl;
 
 	qctrl = (struct qedn_ctrl *)ctrl->private_data;
 	if (!qctrl)
 		return -ENODEV;
 
+	if (nctrl->state == NVME_CTRL_CONNECTING ||
+	    nctrl->state == NVME_CTRL_RESETTING)
+		return 0;
+
 	if (test_and_clear_bit(LLH_FILTER, &qctrl->agg_state) &&
 	    qctrl->llh_filter) {
 		qedn_dec_llh_filter(qctrl->qedn, qctrl->llh_filter);
@@ -333,6 +338,7 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 	qedn_set_pdu_params(conn_ctx);
 
 	init_waitqueue_head(&conn_ctx->conn_waitq);
+	init_waitqueue_head(&conn_ctx->cleanup_waitq);
 	atomic_set(&conn_ctx->est_conn_indicator, 0);
 	atomic_set(&conn_ctx->destroy_conn_indicator, 0);
 
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
index dd0b5f31c052..6664cc587f06 100644
--- a/drivers/nvme/hw/qedn/qedn_task.c
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -324,6 +324,17 @@ void qedn_return_active_tasks(struct qedn_conn_ctx *conn_ctx)
 	/* Return tasks that aren't "Used by FW" to the pool */
 	list_for_each_entry_safe(qedn_task, task_tmp,
 				 &conn_ctx->active_task_list, entry) {
+		/* If we got this far, cleanup was already done
+		 * in which case we want to return the task to the pool and
+		 * release it. So we make sure the cleanup indication is down
+		 */
+		clear_bit(QEDN_TASK_WAIT_FOR_CLEANUP, &qedn_task->flags);
+
+		/* Special handling in case of ICREQ task */
+		if (unlikely(conn_ctx->state ==	CONN_STATE_WAIT_FOR_IC_COMP &&
+			     test_bit(QEDN_TASK_IS_ICREQ, &(qedn_task)->flags)))
+			qedn_common_clear_fw_sgl(&qedn_task->sgl_task_params);
+
 		qedn_clear_task(conn_ctx, qedn_task);
 		num_returned_tasks++;
 	}
@@ -669,7 +680,9 @@ void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
 		return;
 
 	if (likely(cqe->cqe_type == NVMETCP_FW_CQE_TYPE_NORMAL)) {
-		/* Placeholder - verify the connection was established */
+		if (unlikely(test_bit(QEDN_TASK_WAIT_FOR_CLEANUP,
+				      &qedn_task->flags)))
+			return;
 
 		switch (cqe->task_type) {
 		case NVMETCP_TASK_TYPE_HOST_WRITE:
@@ -711,6 +724,19 @@ void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
 			pr_info("Could not identify task type\n");
 		}
 	} else {
-		/* Placeholder - Recovery flows */
+		if (cqe->cqe_type == NVMETCP_FW_CQE_TYPE_CLEANUP) {
+			clear_bit(QEDN_TASK_WAIT_FOR_CLEANUP,
+				  &qedn_task->flags);
+			qedn_return_task_to_pool(conn_ctx, qedn_task);
+			atomic_dec(&conn_ctx->task_cleanups_cnt);
+			wake_up_interruptible(&conn_ctx->cleanup_waitq);
+
+			return;
+		}
+
+		 /* The else is NVMETCP_FW_CQE_TYPE_DUMMY - in which don't
+		  * return the task. The task will return during
+		  * NVMETCP_FW_CQE_TYPE_CLEANUP.
+		  */
 	}
 }
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:24:10 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:24:10 +0300
Subject: [PATCH v3 20/20] qedn: Add support of ASYNC
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-21-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

This patch implement ASYNC request and response event notification
handling at qedn driver level.

NVME Ofld layer's ASYNC request is treated similar to read with
fake CCCID. This CCCID used to route ASYNC notification back to
the NVME ofld layer.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |   8 ++
 drivers/nvme/hw/qedn/qedn_main.c |   1 +
 drivers/nvme/hw/qedn/qedn_task.c | 147 +++++++++++++++++++++++++++++--
 3 files changed, 148 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 5539791fc49b..b275d6b9bc59 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -96,6 +96,9 @@
 #define QEDN_TASK_CLEANUP_TMO 3000 /* 3 sec */
 #define QEDN_DRAIN_TMO 1000 /* 1 sec */
 
+#define QEDN_MAX_OUTSTAND_ASYNC 32
+#define QEDN_INVALID_CCCID (-1)
+
 enum qedn_state {
 	QEDN_STATE_CORE_PROBED = 0,
 	QEDN_STATE_CORE_OPEN,
@@ -178,6 +181,7 @@ struct qedn_ctx {
 
 enum qedn_task_flags {
 	QEDN_TASK_IS_ICREQ,
+	QEDN_TASK_ASYNC,
 	QEDN_TASK_USED_BY_FW,
 	QEDN_TASK_WAIT_FOR_CLEANUP,
 };
@@ -346,6 +350,10 @@ struct qedn_conn_ctx {
 	struct nvme_tcp_icresp_pdu icresp;
 	struct qedn_icreq_padding *icreq_pad;
 
+	DECLARE_BITMAP(async_cccid_idx_map, QEDN_MAX_OUTSTAND_ASYNC);
+	/* Spinlock for fetching pseudo CCCID for async request */
+	spinlock_t async_cccid_bitmap_lock;
+
 	/* "dummy" socket */
 	struct socket *sock;
 };
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index d18cc3624764..2aaf1af8f9a7 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -343,6 +343,7 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 	atomic_set(&conn_ctx->destroy_conn_indicator, 0);
 
 	spin_lock_init(&conn_ctx->conn_state_lock);
+	spin_lock_init(&conn_ctx->async_cccid_bitmap_lock);
 
 	conn_ctx->qid = qid;
 
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
index 6664cc587f06..351afbc750b2 100644
--- a/drivers/nvme/hw/qedn/qedn_task.c
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -266,9 +266,45 @@ void qedn_common_clear_fw_sgl(struct storage_sgl_task_params *sgl_task_params)
 }
 
 inline void qedn_host_reset_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx,
-					     u16 cccid)
+					     u16 cccid, bool async)
 {
 	conn_ctx->host_cccid_itid[cccid].itid = cpu_to_le16(QEDN_INVALID_ITID);
+	if (unlikely(async))
+		clear_bit(cccid - NVME_AQ_DEPTH,
+			  conn_ctx->async_cccid_idx_map);
+}
+
+static int qedn_get_free_idx(struct qedn_conn_ctx *conn_ctx, unsigned int size)
+{
+	int idx;
+
+	spin_lock(&conn_ctx->async_cccid_bitmap_lock);
+	idx = find_first_zero_bit(conn_ctx->async_cccid_idx_map, size);
+	if (unlikely(idx >= size)) {
+		idx = -1;
+		spin_unlock(&conn_ctx->async_cccid_bitmap_lock);
+		goto err_idx;
+	}
+	set_bit(idx, conn_ctx->async_cccid_idx_map);
+	spin_unlock(&conn_ctx->async_cccid_bitmap_lock);
+
+err_idx:
+
+	return idx;
+}
+
+int qedn_get_free_async_cccid(struct qedn_conn_ctx *conn_ctx)
+{
+	int async_cccid;
+
+	async_cccid =
+		qedn_get_free_idx(conn_ctx, QEDN_MAX_OUTSTAND_ASYNC);
+	if (unlikely(async_cccid == QEDN_INVALID_CCCID))
+		pr_err("No available CCCID for Async.\n");
+	else
+		async_cccid += NVME_AQ_DEPTH;
+
+	return async_cccid;
 }
 
 inline void qedn_host_set_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx,
@@ -361,10 +397,12 @@ void qedn_return_task_to_pool(struct qedn_conn_ctx *conn_ctx,
 	struct qedn_fp_queue *fp_q = conn_ctx->fp_q;
 	struct qedn_io_resources *io_resrc;
 	unsigned long lock_flags;
+	bool async;
 
 	io_resrc = &fp_q->host_resrc;
 
 	spin_lock_irqsave(&qedn_task->lock, lock_flags);
+	async = test_bit(QEDN_TASK_ASYNC, &(qedn_task)->flags);
 	qedn_task->valid = 0;
 	qedn_task->flags = 0;
 	qedn_clear_sgl(conn_ctx->qedn, qedn_task);
@@ -372,7 +410,7 @@ void qedn_return_task_to_pool(struct qedn_conn_ctx *conn_ctx,
 
 	spin_lock(&conn_ctx->task_list_lock);
 	list_del(&qedn_task->entry);
-	qedn_host_reset_cccid_itid_entry(conn_ctx, qedn_task->cccid);
+	qedn_host_reset_cccid_itid_entry(conn_ctx, qedn_task->cccid, async);
 	spin_unlock(&conn_ctx->task_list_lock);
 
 	atomic_dec(&conn_ctx->num_active_tasks);
@@ -419,6 +457,60 @@ qedn_get_free_task_from_pool(struct qedn_conn_ctx *conn_ctx, u16 cccid)
 	return qedn_task;
 }
 
+void qedn_send_async_event_cmd(struct qedn_task_ctx *qedn_task,
+			       struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_tcp_ofld_req *async_req = qedn_task->req;
+	struct nvme_command *nvme_cmd = &async_req->nvme_cmd;
+	struct storage_sgl_task_params *sgl_task_params;
+	struct nvmetcp_task_params task_params;
+	struct nvme_tcp_cmd_pdu cmd_hdr;
+	struct nvmetcp_wqe *chain_sqe;
+	struct nvmetcp_wqe local_sqe;
+
+	set_bit(QEDN_TASK_ASYNC, &qedn_task->flags);
+	nvme_cmd->common.command_id = qedn_task->cccid;
+	qedn_task->task_size = 0;
+
+	/* Initialize sgl params */
+	sgl_task_params = &qedn_task->sgl_task_params;
+	sgl_task_params->total_buffer_size = 0;
+	sgl_task_params->num_sges = 0;
+	sgl_task_params->small_mid_sge = false;
+
+	task_params.opq.lo = cpu_to_le32(((u64)(qedn_task)) & 0xffffffff);
+	task_params.opq.hi = cpu_to_le32(((u64)(qedn_task)) >> 32);
+
+	/* Initialize task params */
+	task_params.context = qedn_task->fw_task_ctx;
+	task_params.sqe = &local_sqe;
+	task_params.tx_io_size = 0;
+	task_params.rx_io_size = 0;
+	task_params.conn_icid = (u16)conn_ctx->conn_handle;
+	task_params.itid = qedn_task->itid;
+	task_params.cq_rss_number = conn_ctx->default_cq;
+	task_params.send_write_incapsule = 0;
+
+	/* Internal impl. - async is treated like zero len read */
+	cmd_hdr.hdr.type = nvme_tcp_cmd;
+	cmd_hdr.hdr.flags = 0;
+	cmd_hdr.hdr.hlen = sizeof(cmd_hdr);
+	cmd_hdr.hdr.pdo = 0x0;
+	cmd_hdr.hdr.plen = cpu_to_le32(cmd_hdr.hdr.hlen);
+
+	qed_ops->init_read_io(&task_params, &cmd_hdr, nvme_cmd,
+			      &qedn_task->sgl_task_params);
+
+	set_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags);
+	atomic_inc(&conn_ctx->num_active_fw_tasks);
+
+	spin_lock(&conn_ctx->ep.doorbell_lock);
+	chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain);
+	memcpy(chain_sqe, &local_sqe, sizeof(local_sqe));
+	qedn_ring_doorbell(conn_ctx);
+	spin_unlock(&conn_ctx->ep.doorbell_lock);
+}
+
 int qedn_send_read_cmd(struct qedn_task_ctx *qedn_task,
 		       struct qedn_conn_ctx *conn_ctx)
 {
@@ -533,6 +625,21 @@ int qedn_send_write_cmd(struct qedn_task_ctx *qedn_task,
 	return 0;
 }
 
+static void qedn_return_error_req(struct nvme_tcp_ofld_req *req)
+{
+	__le16 status = cpu_to_le16(NVME_SC_HOST_PATH_ERROR << 1);
+	union nvme_result res = {};
+
+	if (!req)
+		return;
+
+	/* Call request done to compelete the request */
+	if (req->done)
+		req->done(req, &res, status);
+	else
+		pr_err("request done not set !!!\n");
+}
+
 int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 		       struct nvme_tcp_ofld_req *req)
 {
@@ -543,9 +650,17 @@ int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 
 	rq = blk_mq_rq_from_pdu(req);
 
-	/* Placeholder - async */
+	if (unlikely(req->async)) {
+		cccid = qedn_get_free_async_cccid(qedn_conn);
+		if (cccid == QEDN_INVALID_CCCID) {
+			qedn_return_error_req(req);
+
+			return BLK_STS_NOTSUPP;
+		}
+	} else {
+		cccid = rq->tag;
+	}
 
-	cccid = rq->tag;
 	qedn_task = qedn_get_free_task_from_pool(qedn_conn, cccid);
 	if (unlikely(!qedn_task)) {
 		pr_err("Not able to allocate task context resource\n");
@@ -556,7 +671,11 @@ int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 	req->private_data = qedn_task;
 	qedn_task->req = req;
 
-	/* Placeholder - handle (req->async) */
+	if (unlikely(req->async)) {
+		qedn_send_async_event_cmd(qedn_task, qedn_conn);
+
+		return BLK_STS_TRANSPORT;
+	}
 
 	/* Check if there are physical segments in request to determine the
 	 * task size. The logic of nvme_tcp_set_sg_null() will be implemented
@@ -629,16 +748,28 @@ static inline int qedn_comp_valid_task(struct qedn_task_ctx *qedn_task,
 int qedn_process_nvme_cqe(struct qedn_task_ctx *qedn_task,
 			  struct nvme_completion *cqe)
 {
+	struct qedn_conn_ctx *conn_ctx = qedn_task->qedn_conn;
+	struct nvme_tcp_ofld_req *req;
 	int rc = 0;
+	bool async;
+
+	async = test_bit(QEDN_TASK_ASYNC, &(qedn_task)->flags);
 
 	/* CQE arrives swapped
 	 * Swapping requirement will be removed in future FW versions
 	 */
 	qedn_swap_bytes((u32 *)cqe, (sizeof(*cqe) / sizeof(u32)));
 
-	/* Placeholder - async */
-
-	rc = qedn_comp_valid_task(qedn_task, &cqe->result, cqe->status);
+	if (unlikely(async)) {
+		qedn_return_task_to_pool(conn_ctx, qedn_task);
+		req = qedn_task->req;
+		if (req->done)
+			req->done(req, &cqe->result, cqe->status);
+		else
+			pr_err("request done not set for async request !!!\n");
+	} else {
+		rc = qedn_comp_valid_task(qedn_task, &cqe->result, cqe->status);
+	}
 
 	return rc;
 }
-- 
2.22.0



From smalin at marvell.com  Thu Jun 24 10:24:07 2021
From: smalin at marvell.com (Shai Malin)
Date: Thu, 24 Jun 2021 20:24:07 +0300
Subject: [PATCH v3 17/20] qedn: Add support of NVME ICReq & ICResp
In-Reply-To: <20210624172410.14225-1-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
Message-ID: <20210624172410.14225-18-smalin@marvell.com>

From: Prabhakar Kushwaha <pkushwaha at marvell.com>

Once a TCP connection established, the host sends an Initialize
Connection Request (ICReq) PDU to the controller.
Further Initialize Connection Response (ICResp) PDU received from
controller is processed by host to establish a connection and
exchange connection configuration parameters.

This patch present support of generation of ICReq and processing of
ICResp. It also update host configuration based on exchanged parameters.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |  38 ++++
 drivers/nvme/hw/qedn/qedn_conn.c | 334 ++++++++++++++++++++++++++++++-
 drivers/nvme/hw/qedn/qedn_main.c |  14 ++
 drivers/nvme/hw/qedn/qedn_task.c |   8 +-
 4 files changed, 390 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 78efa8c02810..829d474b3ab1 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -16,6 +16,7 @@
 
 /* Driver includes */
 #include "../../host/tcp-offload.h"
+#include <linux/nvme-tcp.h>
 
 #define QEDN_MODULE_NAME "qedn"
 
@@ -42,6 +43,8 @@
 
 /* Protocol defines */
 #define QEDN_MAX_IO_SIZE QED_NVMETCP_MAX_IO_SIZE
+#define QEDN_MAX_PDU_SIZE 0x80000 /* 512KB */
+#define QEDN_MAX_OUTSTANDING_R2T_PDUS 0 /* 0 Based == 1 max R2T */
 
 #define QEDN_FW_SLOW_IO_MIN_SGE_LIMIT (9700 / 6)
 
@@ -50,6 +53,13 @@
 
 #define QEDN_INVALID_ITID 0xFFFF
 
+#define QEDN_ICREQ_FW_PAYLOAD (sizeof(struct nvme_tcp_icreq_pdu) -\
+			       QED_NVMETCP_NON_IO_HDR_SIZE)
+#define QEDN_ICREQ_FW_PAYLOAD_START 8
+
+/* The FW will handle the ICReq as CCCID 0 (FW internal design) */
+#define QEDN_ICREQ_CCCID 0
+
 /*
  * TCP offload stack default configurations and defines.
  * Future enhancements will allow controlling the configurable
@@ -120,6 +130,16 @@ struct qedn_fp_queue {
 	char irqname[QEDN_IRQ_NAME_LEN];
 };
 
+struct qedn_negotiation_params {
+	u32 maxh2cdata; /* Negotiation */
+	u32 maxr2t; /* Validation */
+	u16 pfv; /* Validation */
+	bool hdr_digest; /* Negotiation */
+	bool data_digest; /* Negotiation */
+	u8 cpda; /* Negotiation */
+	u8 hpda; /* Validation */
+};
+
 struct qedn_ctx {
 	struct pci_dev *pdev;
 	struct qed_dev *cdev;
@@ -176,6 +196,9 @@ struct qedn_endpoint {
 	struct nvmetcp_db_data db_data;
 	void __iomem *p_doorbell;
 
+	/* Spinlock for accessing FW queue */
+	spinlock_t doorbell_lock;
+
 	/* TCP Params */
 	__be32 dst_addr[4]; /* In network order */
 	__be32 src_addr[4]; /* In network order */
@@ -252,6 +275,12 @@ struct qedn_ctrl {
 	atomic_t host_num_active_conns;
 };
 
+struct qedn_icreq_padding {
+	u32 *buffer;
+	dma_addr_t pa;
+	struct nvmetcp_sge sge;
+};
+
 /* Connection level struct */
 struct qedn_conn_ctx {
 	/* IO path */
@@ -300,6 +329,11 @@ struct qedn_conn_ctx {
 
 	size_t sq_depth;
 
+	struct qedn_negotiation_params required_params;
+	struct qedn_negotiation_params pdu_params;
+	struct nvme_tcp_icresp_pdu icresp;
+	struct qedn_icreq_padding *icreq_pad;
+
 	/* "dummy" socket */
 	struct socket *sock;
 };
@@ -308,6 +342,7 @@ enum qedn_conn_resources_state {
 	QEDN_CONN_RESRC_FW_SQ,
 	QEDN_CONN_RESRC_ACQUIRE_CONN,
 	QEDN_CONN_RESRC_TASKS,
+	QEDN_CONN_RESRC_ICREQ_PAD,
 	QEDN_CONN_RESRC_CCCID_ITID_MAP,
 	QEDN_CONN_RESRC_TCP_PORT,
 	QEDN_CONN_RESRC_DB_ADD,
@@ -339,5 +374,8 @@ struct qedn_task_ctx *
 qedn_get_free_task_from_pool(struct qedn_conn_ctx *conn_ctx, u16 cccid);
 void qedn_destroy_free_tasks(struct qedn_fp_queue *fp_q,
 			     struct qedn_io_resources *io_resrc);
+void qedn_prep_icresp(struct qedn_conn_ctx *conn_ctx,
+		      struct nvmetcp_fw_cqe *cqe);
+void qedn_ring_doorbell(struct qedn_conn_ctx *conn_ctx);
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
index 5c5b365df522..b4c0a1a3e890 100644
--- a/drivers/nvme/hw/qedn/qedn_conn.c
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -34,6 +34,18 @@ inline int qedn_qid(struct nvme_tcp_ofld_queue *queue)
 	return queue - queue->ctrl->queues;
 }
 
+void qedn_ring_doorbell(struct qedn_conn_ctx *conn_ctx)
+{
+	u16 prod_idx;
+
+	prod_idx = qed_chain_get_prod_idx(&conn_ctx->ep.fw_sq_chain);
+	conn_ctx->ep.db_data.sq_prod = cpu_to_le16(prod_idx);
+
+	/* wmb - Make sure fw idx is coherent */
+	wmb();
+	writel(*(u32 *)&conn_ctx->ep.db_data, conn_ctx->ep.p_doorbell);
+}
+
 int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
 		       enum qedn_conn_state new_state)
 {
@@ -143,6 +155,71 @@ int qedn_initialize_endpoint(struct qedn_endpoint *ep, u8 *local_mac_addr,
 	return -1;
 }
 
+static int qedn_alloc_icreq_pad(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct qedn_icreq_padding *icreq_pad;
+	u32 *buffer;
+	int rc = 0;
+
+	icreq_pad = kzalloc(sizeof(*icreq_pad), GFP_KERNEL);
+	if (!icreq_pad)
+		return -ENOMEM;
+
+	conn_ctx->icreq_pad = icreq_pad;
+	memset(&icreq_pad->sge, 0, sizeof(icreq_pad->sge));
+	buffer = dma_alloc_coherent(&qedn->pdev->dev,
+				    QEDN_ICREQ_FW_PAYLOAD,
+				    &icreq_pad->pa,
+				    GFP_KERNEL);
+	if (!buffer) {
+		pr_err("Could not allocate icreq_padding SGE buffer.\n");
+		rc =  -ENOMEM;
+		goto release_icreq_pad;
+	}
+
+	DMA_REGPAIR_LE(icreq_pad->sge.sge_addr, icreq_pad->pa);
+	icreq_pad->sge.sge_len = cpu_to_le32(QEDN_ICREQ_FW_PAYLOAD);
+	icreq_pad->buffer = buffer;
+	set_bit(QEDN_CONN_RESRC_ICREQ_PAD, &conn_ctx->resrc_state);
+
+	return 0;
+
+release_icreq_pad:
+	kfree(icreq_pad);
+	conn_ctx->icreq_pad = NULL;
+
+	return rc;
+}
+
+static void qedn_free_icreq_pad(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct qedn_icreq_padding *icreq_pad;
+	u32 *buffer;
+
+	icreq_pad = conn_ctx->icreq_pad;
+	if (unlikely(!icreq_pad)) {
+		pr_err("null ptr in icreq_pad in conn_ctx\n");
+		goto finally;
+	}
+
+	buffer = icreq_pad->buffer;
+	if (buffer) {
+		dma_free_coherent(&qedn->pdev->dev,
+				  QEDN_ICREQ_FW_PAYLOAD,
+				  (void *)buffer,
+				  icreq_pad->pa);
+		icreq_pad->buffer = NULL;
+	}
+
+	kfree(icreq_pad);
+	conn_ctx->icreq_pad = NULL;
+
+finally:
+	clear_bit(QEDN_CONN_RESRC_ICREQ_PAD, &conn_ctx->resrc_state);
+}
+
 static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx)
 {
 	struct qedn_ctx *qedn = conn_ctx->qedn;
@@ -175,6 +252,9 @@ static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx)
 		clear_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state);
 	}
 
+	if (test_bit(QEDN_CONN_RESRC_ICREQ_PAD, &conn_ctx->resrc_state))
+		qedn_free_icreq_pad(conn_ctx);
+
 	if (test_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state)) {
 		clear_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state);
 			qedn_return_active_tasks(conn_ctx);
@@ -336,6 +416,215 @@ void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx)
 	queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
 }
 
+static int qedn_nvmetcp_update_conn(struct qedn_ctx *qedn,
+				    struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_negotiation_params *pdu_params = &conn_ctx->pdu_params;
+	struct qed_nvmetcp_params_update *conn_info;
+	int rc;
+
+	conn_info = kzalloc(sizeof(*conn_info), GFP_KERNEL);
+	if (!conn_info)
+		return -ENOMEM;
+
+	conn_info->hdr_digest_en = pdu_params->hdr_digest;
+	conn_info->data_digest_en = pdu_params->data_digest;
+	conn_info->max_recv_pdu_length = QEDN_MAX_PDU_SIZE;
+	conn_info->max_io_size = QEDN_MAX_IO_SIZE;
+	conn_info->max_send_pdu_length = pdu_params->maxh2cdata;
+
+	rc = qed_ops->update_conn(qedn->cdev, conn_ctx->conn_handle, conn_info);
+	if (rc) {
+		pr_err("Could not update connection\n");
+		rc = -ENXIO;
+	}
+
+	kfree(conn_info);
+
+	return rc;
+}
+
+static int qedn_update_ramrod(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int rc = 0;
+
+	rc = qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_UPDATE_EQE);
+	if (rc)
+		return rc;
+
+	rc = qedn_nvmetcp_update_conn(qedn, conn_ctx);
+	if (rc)
+		return rc;
+
+	if (conn_ctx->state != CONN_STATE_WAIT_FOR_UPDATE_EQE) {
+		pr_err("cid 0x%x: Unexpected state 0x%x after update ramrod\n",
+		       conn_ctx->fw_cid, conn_ctx->state);
+
+		return -EINVAL;
+	}
+
+	return rc;
+}
+
+static int qedn_send_icreq(struct qedn_conn_ctx *conn_ctx)
+{
+	struct storage_sgl_task_params *sgl_task_params;
+	struct nvmetcp_task_params task_params;
+	struct qedn_task_ctx *qedn_task = NULL;
+	struct nvme_tcp_icreq_pdu icreq;
+	struct nvmetcp_wqe *chain_sqe;
+	struct nvmetcp_wqe local_sqe;
+
+	qedn_task = qedn_get_free_task_from_pool(conn_ctx, QEDN_ICREQ_CCCID);
+	if (!qedn_task)
+		return -EINVAL;
+
+	memset(&icreq, 0, sizeof(icreq));
+	memset(&local_sqe, 0, sizeof(local_sqe));
+
+	/* Initialize ICReq */
+	icreq.hdr.type = nvme_tcp_icreq;
+	icreq.hdr.hlen = sizeof(icreq);
+	icreq.hdr.pdo = 0;
+	icreq.hdr.plen = cpu_to_le32(icreq.hdr.hlen);
+	icreq.pfv = cpu_to_le16(conn_ctx->required_params.pfv);
+	icreq.maxr2t = cpu_to_le32(conn_ctx->required_params.maxr2t);
+	icreq.hpda = conn_ctx->required_params.hpda;
+	if (conn_ctx->required_params.hdr_digest)
+		icreq.digest |= NVME_TCP_HDR_DIGEST_ENABLE;
+	if (conn_ctx->required_params.data_digest)
+		icreq.digest |= NVME_TCP_DATA_DIGEST_ENABLE;
+
+	/* Initialize task params */
+	task_params.opq.lo = cpu_to_le32(((u64)(qedn_task)) & 0xffffffff);
+	task_params.opq.hi = cpu_to_le32(((u64)(qedn_task)) >> 32);
+	task_params.context = qedn_task->fw_task_ctx;
+	task_params.sqe = &local_sqe;
+	task_params.conn_icid = (u16)conn_ctx->conn_handle;
+	task_params.itid = qedn_task->itid;
+	task_params.cq_rss_number = conn_ctx->default_cq;
+	task_params.tx_io_size = QEDN_ICREQ_FW_PAYLOAD;
+	task_params.rx_io_size = 0; /* Rx doesn't use SGL for icresp */
+
+	/* Init SGE for ICReq padding */
+	sgl_task_params = &qedn_task->sgl_task_params;
+	sgl_task_params->total_buffer_size = task_params.tx_io_size;
+	sgl_task_params->small_mid_sge = false;
+	sgl_task_params->num_sges = 1;
+	memcpy(sgl_task_params->sgl, &conn_ctx->icreq_pad->sge,
+	       sizeof(conn_ctx->icreq_pad->sge));
+
+	/* ICReq is sent as two parts.
+	 * First part: (16 bytes + First 8 bytes of icreq.rsvd2[]) are sent
+	 *             via task context which is initialized above in icreq
+	 * Second part: Rest bytes are sent via SGE, happening here
+	 */
+	memcpy(conn_ctx->icreq_pad->buffer,
+	       &icreq.rsvd2[QEDN_ICREQ_FW_PAYLOAD_START],
+	       QEDN_ICREQ_FW_PAYLOAD);
+
+	qed_ops->init_icreq_exchange(&task_params, &icreq,
+				     sgl_task_params,  NULL);
+
+	qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_IC_COMP);
+	atomic_inc(&conn_ctx->num_active_fw_tasks);
+
+	/* spin_lock - doorbell is accessed  both Rx flow and response flow */
+	spin_lock(&conn_ctx->ep.doorbell_lock);
+	chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain);
+	memcpy(chain_sqe, &local_sqe, sizeof(local_sqe));
+	qedn_ring_doorbell(conn_ctx);
+	spin_unlock(&conn_ctx->ep.doorbell_lock);
+
+	return 0;
+}
+
+void qedn_prep_icresp(struct qedn_conn_ctx *conn_ctx,
+		      struct nvmetcp_fw_cqe *cqe)
+{
+	struct nvmetcp_icresp_mdata *icresp_from_cqe =
+		(struct nvmetcp_icresp_mdata *)&cqe->cqe_data.icresp_mdata;
+	struct nvme_tcp_icresp_pdu *icresp = &conn_ctx->icresp;
+	struct nvme_tcp_ofld_ctrl *ctrl = conn_ctx->ctrl;
+	struct qedn_ctrl *qctrl = NULL;
+
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return;
+
+	icresp->pfv = cpu_to_le16(icresp_from_cqe->pfv);
+	icresp->cpda = icresp_from_cqe->cpda;
+	icresp->digest = icresp_from_cqe->digest;
+	icresp->maxdata = cpu_to_le32(icresp_from_cqe->maxdata);
+
+	qedn_set_sp_wa(conn_ctx, HANDLE_ICRESP);
+	queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
+}
+
+static int qedn_handle_icresp(struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_tcp_icresp_pdu *icresp = &conn_ctx->icresp;
+	struct qedn_negotiation_params *pdu_params;
+	int rc = 0;
+	u16 pfv;
+
+	/* Swapping requirement will be removed in future FW versions */
+	pfv = __swab16(le16_to_cpu(icresp->pfv));
+
+	qedn_free_icreq_pad(conn_ctx);
+
+	/* Validate ICResp */
+	if (pfv != conn_ctx->required_params.pfv) {
+		pr_err("cid %u: unsupported pfv %u\n", conn_ctx->fw_cid, pfv);
+
+		return -EINVAL;
+	}
+
+	if (icresp->cpda > conn_ctx->required_params.cpda) {
+		pr_err("cid %u: unsupported cpda %u\n",
+		       conn_ctx->fw_cid, icresp->cpda);
+
+		return -EINVAL;
+	}
+
+	if ((NVME_TCP_HDR_DIGEST_ENABLE & icresp->digest) !=
+	    conn_ctx->required_params.hdr_digest) {
+		if ((NVME_TCP_HDR_DIGEST_ENABLE & icresp->digest) >
+		    conn_ctx->required_params.hdr_digest) {
+			pr_err("cid 0x%x: invalid header digest bit\n",
+			       conn_ctx->fw_cid);
+		}
+	}
+
+	if ((NVME_TCP_DATA_DIGEST_ENABLE & icresp->digest) !=
+	    conn_ctx->required_params.data_digest) {
+		if ((NVME_TCP_DATA_DIGEST_ENABLE & icresp->digest) >
+		    conn_ctx->required_params.data_digest) {
+			pr_err("cid 0x%x: invalid data digest bit\n",
+			       conn_ctx->fw_cid);
+	}
+	}
+
+	pdu_params = &conn_ctx->pdu_params;
+	memset(pdu_params, 0, sizeof(conn_ctx->pdu_params));
+	/* Swapping requirement will be removed in future FW versions */
+	pdu_params->maxh2cdata = __swab32(le32_to_cpu(icresp->maxdata));
+	pdu_params->maxh2cdata = QEDN_MAX_PDU_SIZE;
+	if (pdu_params->maxh2cdata > QEDN_MAX_PDU_SIZE)
+		pdu_params->maxh2cdata = QEDN_MAX_PDU_SIZE;
+
+	pdu_params->pfv = pfv;
+	pdu_params->cpda = icresp->cpda;
+	pdu_params->hpda = conn_ctx->required_params.hpda;
+	pdu_params->hdr_digest = NVME_TCP_HDR_DIGEST_ENABLE & icresp->digest;
+	pdu_params->data_digest = NVME_TCP_DATA_DIGEST_ENABLE & icresp->digest;
+	pdu_params->maxr2t = conn_ctx->required_params.maxr2t;
+	rc = qedn_update_ramrod(conn_ctx);
+
+	return rc;
+}
+
 /* Slowpath EQ Callback */
 int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
 {
@@ -393,7 +682,8 @@ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
 			if (rc)
 				return rc;
 
-			/* Placeholder - for ICReq flow */
+			qedn_set_sp_wa(conn_ctx, SEND_ICREQ);
+			queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
 		}
 
 		break;
@@ -445,6 +735,8 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 
 	set_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state);
 
+	spin_lock_init(&conn_ctx->ep.doorbell_lock);
+
 	atomic_set(&conn_ctx->num_active_tasks, 0);
 	atomic_set(&conn_ctx->num_active_fw_tasks, 0);
 
@@ -509,6 +801,11 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 
 	memset(conn_ctx->host_cccid_itid, 0xFF, dma_size);
 	set_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state);
+
+	rc = qedn_alloc_icreq_pad(conn_ctx);
+		if (rc)
+			goto rel_conn;
+
 	rc = qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_CONNECT_DONE);
 	if (rc)
 		goto rel_conn;
@@ -581,6 +878,9 @@ void qedn_sp_wq_handler(struct work_struct *work)
 
 	qedn = conn_ctx->qedn;
 	if (test_bit(DESTROY_CONNECTION, &conn_ctx->agg_work_action)) {
+		if (test_bit(HANDLE_ICRESP, &conn_ctx->agg_work_action))
+			qedn_clr_sp_wa(conn_ctx, HANDLE_ICRESP);
+
 		qedn_destroy_connection(conn_ctx);
 
 		return;
@@ -595,6 +895,38 @@ void qedn_sp_wq_handler(struct work_struct *work)
 			return;
 		}
 	}
+
+	if (test_bit(SEND_ICREQ, &conn_ctx->agg_work_action)) {
+		qedn_clr_sp_wa(conn_ctx, SEND_ICREQ);
+		rc = qedn_send_icreq(conn_ctx);
+		if (rc)
+			return;
+
+		return;
+	}
+
+	if (test_bit(HANDLE_ICRESP, &conn_ctx->agg_work_action)) {
+		rc = qedn_handle_icresp(conn_ctx);
+
+		qedn_clr_sp_wa(conn_ctx, HANDLE_ICRESP);
+		if (rc) {
+			pr_err("IC handling returned with 0x%x\n", rc);
+			if (test_and_set_bit(DESTROY_CONNECTION,
+					     &conn_ctx->agg_work_action))
+				return;
+
+			qedn_destroy_connection(conn_ctx);
+
+			return;
+		}
+
+		atomic_inc(&conn_ctx->est_conn_indicator);
+		qedn_set_con_state(conn_ctx,
+				   CONN_STATE_NVMETCP_CONN_ESTABLISHED);
+		wake_up_interruptible(&conn_ctx->conn_waitq);
+
+		return;
+	}
 }
 
 /* Clear connection aggregative slowpath work action */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 42c8ad6ac2d6..3cf913d527c0 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -286,6 +286,19 @@ static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 	return 0;
 }
 
+static void qedn_set_pdu_params(struct qedn_conn_ctx *conn_ctx)
+{
+	/* Enable digest once supported */
+	conn_ctx->required_params.hdr_digest = 0;
+	conn_ctx->required_params.data_digest = 0;
+
+	conn_ctx->required_params.maxr2t = QEDN_MAX_OUTSTANDING_R2T_PDUS;
+	conn_ctx->required_params.pfv = NVME_TCP_PFV_1_0;
+	conn_ctx->required_params.cpda = 0;
+	conn_ctx->required_params.hpda = 0;
+	conn_ctx->required_params.maxh2cdata = QEDN_MAX_PDU_SIZE;
+}
+
 static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 			     size_t queue_size)
 {
@@ -317,6 +330,7 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 	conn_ctx->ctrl = ctrl;
 	conn_ctx->sq_depth = queue_size;
 	mutex_init(&conn_ctx->send_mutex);
+	qedn_set_pdu_params(conn_ctx);
 
 	init_waitqueue_head(&conn_ctx->conn_waitq);
 	atomic_set(&conn_ctx->est_conn_indicator, 0);
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
index 7b228b5c5169..e52460f9650e 100644
--- a/drivers/nvme/hw/qedn/qedn_task.c
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -392,9 +392,11 @@ void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
 			break;
 
 		case NVMETCP_TASK_TYPE_INIT_CONN_REQUEST:
-
-			/* Placeholder - ICReq flow */
-
+			/* Clear ICReq-padding SGE from SGL */
+			qedn_common_clear_fw_sgl(&qedn_task->sgl_task_params);
+			/* Task is not required for icresp processing */
+			qedn_return_task_to_pool(conn_ctx, qedn_task);
+			qedn_prep_icresp(conn_ctx, cqe);
 			break;
 		default:
 			pr_info("Could not identify task type\n");
-- 
2.22.0



From cachen at purestorage.com  Thu Jun 24 10:31:23 2021
From: cachen at purestorage.com (Casey Chen)
Date: Thu, 24 Jun 2021 10:31:23 -0700
Subject: [PATCH 0/2] rr: nvme-pci: fix races and UAF
Message-ID: <20210624173125.20376-1-cachen@purestorage.com>

Found two bugs while power-cycling PCIe NVMe drive for hours:
- Races in nvme_setup_io_queues()
- UAF introduced by nvme_dev_remove_admin(), which is found
after fixing the races. Without fixing the races, the system
just crashes and cannot survive to reproduce the UAF.

The proposed fixes have been tested for several days for correctness.

0. Code baseline

Tag nvme-5.14-2021-06-08 of repo http://git.infradead.org/nvme.git

1. Testing method

while :; power off one drive;
    sleep $((RANDOM%3)).$((RANDOM%10));
    power on the same drive;
    sleep $((RANDOM%3)).$((RANDOM%10));
done

2. Sample crash trace due to races in nvme_setup_io_queues()
(task ID shown after the timestamps)

[11668.533431][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Card present
...
[11668.681298][T251231] nvme nvme12: pci function 0000:8c:00.0
[11668.681354][T26714] nvme 0000:8c:00.0: enabling device (0100 -> 0102)
[11669.046119][   C31] pcieport 0000:87:08.0: pciehp: pending interrupts 0x0108 from Slot Status
[11669.046142][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Link Down
[11669.046146][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Card not present
[11669.046149][  T716] pcieport 0000:87:08.0: pciehp: pciehp_unconfigure_device: domain:bus:dev = 0000:8c:00
[11669.077428][  T716] ------------[ cut here ]------------
[11669.077431][  T716] kernel BUG at drivers/pci/msi.c:348!
[11669.077555][  T716] invalid opcode: 0000 [#1] SMP KASAN
[11669.077658][  T716] CPU: 31 PID: 716 Comm: irq/127-pciehp Not tainted 5.13.0-rc3+
[11669.078022][  T716] RIP: 0010:free_msi_irqs+0x28a/0x2d0
...
[11669.093982][  T716] Call Trace:
[11669.096850][  T716]  pci_free_irq_vectors+0xe/0x20
[11669.099695][  T716]  nvme_dev_disable+0x140/0x760 [nvme]
[11669.102503][  T716]  ? _raw_spin_lock_irqsave+0x9c/0x100
[11669.105271][  T716]  ? trace_hardirqs_on+0x2c/0xe0
[11669.107994][  T716]  nvme_remove+0x191/0x1e0 [nvme]
[11669.110689][  T716]  pci_device_remove+0x6b/0x110
[11669.113316][  T716]  device_release_driver_internal+0x14f/0x280
[11669.115939][  T716]  pci_stop_bus_device+0xcb/0x100
[11669.118515][  T716]  pci_stop_and_remove_bus_device+0xe/0x20
[11669.121079][  T716]  pciehp_unconfigure_device+0xfa/0x200
[11669.123597][  T716]  ? pciehp_configure_device+0x1c0/0x1c0
[11669.126049][  T716]  ? trace_hardirqs_on+0x2c/0xe0
[11669.128444][  T716]  pciehp_disable_slot+0xc4/0x1a0
[11669.130771][  T716]  ? pciehp_runtime_suspend+0x40/0x40
[11669.133054][  T716]  ? __mutex_lock_slowpath+0x10/0x10
[11669.135289][  T716]  ? trace_hardirqs_on+0x2c/0xe0
[11669.137462][  T716]  pciehp_handle_presence_or_link_change+0x15c/0x4f0
[11669.139632][  T716]  ? down_read+0x11f/0x1a0
[11669.141731][  T716]  ? pciehp_handle_disable_request+0x80/0x80
[11669.143817][  T716]  ? rwsem_down_read_slowpath+0x600/0x600
[11669.145851][  T716]  ? __radix_tree_lookup+0xb2/0x130
[11669.147842][  T716]  pciehp_ist+0x19d/0x1a0
[11669.149790][  T716]  ? pciehp_set_indicators+0xe0/0xe0
[11669.151704][  T716]  ? irq_finalize_oneshot.part.46+0x1d0/0x1d0
[11669.153588][  T716]  irq_thread_fn+0x3f/0xa0
[11669.155407][  T716]  irq_thread+0x195/0x290
[11669.157147][  T716]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
[11669.158883][  T716]  ? _raw_read_lock_irq+0x50/0x50
[11669.160611][  T716]  ? _raw_read_lock_irq+0x50/0x50
[11669.162320][  T716]  ? irq_forced_thread_fn+0xf0/0xf0
[11669.164032][  T716]  ? trace_hardirqs_on+0x2c/0xe0
[11669.165731][  T716]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
[11669.167461][  T716]  kthread+0x1c8/0x1f0
[11669.169173][  T716]  ? kthread_parkme+0x40/0x40
[11669.170883][  T716]  ret_from_fork+0x22/0x30

3. KASAN report for the UAF introduced by nvme_dev_remove_admin()
(task ID is shown after the timestamp)

[18319.015748][T246989] nvme nvme13: pci function 0000:8c:00.0
[18319.015795][T215541] nvme 0000:8c:00.0: enabling device (0100 -> 0102)
[18319.369086][   C31] pcieport 0000:87:08.0: pciehp: pending interrupts 0x0108 from Slot Status
[18319.369107][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Link Down
[18319.369111][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Card not present
[18319.369116][  T716] pcieport 0000:87:08.0: pciehp: pciehp_unconfigure_device: domain:bus:dev = 0000:8c:00
[18320.452045][T215541] nvme nvme13: 88/0/0 default/read/poll queues
[18320.469475][T215541] nvme nvme13: failed to mark controller live state
[18320.469483][T215541] nvme nvme13: Removing after probe failure status: -19
[18320.551295][T215541] ==================================================================
[18320.551299][T215541] BUG: KASAN: use-after-free in __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551311][T215541] Read of size 4 at addr ffff888897904d04 by task kworker/u178:2/215541
[18320.551315][T215541] 
[18320.551318][T215541] CPU: 86 PID: 215541 Comm: kworker/u178:2 Not tainted 5.13.0-rc3+
[18320.551327][T215541] Workqueue: nvme-reset-wq nvme_reset_work [nvme]
[18320.551339][T215541] Call Trace:
[18320.551343][T215541]  dump_stack+0xa4/0xdb
[18320.551354][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551359][T215541]  print_address_description.constprop.10+0x3a/0x60
[18320.551366][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551372][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551377][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
[18320.551382][T215541]  kasan_report.cold.15+0x7c/0xd8
[18320.551390][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551395][T215541]  __blk_mq_all_tag_iter+0x9c/0x3f0
[18320.551401][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
[18320.551407][T215541]  ? bt_iter+0xf0/0xf0
[18320.551412][T215541]  ? __blk_mq_all_tag_iter+0x2c9/0x3f0
[18320.551417][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
[18320.551422][T215541]  ? bt_iter+0xf0/0xf0
[18320.551427][T215541]  ? dev_printk_emit+0x95/0xbb
[18320.551436][T215541]  blk_mq_tagset_busy_iter+0x75/0xa0
[18320.551441][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
[18320.551446][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
[18320.551451][T215541]  blk_mq_tagset_wait_completed_request+0x86/0xc0
[18320.551457][T215541]  ? blk_mq_tagset_busy_iter+0xa0/0xa0
[18320.551463][T215541]  ? blk_mq_tagset_busy_iter+0x80/0xa0
[18320.551469][T215541]  ? trace_event_raw_event_nvme_setup_cmd+0x2d0/0x2d0 [nvme_core]
[18320.551493][T215541]  nvme_dev_disable+0x4f6/0x760 [nvme]
[18320.551502][T215541]  ? trace_hardirqs_on+0x2c/0xe0
[18320.551510][T215541]  nvme_reset_work+0x226/0x2060 [nvme]
[18320.551520][T215541]  ? nvme_remove+0x1e0/0x1e0 [nvme]
[18320.551528][T215541]  ? __update_load_avg_cfs_rq+0x1d8/0x550
[18320.551537][T215541]  ? down_read+0x11f/0x1a0
[18320.551545][T215541]  ? newidle_balance+0x444/0x690
[18320.551552][T215541]  ? update_load_avg+0x626/0xbe0
[18320.551557][T215541]  ? update_cfs_group+0x1e/0x150
[18320.551562][T215541]  ? load_balance+0x11d0/0x11d0
[18320.551567][T215541]  ? dequeue_entity+0x150/0x730
[18320.551573][T215541]  ? nvme_irq_check+0x60/0x60 [nvme]
[18320.551581][T215541]  ? finish_task_switch+0x101/0x3d0
[18320.551588][T215541]  ? read_word_at_a_time+0xe/0x20
[18320.551594][T215541]  ? strscpy+0xc1/0x1d0
[18320.551598][T215541]  process_one_work+0x4b9/0x7b0
[18320.551604][T215541]  worker_thread+0x72/0x710
[18320.551610][T215541]  ? process_one_work+0x7b0/0x7b0
[18320.551614][T215541]  kthread+0x1c8/0x1f0
[18320.551618][T215541]  ? kthread_parkme+0x40/0x40
[18320.551622][T215541]  ret_from_fork+0x22/0x30
[18320.551630][T215541] 
[18320.551632][T215541] Allocated by task 215541:
[18320.551635][T215541]  kasan_save_stack+0x19/0x40
[18320.551639][T215541]  __kasan_kmalloc+0x7f/0xa0
[18320.551642][T215541]  kmem_cache_alloc_node_trace+0x187/0x2b0
[18320.551648][T215541]  blk_mq_init_tags+0x47/0x100
[18320.551651][T215541]  blk_mq_alloc_rq_map+0x44/0xf0
[18320.551656][T215541]  __blk_mq_alloc_map_and_request+0x7f/0x140
[18320.551661][T215541]  blk_mq_alloc_tag_set+0x25e/0x510
[18320.551666][T215541]  nvme_reset_work+0x14f9/0x2060 [nvme]
[18320.551674][T215541]  process_one_work+0x4b9/0x7b0
[18320.551678][T215541]  worker_thread+0x72/0x710
[18320.551682][T215541]  kthread+0x1c8/0x1f0
[18320.551685][T215541]  ret_from_fork+0x22/0x30
[18320.551689][T215541] 
[18320.551690][T215541] Freed by task 716:
[18320.551693][T215541]  kasan_save_stack+0x19/0x40
[18320.551696][T215541]  kasan_set_track+0x1c/0x30
[18320.551699][T215541]  kasan_set_free_info+0x20/0x30
[18320.551704][T215541]  __kasan_slab_free+0xec/0x130
[18320.551707][T215541]  kfree+0xa8/0x460
[18320.551712][T215541]  blk_mq_free_map_and_requests+0x8d/0xc0
[18320.551717][T215541]  blk_mq_free_tag_set+0x30/0xf0
[18320.551721][T215541]  nvme_remove+0x199/0x1e0 [nvme]
[18320.551729][T215541]  pci_device_remove+0x6b/0x110
[18320.551735][T215541]  device_release_driver_internal+0x14f/0x280
[18320.551744][T215541]  pci_stop_bus_device+0xcb/0x100
[18320.551750][T215541]  pci_stop_and_remove_bus_device+0xe/0x20
[18320.551754][T215541]  pciehp_unconfigure_device+0xfa/0x200
[18320.551761][T215541]  pciehp_disable_slot+0xc4/0x1a0
[18320.551765][T215541]  pciehp_handle_presence_or_link_change+0x15c/0x4f0
[18320.551770][T215541]  pciehp_ist+0x19d/0x1a0
[18320.551774][T215541]  irq_thread_fn+0x3f/0xa0
[18320.551780][T215541]  irq_thread+0x195/0x290
[18320.551783][T215541]  kthread+0x1c8/0x1f0
[18320.551786][T215541]  ret_from_fork+0x22/0x30
[18320.551791][T215541] 
[18320.551792][T215541] The buggy address belongs to the object at ffff888897904d00
[18320.551792][T215541]  which belongs to the cache kmalloc-192 of size 192
[18320.551795][T215541] The buggy address is located 4 bytes inside of
[18320.551795][T215541]  192-byte region [ffff888897904d00, ffff888897904dc0)
[18320.551800][T215541] The buggy address belongs to the page:
[18320.551802][T215541] page:000000002f3df664 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x897904
[18320.551807][T215541] head:000000002f3df664 order:1 compound_mapcount:0
[18320.551810][T215541] flags: 0x200000000010200(slab|head|node=0|zone=2)
[18320.551819][T215541] raw: 0200000000010200 dead000000000100 dead000000000122 ffff88810004ca00
[18320.551824][T215541] raw: 0000000000000000 0000000000200020 00000001ffffffff 0000000000000000
[18320.551826][T215541] page dumped because: kasan: bad access detected
[18320.551828][T215541] 
[18320.551829][T215541] Memory state around the buggy address:
[18320.551832][T215541]  ffff888897904c00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[18320.551835][T215541]  ffff888897904c80: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
[18320.551838][T215541] >ffff888897904d00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[18320.551841][T215541]                    ^
[18320.551843][T215541]  ffff888897904d80: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
[18320.551846][T215541]  ffff888897904e00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[18320.551848][T215541] ==================================================================
[18320.551850][T215541] Disabling lock debugging due to kernel taint

Casey Chen (2):
  nvme-pci: Fix multiple races in nvme_setup_io_queues()
  nvme-pci: Fix UAF introduced by nvme_dev_remove_admin()

 drivers/nvme/host/pci.c | 60 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 53 insertions(+), 7 deletions(-)

-- 
2.17.1



From cachen at purestorage.com  Thu Jun 24 10:31:24 2021
From: cachen at purestorage.com (Casey Chen)
Date: Thu, 24 Jun 2021 10:31:24 -0700
Subject: [PATCH 1/2] nvme-pci: Fix multiple races in nvme_setup_io_queues()
In-Reply-To: <20210624173125.20376-1-cachen@purestorage.com>
References: <20210624173125.20376-1-cachen@purestorage.com>
Message-ID: <20210624173125.20376-2-cachen@purestorage.com>

Below two paths could overlap each other if we power off a drive quickly
after powering it on. There are multiple races in nvme_setup_io_queues()
because of shutdown_lock missing and improper use of NVMEQ_ENABLED bit.

nvme_reset_work()                                nvme_remove()
  nvme_setup_io_queues()                           nvme_dev_disable()
  ...                                              ...
A1  clear NVMEQ_ENABLED bit for admin queue          lock
    retry:                                       B1  nvme_suspend_io_queues()
A2    pci_free_irq() admin queue                 B2  nvme_suspend_queue() admin queue
A3    pci_free_irq_vectors()                         nvme_pci_disable()
A4    nvme_setup_irqs();                         B3    pci_free_irq_vectors()
      ...                                            unlock
A5    queue_request_irq() for admin queue
      set NVMEQ_ENABLED bit
      ...
      nvme_create_io_queues()
A6      result = queue_request_irq();
        set NVMEQ_ENABLED bit
      ...
      fail to allocate enough IO queues:
A7      nvme_suspend_io_queues()
        goto retry

1). If B3 runs in between A1 and A2, it will crash if irqaction haven't
  been freed by A2. B2 is supposed to free admin queue IRQ but it simply
  can't fulfill the job as A1 has cleared NVMEQ_ENABLED bit.

Fix: combine A1 A2 so IRQ get freed as soon as NVMEQ_ENABLED bit get cleared.

2). After solved #1, A2 could race with B3 if A2 is freeing IRQ while B3
  is checking irqaction. A3 also could race with B2 if B2 is freeing
  IRQ while A3 is checking irqaction.

Fix: A2 and A3 take lock for mutual exclusion.

3). A3 could race with B3 since they could run free_msi_irqs() in parallel.

Fix: A3 takes lock for mutual exclusion.

4). A4 could fail to allocate all needed IRQ vectors if A3 and A4 are
  interrupted by B3.

Fix: A4 takes lock for mutual exclusion.

5). If A5/A6 happened after B2/B1, B3 will crash since irqaction is not NULL.
  They are just allocated by A5/A6.

Fix: Lock queue_request_irq() and setting of NVMEQ_ENABLED bit.

6). A7 could get chance to pci_free_irq() for certain IO queue while B3
  is checking irqaction.

Fix: A7 takes lock.

7). nvme_dev->online_queues need to be protected by shutdown_lock. Since it
  is not atomic, both paths could modify it using its own copy.

Co-developed-by: Yuanyuan Zhong <yzhong at purestorage.com>
Signed-off-by: Casey Chen <cachen at purestorage.com>
---
 drivers/nvme/host/pci.c | 59 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 53 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3aa7245a505f..378ce9c060af 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1562,6 +1562,29 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
 	wmb(); /* ensure the first interrupt sees the initialization */
 }
 
+/*
+ * Try getting shutdown_lock while setting up IO queues.
+ * Caller remember to unlock if success.
+ */
+static int nvme_setup_io_queues_trylock(struct nvme_dev *dev)
+{
+	/*
+	 * Lock is being held by nvme_dev_disable(), fail early.
+	 */
+	if (!mutex_trylock(&dev->shutdown_lock))
+		return -ENODEV;
+
+	/*
+	 * Controller is in wrong state, fail early.
+	 */
+	if (dev->ctrl.state != NVME_CTRL_CONNECTING) {
+		mutex_unlock(&dev->shutdown_lock);
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
 static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
 {
 	struct nvme_dev *dev = nvmeq->dev;
@@ -1590,8 +1613,10 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
 		goto release_cq;
 
 	nvmeq->cq_vector = vector;
-	nvme_init_queue(nvmeq, qid);
 
+	if ((result = nvme_setup_io_queues_trylock(dev)))
+		return result;
+	nvme_init_queue(nvmeq, qid);
 	if (!polled) {
 		result = queue_request_irq(nvmeq);
 		if (result < 0)
@@ -1599,10 +1624,12 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
 	}
 
 	set_bit(NVMEQ_ENABLED, &nvmeq->flags);
+	mutex_unlock(&dev->shutdown_lock);
 	return result;
 
 release_sq:
 	dev->online_queues--;
+	mutex_unlock(&dev->shutdown_lock);
 	adapter_delete_sq(dev, qid);
 release_cq:
 	adapter_delete_cq(dev, qid);
@@ -2176,7 +2203,17 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	if (nr_io_queues == 0)
 		return 0;
 
-	clear_bit(NVMEQ_ENABLED, &adminq->flags);
+	/*
+	 * Free IRQ resources as soon as NVMEQ_ENABLED bit transitions
+	 * from set to unset. If there is a window to it is truely freed,
+	 * pci_free_irq_vectors() jumping into this window will crash.
+	 * And take lock to avoid racing with pci_free_irq_vectors() in
+	 * nvme_dev_disable() path.
+	 */
+	if ((result = nvme_setup_io_queues_trylock(dev)))
+		return result;
+	if (test_and_clear_bit(NVMEQ_ENABLED, &adminq->flags))
+		pci_free_irq(pdev, 0, adminq);
 
 	if (dev->cmb_use_sqes) {
 		result = nvme_cmb_qdepth(dev, nr_io_queues,
@@ -2192,14 +2229,17 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 		result = nvme_remap_bar(dev, size);
 		if (!result)
 			break;
-		if (!--nr_io_queues)
+		if (!--nr_io_queues) {
+			mutex_unlock(&dev->shutdown_lock);
 			return -ENOMEM;
+		}
 	} while (1);
 	adminq->q_db = dev->dbs;
 
  retry:
 	/* Deregister the admin queue's interrupt */
-	pci_free_irq(pdev, 0, adminq);
+	if (test_and_clear_bit(NVMEQ_ENABLED, &adminq->flags))
+		pci_free_irq(pdev, 0, adminq);
 
 	/*
 	 * If we enable msix early due to not intx, disable it again before
@@ -2208,8 +2248,10 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	pci_free_irq_vectors(pdev);
 
 	result = nvme_setup_irqs(dev, nr_io_queues);
-	if (result <= 0)
+	if (result <= 0) {
+		mutex_unlock(&dev->shutdown_lock);
 		return -EIO;
+	}
 
 	dev->num_vecs = result;
 	result = max(result - 1, 1);
@@ -2222,9 +2264,12 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	 * number of interrupts.
 	 */
 	result = queue_request_irq(adminq);
-	if (result)
+	if (result) {
+		mutex_unlock(&dev->shutdown_lock);
 		return result;
+	}
 	set_bit(NVMEQ_ENABLED, &adminq->flags);
+	mutex_unlock(&dev->shutdown_lock);
 
 	result = nvme_create_io_queues(dev);
 	if (result || dev->online_queues < 2)
@@ -2233,6 +2278,8 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	if (dev->online_queues - 1 < dev->max_qid) {
 		nr_io_queues = dev->online_queues - 1;
 		nvme_disable_io_queues(dev);
+		if ((result = nvme_setup_io_queues_trylock(dev)))
+			return result;
 		nvme_suspend_io_queues(dev);
 		goto retry;
 	}
-- 
2.17.1



From cachen at purestorage.com  Thu Jun 24 10:31:25 2021
From: cachen at purestorage.com (Casey Chen)
Date: Thu, 24 Jun 2021 10:31:25 -0700
Subject: [PATCH 2/2] nvme-pci: Fix UAF introduced by nvme_dev_remove_admin()
In-Reply-To: <20210624173125.20376-1-cachen@purestorage.com>
References: <20210624173125.20376-1-cachen@purestorage.com>
Message-ID: <20210624173125.20376-3-cachen@purestorage.com>

nvme_dev_remove_admin() could free admin_q and admin_tagset while
they are being accessed by nvme_dev_disable(), which could come from
nvme_remove_dead_ctrl() by nvme_reset_work() during cleanup.

Commit cb4bfda62afa ("nvme-pci: fix hot removal during error handling") was
to avoid requests being stuck on a removed controller by killing admin queue.
But the later fix c8e9e9b7646e ("nvme-pci: unquiesce admin queue on shutdown"),
together with nvme_dev_disable(dev, true) right before nvme_dev_remove_admin()
could help dispatch requests and fail them early, so we don't need
nvme_dev_remove_admin() any more.

Fixes: cb4bfda62afa ("nvme-pci: fix hot removal during error handling")

Signed-off-by: Casey Chen <cachen at purestorage.com>
---
 drivers/nvme/host/pci.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 378ce9c060af..9e843d571617 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -3045,7 +3045,6 @@ static void nvme_remove(struct pci_dev *pdev)
 	if (!pci_device_is_present(pdev)) {
 		nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DEAD);
 		nvme_dev_disable(dev, true);
-		nvme_dev_remove_admin(dev);
 	}
 
 	flush_work(&dev->ctrl.reset_work);
-- 
2.17.1



From Chaitanya.Kulkarni at wdc.com  Thu Jun 24 15:42:57 2021
From: Chaitanya.Kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Thu, 24 Jun 2021 22:42:57 +0000
Subject: [PATCH 2/2] nvme-pci: Fix UAF introduced by
 nvme_dev_remove_admin()
References: <20210624173125.20376-1-cachen@purestorage.com>
 <20210624173125.20376-3-cachen@purestorage.com>
Message-ID: <BYAPR04MB496562CD675B008BEE8AB2B386079@BYAPR04MB4965.namprd04.prod.outlook.com>

On 6/24/21 11:08, Casey Chen wrote:
> nvme_dev_remove_admin() could free admin_q and admin_tagset while
> they are being accessed by nvme_dev_disable(), which could come from
> nvme_remove_dead_ctrl() by nvme_reset_work() during cleanup.
>
> Commit cb4bfda62afa ("nvme-pci: fix hot removal during error handling") was
> to avoid requests being stuck on a removed controller by killing admin queue.
> But the later fix c8e9e9b7646e ("nvme-pci: unquiesce admin queue on shutdown"),
> together with nvme_dev_disable(dev, true) right before nvme_dev_remove_admin()
> could help dispatch requests and fail them early, so we don't need
> nvme_dev_remove_admin() any more.
>
> Fixes: cb4bfda62afa ("nvme-pci: fix hot removal during error handling")

consider following reformatted commit message without any content change :-

nvme_dev_remove_admin() could free admin_q and admin_tagset while
they are being accessed by nvme_dev_disable(), which could come from
nvme_remove_dead_ctrl() by nvme_reset_work() during cleanup.

Commit cb4bfda62afa ("nvme-pci: fix hot removal during error handling")
was to avoid requests being stuck on a removed controller by killing
admin queue. But the later fix c8e9e9b7646e ("nvme-pci: unquiesce
admin queue on shutdown"), together with nvme_dev_disable(dev, true)
right before nvme_dev_remove_admin() could help dispatch requests and
fail them early, so we don't need nvme_dev_remove_admin() any more.
                                                                                                      

Fixes: cb4bfda62afa ("nvme-pci: fix hot removal during error handling")




From axboe at kernel.dk  Thu Jun 24 17:47:56 2021
From: axboe at kernel.dk (Jens Axboe)
Date: Thu, 24 Jun 2021 18:47:56 -0600
Subject: [PATCHv4 0/4] block and nvme passthrough error handling
In-Reply-To: <20210622145718.GA11584@redsun51.ssa.fujisawa.hgst.com>
References: <20210610214437.641245-1-kbusch@kernel.org>
 <20210622145718.GA11584@redsun51.ssa.fujisawa.hgst.com>
Message-ID: <f6df0d6d-9315-7a16-545e-9f752f65df00@kernel.dk>

On 6/22/21 8:57 AM, Keith Busch wrote:
> Jens,
> 
> Do you have any thoughts on this series? I think it's good to go and
> have received reviews, and just want to check if you are okay to take
> this through block.

Looks fine to me, I have applied it for 5.14. Thanks.

-- 
Jens Axboe



From lkp at intel.com  Thu Jun 24 21:44:48 2021
From: lkp at intel.com (kernel test robot)
Date: Fri, 25 Jun 2021 12:44:48 +0800
Subject: [linux-nvme:nvme-5.14] BUILD SUCCESS
 ff5af4bfb6af43935a54135cb82df4a5989cfbf3
Message-ID: <60d55f40.UM+C8MWA8vwpvwfB%lkp@intel.com>

tree/branch: git://git.infradead.org/nvme.git nvme-5.14
branch HEAD: ff5af4bfb6af43935a54135cb82df4a5989cfbf3  nvme-tcp: use __dev_get_by_name instead dev_get_by_name for OPT_HOST_IFACE

elapsed time: 1303m

configs tested: 131
configs skipped: 3

The following configs have been built successfully.
More configs may be tested in the coming days.

gcc tested configs:
arm                                 defconfig
arm64                            allyesconfig
arm64                               defconfig
arm                              allyesconfig
arm                              allmodconfig
arm                             rpc_defconfig
arm                        magician_defconfig
mips                      maltasmvp_defconfig
s390                                defconfig
powerpc                      ppc44x_defconfig
alpha                            alldefconfig
m68k                          amiga_defconfig
powerpc                     ep8248e_defconfig
mips                        omega2p_defconfig
arm                         s5pv210_defconfig
sparc64                          alldefconfig
arm                        keystone_defconfig
mips                          ath79_defconfig
mips                  maltasmvp_eva_defconfig
arm                        spear6xx_defconfig
sh                             espt_defconfig
mips                        qi_lb60_defconfig
h8300                            alldefconfig
powerpc                     akebono_defconfig
xtensa                generic_kc705_defconfig
riscv                             allnoconfig
openrisc                    or1ksim_defconfig
mips                       capcella_defconfig
arm                        mini2440_defconfig
arm                            hisi_defconfig
arm                        neponset_defconfig
mips                        bcm63xx_defconfig
sh                        edosk7760_defconfig
powerpc                      katmai_defconfig
powerpc                 mpc834x_mds_defconfig
arc                        nsimosci_defconfig
mips                        workpad_defconfig
sh                     sh7710voipgw_defconfig
powerpc               mpc834x_itxgp_defconfig
powerpc                 mpc8313_rdb_defconfig
arm                   milbeaut_m10v_defconfig
arm                           tegra_defconfig
s390                          debug_defconfig
powerpc                       ebony_defconfig
powerpc                       holly_defconfig
m68k                       m5249evb_defconfig
x86_64                            allnoconfig
ia64                             allmodconfig
ia64                                defconfig
ia64                             allyesconfig
m68k                             allmodconfig
m68k                                defconfig
m68k                             allyesconfig
nios2                               defconfig
arc                              allyesconfig
nds32                             allnoconfig
nds32                               defconfig
nios2                            allyesconfig
csky                                defconfig
alpha                               defconfig
alpha                            allyesconfig
xtensa                           allyesconfig
h8300                            allyesconfig
arc                                 defconfig
sh                               allmodconfig
parisc                              defconfig
s390                             allyesconfig
s390                             allmodconfig
parisc                           allyesconfig
i386                             allyesconfig
sparc                            allyesconfig
sparc                               defconfig
i386                                defconfig
mips                             allyesconfig
mips                             allmodconfig
powerpc                          allyesconfig
powerpc                          allmodconfig
powerpc                           allnoconfig
i386                 randconfig-a001-20210622
i386                 randconfig-a002-20210622
i386                 randconfig-a003-20210622
i386                 randconfig-a006-20210622
i386                 randconfig-a005-20210622
i386                 randconfig-a004-20210622
i386                 randconfig-a002-20210623
i386                 randconfig-a001-20210623
i386                 randconfig-a003-20210623
i386                 randconfig-a006-20210623
i386                 randconfig-a005-20210623
i386                 randconfig-a004-20210623
x86_64               randconfig-a012-20210622
x86_64               randconfig-a016-20210622
x86_64               randconfig-a015-20210622
x86_64               randconfig-a014-20210622
x86_64               randconfig-a013-20210622
x86_64               randconfig-a011-20210622
i386                 randconfig-a011-20210623
i386                 randconfig-a014-20210623
i386                 randconfig-a013-20210623
i386                 randconfig-a015-20210623
i386                 randconfig-a012-20210623
i386                 randconfig-a016-20210623
i386                 randconfig-a011-20210622
i386                 randconfig-a014-20210622
i386                 randconfig-a013-20210622
i386                 randconfig-a015-20210622
i386                 randconfig-a012-20210622
i386                 randconfig-a016-20210622
riscv                    nommu_k210_defconfig
riscv                            allyesconfig
riscv                    nommu_virt_defconfig
riscv                               defconfig
riscv                          rv32_defconfig
riscv                            allmodconfig
um                           x86_64_defconfig
um                             i386_defconfig
um                            kunit_defconfig
x86_64                           allyesconfig
x86_64                    rhel-8.3-kselftests
x86_64                              defconfig
x86_64                               rhel-8.3
x86_64                      rhel-8.3-kbuiltin
x86_64                                  kexec

clang tested configs:
x86_64               randconfig-b001-20210622
x86_64               randconfig-b001-20210624
x86_64               randconfig-a002-20210622
x86_64               randconfig-a001-20210622
x86_64               randconfig-a005-20210622
x86_64               randconfig-a003-20210622
x86_64               randconfig-a004-20210622
x86_64               randconfig-a006-20210622

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all at lists.01.org


From kai.heng.feng at canonical.com  Thu Jun 24 22:29:44 2021
From: kai.heng.feng at canonical.com (Kai-Heng Feng)
Date: Fri, 25 Jun 2021 13:29:44 +0800
Subject: [PATCH] Revert "nvme-pci: use simple suspend when a HMB is enabled"
Message-ID: <20210625052944.4140054-1-kai.heng.feng@canonical.com>

This reverts commit b97120b15ebd3de51325084136d3b9c3cce656d6.

There are at least two issues introduced by that commit.
Config space becomes inaccessible after s2idle:
[  827.776792] nvme 0000:58:00.0: can't change power state from D3cold to D0 (config space inaccessible)
The NVMe in question is recovered correctly but PCIe ASPM setting is lost as result.

The second one is more severe, the NVMe becomes unreliable and cause the
rootfs unreadable after s2idle.

Since reset DMA is not possible on suspend/resume, revert the commit to
solve the regression.

Cc: Koba Ko <koba.ko at canonical.com>
BugLink: https://bugs.launchpad.net/bugs/1912057
BugLink: https://bugs.launchpad.net/bugs/1927866
Fixes: b97120b15ebd ("nvme-pci: use simple suspend when a HMB is enabled")
Signed-off-by: Kai-Heng Feng <kai.heng.feng at canonical.com>
---
 drivers/nvme/host/pci.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index a29b170701fc6..72d2121f82068 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -3079,15 +3079,9 @@ static int nvme_suspend(struct device *dev)
 	 * the PCI bus layer to put it into D3 in order to take the PCIe link
 	 * down, so as to allow the platform to achieve its minimum low-power
 	 * state (which may not be possible if the link is up).
-	 *
-	 * If a host memory buffer is enabled, shut down the device as the NVMe
-	 * specification allows the device to access the host memory buffer in
-	 * host DRAM from all power states, but hosts will fail access to DRAM
-	 * during S3.
 	 */
 	if (pm_suspend_via_firmware() || !ctrl->npss ||
 	    !pcie_aspm_enabled(pdev) ||
-	    ndev->nr_host_mem_descs ||
 	    (ndev->ctrl.quirks & NVME_QUIRK_SIMPLE_SUSPEND))
 		return nvme_disable_prepare_reset(ndev, true);
 
-- 
2.31.1



From dwagner at suse.de  Fri Jun 25 03:16:47 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Fri, 25 Jun 2021 12:16:47 +0200
Subject: [PATCH 0/2] Handle update hardware queues and queue freeze more
 carefully
Message-ID: <20210625101649.49296-1-dwagner@suse.de>

Hi,

this is a followup on the crash I reported in

  https://lore.kernel.org/linux-block/20210608183339.70609-1-dwagner at suse.de/

By moving the hardware check up the crash was gone. Unfortuntatly, I
don't understand why this fixes the crash. The per-cpu access is
crashing but I can't see why the blk_mq_update_nr_hw_queues() is
fixing this problem.

Even though I can't explain why it fixes it, I think it makes sense to
update the hardware queue mapping bevore we recreate the IO
queues. Thus I avoided in the commit message to say it fixes
something.

Also during testing I observed the we hang indivinetly in
blk_mq_freeze_queue_wait(). Again I can't explain why we get stuck
there but given a common pattern for the nvme_wait_freeze() is to use
it with a timeout I think the timeout should be used too :)

Anyway, someone with more undertanding of the stack can explain the
problems.

Thanks,
Daniel


Daniel Wagner (2):
  nvme-fc: Update hardware queues before using them
  nvme-fc: Wait with a timeout for queue to freeze

 drivers/nvme/host/fc.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

-- 
2.29.2



From dwagner at suse.de  Fri Jun 25 03:16:48 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Fri, 25 Jun 2021 12:16:48 +0200
Subject: [PATCH 1/2] nvme-fc: Update hardware queues before using them
In-Reply-To: <20210625101649.49296-1-dwagner@suse.de>
References: <20210625101649.49296-1-dwagner@suse.de>
Message-ID: <20210625101649.49296-2-dwagner@suse.de>

In case the number of hardware queues changes, do the update the
tagset and ctx to hctx first before using the mapping to recreate and
connnect the IO queues.

Signed-off-by: Daniel Wagner <dwagner at suse.de>
---
 drivers/nvme/host/fc.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 8a3c4814d21b..a9645cd89eca 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2951,14 +2951,6 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 	if (ctrl->ctrl.queue_count == 1)
 		return 0;
 
-	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
-	if (ret)
-		goto out_free_io_queues;
-
-	ret = nvme_fc_connect_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
-	if (ret)
-		goto out_delete_hw_queues;
-
 	if (prior_ioq_cnt != nr_io_queues) {
 		dev_info(ctrl->ctrl.device,
 			"reconnect: revising io queue count from %d to %d\n",
@@ -2968,6 +2960,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 		nvme_unfreeze(&ctrl->ctrl);
 	}
 
+	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
+	if (ret)
+		goto out_free_io_queues;
+
+	ret = nvme_fc_connect_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
+	if (ret)
+		goto out_delete_hw_queues;
+
 	return 0;
 
 out_delete_hw_queues:
-- 
2.29.2



From dwagner at suse.de  Fri Jun 25 03:16:49 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Fri, 25 Jun 2021 12:16:49 +0200
Subject: [PATCH 2/2] nvme-fc: Wait with a timeout for queue to freeze
In-Reply-To: <20210625101649.49296-1-dwagner@suse.de>
References: <20210625101649.49296-1-dwagner@suse.de>
Message-ID: <20210625101649.49296-3-dwagner@suse.de>

Do not wait indifinitly for all queues to freeze. Instead use a
timeout and abort the operation if we get stuck.

Signed-off-by: Daniel Wagner <dwagner at suse.de>
---
 drivers/nvme/host/fc.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index a9645cd89eca..d8db85aa5417 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2955,7 +2955,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
 		dev_info(ctrl->ctrl.device,
 			"reconnect: revising io queue count from %d to %d\n",
 			prior_ioq_cnt, nr_io_queues);
-		nvme_wait_freeze(&ctrl->ctrl);
+		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
+			/*
+			 * If we timed out waiting for freeze we are likely to
+			 * be stuck.  Fail the controller initialization just
+			 * to be safe.
+			 */
+			return -ENODEV;
+		}
 		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
 		nvme_unfreeze(&ctrl->ctrl);
 	}
-- 
2.29.2



From dwagner at suse.de  Fri Jun 25 05:21:56 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Fri, 25 Jun 2021 14:21:56 +0200
Subject: [PATCH 0/2] Handle update hardware queues and queue freeze more
 carefully
In-Reply-To: <20210625101649.49296-1-dwagner@suse.de>
References: <20210625101649.49296-1-dwagner@suse.de>
Message-ID: <20210625122156.x5yzoobuaaec5hss@beryllium.lan>

On Fri, Jun 25, 2021 at 12:16:47PM +0200, Daniel Wagner wrote:
> this is a followup on the crash I reported in
> 
>   https://lore.kernel.org/linux-block/20210608183339.70609-1-dwagner at suse.de/
> 
> By moving the hardware check up the crash was gone. Unfortuntatly, I
> don't understand why this fixes the crash. The per-cpu access is
> crashing but I can't see why the blk_mq_update_nr_hw_queues() is
> fixing this problem.
> 
> Even though I can't explain why it fixes it, I think it makes sense to
> update the hardware queue mapping bevore we recreate the IO
> queues. Thus I avoided in the commit message to say it fixes
> something.

I just discussed this with Hannes and we figured out how the crash is
fixed by moving the blk_mq_update_nr_hw_queues() before the
nvme_fc_create_hw_io_queues()/nvme_fc_connect_io_queues().

First of all, blk_mq_update_nr_hw_queues() operates on the normal
tag_set and not the admin_tag_set. That means when we move the
blk_mq_update_nr_hw_queues() before the nvme_fc_connect_io_queues(), we
update the mapping to only CPUs and hwctx which are available. When we
then do the connect call nvmf_connect_io_queue() we will only allocate
tags from queues which are not in the BLK_MQ_S_INACTIVE anymore. Hence
we skip the blk_mq_put_tag() call.

> Also during testing I observed the we hang indivinetly in
> blk_mq_freeze_queue_wait(). Again I can't explain why we get stuck
> there but given a common pattern for the nvme_wait_freeze() is to use
> it with a timeout I think the timeout should be used too :)

The nvme_wait_freeeze() is probably not needed at all,
__blk_mq_update_nr_hw_queues() already calls blk_mq_freeze_queue(). So
there this is not needed at all. Furthermore, if we move
blk_mq_update_nr_hw_queues() in front of nvme_fc_create_hw_io_queues()
there can't be any pending I/Os because there are not queues.


From ming.lei at redhat.com  Fri Jun 25 06:00:25 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Fri, 25 Jun 2021 21:00:25 +0800
Subject: [PATCH 0/2] Handle update hardware queues and queue freeze more
 carefully
In-Reply-To: <20210625122156.x5yzoobuaaec5hss@beryllium.lan>
References: <20210625101649.49296-1-dwagner@suse.de>
 <20210625122156.x5yzoobuaaec5hss@beryllium.lan>
Message-ID: <YNXTaUMAFCA84jfZ@T590>

On Fri, Jun 25, 2021 at 02:21:56PM +0200, Daniel Wagner wrote:
> On Fri, Jun 25, 2021 at 12:16:47PM +0200, Daniel Wagner wrote:
> > this is a followup on the crash I reported in
> > 
> >   https://lore.kernel.org/linux-block/20210608183339.70609-1-dwagner at suse.de/
> > 
> > By moving the hardware check up the crash was gone. Unfortuntatly, I
> > don't understand why this fixes the crash. The per-cpu access is
> > crashing but I can't see why the blk_mq_update_nr_hw_queues() is
> > fixing this problem.
> > 
> > Even though I can't explain why it fixes it, I think it makes sense to
> > update the hardware queue mapping bevore we recreate the IO
> > queues. Thus I avoided in the commit message to say it fixes
> > something.
> 
> I just discussed this with Hannes and we figured out how the crash is
> fixed by moving the blk_mq_update_nr_hw_queues() before the
> nvme_fc_create_hw_io_queues()/nvme_fc_connect_io_queues().
> 
> First of all, blk_mq_update_nr_hw_queues() operates on the normal
> tag_set and not the admin_tag_set. That means when we move the
> blk_mq_update_nr_hw_queues() before the nvme_fc_connect_io_queues(), we
> update the mapping to only CPUs and hwctx which are available. When we
> then do the connect call nvmf_connect_io_queue() we will only allocate
> tags from queues which are not in the BLK_MQ_S_INACTIVE anymore. Hence
> we skip the blk_mq_put_tag() call.

Your patch just reduces the race window, what if all cpus in
hctx->cpumask become offline when calling blk_mq_alloc_request_hctx()?


Thanks,
Ming



From lengchao at huawei.com  Fri Jun 25 18:03:36 2021
From: lengchao at huawei.com (Chao Leng)
Date: Sat, 26 Jun 2021 09:03:36 +0800
Subject: [PATCH 0/2] nvme-fabrics: short-circuit connect retries
In-Reply-To: <91ff9961-6537-05d1-c552-18a02d13bf0c@suse.de>
References: <20210623143250.82445-1-hare@suse.de>
 <8d7c79d9-0928-8c71-74fd-7185d67e7895@grimberg.me>
 <204a6656-ef83-b688-4979-26fe17f57974@suse.de>
 <cdf40226-12ed-5f87-8549-10b4defceae5@huawei.com>
 <91ff9961-6537-05d1-c552-18a02d13bf0c@suse.de>
Message-ID: <984b2d1a-f319-1a09-7347-6c4edd1dbab6@huawei.com>



On 2021/6/24 16:10, Hannes Reinecke wrote:
> On 6/24/21 9:29 AM, Chao Leng wrote:
>>
>>
>> On 2021/6/24 13:51, Hannes Reinecke wrote:
>>> On 6/23/21 11:38 PM, Sagi Grimberg wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> commit f25f8ef70ce2 ("nvme-fc: short-circuit reconnect retries")
>>>>> allowed the fc transport to honour the DNR bit during reconnect
>>>>> retries, allowing to speed up error recovery.
>>>>
>>>> How does this speed up error recovery?
>>>
>>> Well, not exactly error recovery (as there is nothing to recover).
>>> But we won't attempt pointless retries, thereby reducing the noise in
>>> the message log.
>> This conflict with the tcp and rdma target.
>> You may need to delete the improper NVME_SC_DNR at the target.
>> However, this will cause compatibility issues between different versions.
> 
> Which ones?
In many scenarios, the destination sets DNR for abnormal packets,
but each new connection may not have the same error.
> I checked the DNR usage in the target code, and they seem to set it
> correctly (ie the result would not change when the command is retried).
> With the possible exception of ENOSPC handling, as this is arguably
> dynamic and might change with a retry.
The DNR status of the old connection may not be relevant to the re-established connection.
> 
> Cheers,
> 
> Hannes
> 


From hare at suse.de  Sat Jun 26 05:09:48 2021
From: hare at suse.de (Hannes Reinecke)
Date: Sat, 26 Jun 2021 14:09:48 +0200
Subject: [PATCH 0/2] nvme-fabrics: short-circuit connect retries
In-Reply-To: <984b2d1a-f319-1a09-7347-6c4edd1dbab6@huawei.com>
References: <20210623143250.82445-1-hare@suse.de>
 <8d7c79d9-0928-8c71-74fd-7185d67e7895@grimberg.me>
 <204a6656-ef83-b688-4979-26fe17f57974@suse.de>
 <cdf40226-12ed-5f87-8549-10b4defceae5@huawei.com>
 <91ff9961-6537-05d1-c552-18a02d13bf0c@suse.de>
 <984b2d1a-f319-1a09-7347-6c4edd1dbab6@huawei.com>
Message-ID: <d20b23fc-dd08-cb51-eb29-aa033810b0a4@suse.de>

On 6/26/21 3:03 AM, Chao Leng wrote:
> 
> 
> On 2021/6/24 16:10, Hannes Reinecke wrote:
>> On 6/24/21 9:29 AM, Chao Leng wrote:
>>>
>>>
>>> On 2021/6/24 13:51, Hannes Reinecke wrote:
>>>> On 6/23/21 11:38 PM, Sagi Grimberg wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> commit f25f8ef70ce2 ("nvme-fc: short-circuit reconnect retries")
>>>>>> allowed the fc transport to honour the DNR bit during reconnect
>>>>>> retries, allowing to speed up error recovery.
>>>>>
>>>>> How does this speed up error recovery?
>>>>
>>>> Well, not exactly error recovery (as there is nothing to recover).
>>>> But we won't attempt pointless retries, thereby reducing the noise in
>>>> the message log.
>>> This conflict with the tcp and rdma target.
>>> You may need to delete the improper NVME_SC_DNR at the target.
>>> However, this will cause compatibility issues between different 
>>> versions.
>>
>> Which ones?
> In many scenarios, the destination sets DNR for abnormal packets,
> but each new connection may not have the same error.

This patch series is only for the DNR bit set in response to the 
'connect' command.
If the target is not able to process the 'connect' command, but may be 
so in the future it really should not set the DNR bit.

>> I checked the DNR usage in the target code, and they seem to set it
>> correctly (ie the result would not change when the command is retried).
>> With the possible exception of ENOSPC handling, as this is arguably
>> dynamic and might change with a retry.
> The DNR status of the old connection may not be relevant to the 
> re-established connection.

See above.
We are just checking the DNR settings for the 'connect' command (or any 
other commands being sent during initial controller configuration).
If that fails the connect never was properly initialized; if the 
controller would return a different status after reconnect it simply 
should not set the DNR bit ...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From jsmart2021 at gmail.com  Sun Jun 27 06:39:14 2021
From: jsmart2021 at gmail.com (James Smart)
Date: Sun, 27 Jun 2021 06:39:14 -0700
Subject: [PATCH 0/2] nvme-fabrics: short-circuit connect retries
In-Reply-To: <d20b23fc-dd08-cb51-eb29-aa033810b0a4@suse.de>
References: <20210623143250.82445-1-hare@suse.de>
 <8d7c79d9-0928-8c71-74fd-7185d67e7895@grimberg.me>
 <204a6656-ef83-b688-4979-26fe17f57974@suse.de>
 <cdf40226-12ed-5f87-8549-10b4defceae5@huawei.com>
 <91ff9961-6537-05d1-c552-18a02d13bf0c@suse.de>
 <984b2d1a-f319-1a09-7347-6c4edd1dbab6@huawei.com>
 <d20b23fc-dd08-cb51-eb29-aa033810b0a4@suse.de>
Message-ID: <caa3759f-9a72-5e3b-9097-700132e4ab63@gmail.com>

On 6/26/2021 5:09 AM, Hannes Reinecke wrote:
> On 6/26/21 3:03 AM, Chao Leng wrote:
>>
>>
>> On 2021/6/24 16:10, Hannes Reinecke wrote:
>>> On 6/24/21 9:29 AM, Chao Leng wrote:
>>>>
>>>>
>>>> On 2021/6/24 13:51, Hannes Reinecke wrote:
>>>>> On 6/23/21 11:38 PM, Sagi Grimberg wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> commit f25f8ef70ce2 ("nvme-fc: short-circuit reconnect retries")
>>>>>>> allowed the fc transport to honour the DNR bit during reconnect
>>>>>>> retries, allowing to speed up error recovery.
>>>>>>
>>>>>> How does this speed up error recovery?
>>>>>
>>>>> Well, not exactly error recovery (as there is nothing to recover).
>>>>> But we won't attempt pointless retries, thereby reducing the noise in
>>>>> the message log.
>>>> This conflict with the tcp and rdma target.
>>>> You may need to delete the improper NVME_SC_DNR at the target.
>>>> However, this will cause compatibility issues between different 
>>>> versions.
>>>
>>> Which ones?
>> In many scenarios, the destination sets DNR for abnormal packets,
>> but each new connection may not have the same error.
> 
> This patch series is only for the DNR bit set in response to the 
> 'connect' command.
> If the target is not able to process the 'connect' command, but may be 
> so in the future it really should not set the DNR bit.
> 
>>> I checked the DNR usage in the target code, and they seem to set it
>>> correctly (ie the result would not change when the command is retried).
>>> With the possible exception of ENOSPC handling, as this is arguably
>>> dynamic and might change with a retry.
>> The DNR status of the old connection may not be relevant to the 
>> re-established connection.
> 
> See above.
> We are just checking the DNR settings for the 'connect' command (or any 
> other commands being sent during initial controller configuration).
> If that fails the connect never was properly initialized; if the 
> controller would return a different status after reconnect it simply 
> should not set the DNR bit ...
> 
> Cheers,
> 
> Hannes

Agreed. Since 1.3 spec says: "If set to ?1?, indicates that if the same 
command is re-submitted to any controller in the NVM subsystem, then 
that re-submitted command is expected to fail."

Thus if the initial connect fails in this manner, any new association 
will be on a different controller, where it is now expected connect on 
that controller will fail too.  Thus - why continue to connect when it's 
expected each will fail.

-- james


From jsmart2021 at gmail.com  Sun Jun 27 06:47:23 2021
From: jsmart2021 at gmail.com (James Smart)
Date: Sun, 27 Jun 2021 06:47:23 -0700
Subject: [PATCH 1/2] nvme-fc: Update hardware queues before using them
In-Reply-To: <20210625101649.49296-2-dwagner@suse.de>
References: <20210625101649.49296-1-dwagner@suse.de>
 <20210625101649.49296-2-dwagner@suse.de>
Message-ID: <f5c53785-077e-0e2a-ae3d-178b3e73b80b@gmail.com>

On 6/25/2021 3:16 AM, Daniel Wagner wrote:
> In case the number of hardware queues changes, do the update the
> tagset and ctx to hctx first before using the mapping to recreate and
> connnect the IO queues.
> 
> Signed-off-by: Daniel Wagner <dwagner at suse.de>
> ---
>   drivers/nvme/host/fc.c | 16 ++++++++--------
>   1 file changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index 8a3c4814d21b..a9645cd89eca 100644

Makes sense.  Thanks.   Although it does bring up that perhaps, if the 
hw queue count changes, thus it no longer matches what was set on the 
target, the new value should be set on the target to release resources 
on the target.

Note: the same behavior exists in the other transports as we all started 
from the same lineage. So those should be updated as well.  Granted 
you'll need to break out the queue count set and checking which was done 
on fc but not on the other transports.

Reviewed-by: James Smart <jsmart2021 at gmail.com>

-- james



From jsmart2021 at gmail.com  Sun Jun 27 07:04:53 2021
From: jsmart2021 at gmail.com (James Smart)
Date: Sun, 27 Jun 2021 07:04:53 -0700
Subject: [PATCH 2/2] nvme-fc: Wait with a timeout for queue to freeze
In-Reply-To: <20210625101649.49296-3-dwagner@suse.de>
References: <20210625101649.49296-1-dwagner@suse.de>
 <20210625101649.49296-3-dwagner@suse.de>
Message-ID: <9362fb6c-8a98-3189-453a-69d4c0b622e3@gmail.com>

On 6/25/2021 3:16 AM, Daniel Wagner wrote:
> Do not wait indifinitly for all queues to freeze. Instead use a
> timeout and abort the operation if we get stuck.
> 
> Signed-off-by: Daniel Wagner <dwagner at suse.de>
> ---
>   drivers/nvme/host/fc.c | 9 ++++++++-
>   1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index a9645cd89eca..d8db85aa5417 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -2955,7 +2955,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
>   		dev_info(ctrl->ctrl.device,
>   			"reconnect: revising io queue count from %d to %d\n",
>   			prior_ioq_cnt, nr_io_queues);
> -		nvme_wait_freeze(&ctrl->ctrl);
> +		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
> +			/*
> +			 * If we timed out waiting for freeze we are likely to
> +			 * be stuck.  Fail the controller initialization just
> +			 * to be safe.
> +			 */
> +			return -ENODEV;
> +		}
>   		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
>   		nvme_unfreeze(&ctrl->ctrl);
>   	}
> 

Looks fine. This is one of those that changed in the other transports 
but fc wasn't part of the patch set.

Reviewed-by: James Smart <jsmart2021 at gmail.com>

-- james



From lengchao at huawei.com  Sun Jun 27 18:10:35 2021
From: lengchao at huawei.com (Chao Leng)
Date: Mon, 28 Jun 2021 09:10:35 +0800
Subject: [PATCH 0/2] nvme-fabrics: short-circuit connect retries
In-Reply-To: <caa3759f-9a72-5e3b-9097-700132e4ab63@gmail.com>
References: <20210623143250.82445-1-hare@suse.de>
 <8d7c79d9-0928-8c71-74fd-7185d67e7895@grimberg.me>
 <204a6656-ef83-b688-4979-26fe17f57974@suse.de>
 <cdf40226-12ed-5f87-8549-10b4defceae5@huawei.com>
 <91ff9961-6537-05d1-c552-18a02d13bf0c@suse.de>
 <984b2d1a-f319-1a09-7347-6c4edd1dbab6@huawei.com>
 <d20b23fc-dd08-cb51-eb29-aa033810b0a4@suse.de>
 <caa3759f-9a72-5e3b-9097-700132e4ab63@gmail.com>
Message-ID: <28ec79af-beb0-8ef9-ab6a-d608a632ee2c@huawei.com>



On 2021/6/27 21:39, James Smart wrote:
> On 6/26/2021 5:09 AM, Hannes Reinecke wrote:
>> On 6/26/21 3:03 AM, Chao Leng wrote:
>>>
>>>
>>> On 2021/6/24 16:10, Hannes Reinecke wrote:
>>>> On 6/24/21 9:29 AM, Chao Leng wrote:
>>>>>
>>>>>
>>>>> On 2021/6/24 13:51, Hannes Reinecke wrote:
>>>>>> On 6/23/21 11:38 PM, Sagi Grimberg wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> commit f25f8ef70ce2 ("nvme-fc: short-circuit reconnect retries")
>>>>>>>> allowed the fc transport to honour the DNR bit during reconnect
>>>>>>>> retries, allowing to speed up error recovery.
>>>>>>>
>>>>>>> How does this speed up error recovery?
>>>>>>
>>>>>> Well, not exactly error recovery (as there is nothing to recover).
>>>>>> But we won't attempt pointless retries, thereby reducing the noise in
>>>>>> the message log.
>>>>> This conflict with the tcp and rdma target.
>>>>> You may need to delete the improper NVME_SC_DNR at the target.
>>>>> However, this will cause compatibility issues between different versions.
>>>>
>>>> Which ones?
>>> In many scenarios, the destination sets DNR for abnormal packets,
>>> but each new connection may not have the same error.
>>
>> This patch series is only for the DNR bit set in response to the 'connect' command.
>> If the target is not able to process the 'connect' command, but may be so in the future it really should not set the DNR bit.
>>
>>>> I checked the DNR usage in the target code, and they seem to set it
>>>> correctly (ie the result would not change when the command is retried).
>>>> With the possible exception of ENOSPC handling, as this is arguably
>>>> dynamic and might change with a retry.
>>> The DNR status of the old connection may not be relevant to the re-established connection.
>>
>> See above.
>> We are just checking the DNR settings for the 'connect' command (or any other commands being sent during initial controller configuration).
>> If that fails the connect never was properly initialized; if the controller would return a different status after reconnect it simply should not set the DNR bit ...
>>
>> Cheers,
>>
>> Hannes
> 
> Agreed. Since 1.3 spec says: "If set to ?1?, indicates that if the same command is re-submitted to any controller in the NVM subsystem, then that re-submitted command is expected to fail."
According to the definition of the protocol, this is not strictly implemented on the target side.
In nvme/target, there are many external errors, the DNR is set to 1.
For example, abormal fabrics cmd.
> 
> Thus if the initial connect fails in this manner, any new association will be on a different controller, where it is now expected connect on that controller will fail too.? Thus - why continue to connect when it's expected each will fail.
Agree.
We should not attempt to re-establish the connection if target can not work due to target inner error .
However, now  target does not behave exactly like this, so there are conflicts and compatibility issues.
> 
> -- james
> .


From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:03 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:03 -0700
Subject: [PATCH 00/14] nvmet: remove repeated calls to nvmet_req_complete()
Message-ID: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>

Hi,

Since the addition of the NVMeOF target time, we've added various log
page and identify command handlers where we complete the request in
those handlers. This adds 10 repeated nvmet_req_complete() calls.

This patch series removes 10 repeated calls to the nvmet_req_complete()
in for various log page and identify command handlers by completing the
request in the nvmet_execute_get_log_page() and nvmet_execute_identify().
Also, with this change we prevent future repeatation of such calls.

This also allows us to remove 8 goto labels.

I've tested this code on nvme-5.14 [1], blktests and zonefs tests seems
to pass.

-ck

Chaitanya Kulkarni (14):
  nvmet: add common req complete for log page
  nvmet: remove repeated call for error log page
  nvmet: remove repeated call for smart log page
  nvmet: remove repeated call for fw slot log page
  nvmet: remove repeated call for changed-ns log page
  nvmet: remove repeated call for effects log page
  nvmet: remove repeated call for ana log page
  nvmet: add default case for nvmet_execute_identify
  nvmet: remove repeated call for id-ctrl
  nvmet: remove repeated call for id-ns
  nvmet: remove repeated call for nslist
  nvmet: remove repeated call for desclist
  nvmet: remove repeated call for id-cns-ns
  nvmet: remove repeated call for id-cns-ctrl

 drivers/nvme/target/admin-cmd.c | 180 +++++++++++++++++---------------
 drivers/nvme/target/nvmet.h     |  12 +--
 drivers/nvme/target/zns.c       |  27 ++---
 3 files changed, 108 insertions(+), 111 deletions(-)

# ./check nvme 
nvme/002 (create many subsystems and test discovery)         [passed]
    runtime  38.825s  ...  40.506s
nvme/003 (test if we're sending keep-alives to a discovery controller) [passed]
    runtime  10.176s  ...  10.184s
nvme/004 (test nvme and nvmet UUID NS descriptors)           [passed]
    runtime  1.781s  ...  1.776s
nvme/005 (reset local loopback target)                       [passed]
    runtime  2.334s  ...  2.243s
nvme/006 (create an NVMeOF target with a block device-backed ns) [passed]
    runtime  0.155s  ...  0.149s
nvme/007 (create an NVMeOF target with a file-backed ns)     [passed]
    runtime  0.102s  ...  0.104s
nvme/008 (create an NVMeOF host with a block device-backed ns) [passed]
    runtime  1.779s  ...  1.785s
nvme/009 (create an NVMeOF host with a file-backed ns)       [passed]
    runtime  1.747s  ...  1.755s
nvme/010 (run data verification fio job on NVMeOF block device-backed ns) [passed]
    runtime  34.091s  ...  26.953s
nvme/011 (run data verification fio job on NVMeOF file-backed ns) [passed]
    runtime  288.054s  ...  320.502s
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]
    runtime  8.515s  ...  9.634s
nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed ns) [passed]
    runtime  27.329s  ...  24.710s
nvme/014 (flush a NVMeOF block device-backed ns)             [passed]
    runtime  20.028s  ...  21.024s
nvme/015 (unit test for NVMe flush for file backed ns)       [passed]
    runtime  20.615s  ...  21.140s
nvme/016 (create/delete many NVMeOF block device-backed ns and test discovery) [passed]
    runtime  20.653s  ...  21.571s
nvme/017 (create/delete many file-ns and test discovery)     [passed]
    runtime  20.402s  ...  22.023s
nvme/018 (unit test NVMe-oF out of range access on a file backend) [passed]
    runtime  1.747s  ...  1.767s
nvme/019 (test NVMe DSM Discard command on NVMeOF block-device ns) [passed]
    runtime  1.771s  ...  1.805s
nvme/020 (test NVMe DSM Discard command on NVMeOF file-backed ns) [passed]
    runtime  1.735s  ...  1.739s
nvme/021 (test NVMe list command on NVMeOF file-backed ns)   [passed]
    runtime  1.764s  ...  1.752s
nvme/022 (test NVMe reset command on NVMeOF file-backed ns)  [passed]
    runtime  2.163s  ...  2.182s
nvme/023 (test NVMe smart-log command on NVMeOF block-device ns) [passed]
    runtime  1.780s  ...  1.815s
nvme/024 (test NVMe smart-log command on NVMeOF file-backed ns) [passed]
    runtime  1.739s  ...  1.750s
nvme/025 (test NVMe effects-log command on NVMeOF file-backed ns) [passed]
    runtime  1.726s  ...  1.756s
nvme/026 (test NVMe ns-descs command on NVMeOF file-backed ns) [passed]
    runtime  1.740s  ...  1.750s
nvme/027 (test NVMe ns-rescan command on NVMeOF file-backed ns) [passed]
    runtime  1.735s  ...  1.757s
nvme/028 (test NVMe list-subsys command on NVMeOF file-backed ns) [passed]
    runtime  1.738s  ...  1.751s
nvme/029 (test userspace IO via nvme-cli read/write interface) [passed]
    runtime  2.213s  ...  2.253s
nvme/030 (ensure the discovery generation counter is updated appropriately) [passed]
    runtime  0.404s  ...  0.426s
nvme/031 (test deletion of NVMeOF controllers immediately after setup) [passed]
    runtime  6.396s  ...  6.226s
nvme/038 (test deletion of NVMeOF subsystem without enabling) [passed]
    runtime  0.061s  ...  0.058s

# ./zonefs-tests.sh /dev/nvme1n1 
Gathering information on /dev/nvme1n1...
zonefs-tests on /dev/nvme1n1:
  32 zones (0 conventional zones, 32 sequential zones)
  131072 512B sectors zone size (64 MiB)
  0 max open zones
Running tests
  Test 0010:  mkzonefs (options)                                   ... PASS
  Test 0011:  mkzonefs (force format)                              ... PASS
  Test 0012:  mkzonefs (invalid device)                            ... PASS
  Test 0013:  mkzonefs (super block zone state)                    ... PASS
  Test 0020:  mount (default)                                      ... PASS
  Test 0021:  mount (invalid device)                               ... PASS
  Test 0022:  mount (check mount directory sub-directories)        ... PASS
  Test 0023:  mount (options)                                      ... PASS
  Test 0030:  Number of files (default)                            ... PASS
  Test 0031:  Number of files (aggr_cnv)                           ... skip
  Test 0032:  Number of files using stat (default)                 ... PASS
  Test 0033:  Number of files using stat (aggr_cnv)                ... PASS
  Test 0034:  Number of blocks using stat (default)                ... PASS
  Test 0035:  Number of blocks using stat (aggr_cnv)               ... PASS
  Test 0040:  Files permissions (default)                          ... PASS
  Test 0041:  Files permissions (aggr_cnv)                         ... skip
  Test 0042:  Files permissions (set value)                        ... PASS
  Test 0043:  Files permissions (set value + aggr_cnv)             ... skip
  Test 0050:  Files owner (default)                                ... PASS
  Test 0051:  Files owner (aggr_cnv)                               ... skip
  Test 0052:  Files owner (set value)                              ... PASS
  Test 0053:  Files owner (set value + aggr_cnv)                   ... skip
  Test 0060:  Files size (default)                                 ... PASS
  Test 0061:  Files size (aggr_cnv)                                ... skip
  Test 0070:  Conventional file truncate                           ... skip
  Test 0071:  Conventional file truncate (aggr_cnv)                ... skip
  Test 0072:  Conventional file unlink                             ... skip
  Test 0073:  Conventional file unlink (aggr_cnv)                  ... skip
  Test 0074:  Conventional file random write                       ... skip
  Test 0075:  Conventional file random write (direct)              ... skip
  Test 0076:  Conventional file random write (aggr_cnv)            ... skip
  Test 0077:  Conventional file random write (aggr_cnv, direct)    ... skip
  Test 0078:  Conventional file mmap read/write                    ... skip
  Test 0079:  Conventional file mmap read/write (aggr_cnv)         ... skip
  Test 0080:  Sequential file truncate                             ... PASS
  Test 0081:  Sequential file unlink                               ... PASS
  Test 0082:  Sequential file buffered write IO                    ... PASS
  Test 0083:  Sequential file overwrite                            ... PASS
  Test 0084:  Sequential file unaligned write (sync IO)            ... PASS
  Test 0085:  Sequential file unaligned write (async IO)           ... PASS
  Test 0086:  Sequential file append (sync)                        ... PASS
  Test 0087:  Sequential file append (async)                       ... PASS
  Test 0088:  Sequential file random read                          ... PASS
  Test 0089:  Sequential file mmap read/write                      ... PASS
  Test 0090:  sequential file 4K synchronous write                 ... PASS
  Test 0091:  Sequential file large synchronous write              ... PASS

-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:04 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:04 -0700
Subject: [PATCH 01/14] nvmet: add common req complete for log page
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-2-chaitanya.kulkarni@wdc.com>

For set/get features command handlers we don't complete the request in
each feature handler instead we complete the request in the original
callback i.e. in nvmet_execute_set_feature() and
nvmet_execute_get_feature(). This reduces the number of repeated
nvmet_req_complete() calls needed for each feature to complete the
request.

A similar pattern can be used for the log page handler callback and its
helpers.

In this prep patch we add a default caes for unhandled log pages and
complete the request in the nvmet_execute_get_log_page(). Next patches
will use this call for request completion so we can remove the local
call from each log page handler.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 0cb98f2bbc8c..03bfd8676456 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -314,6 +314,8 @@ static void nvmet_execute_get_log_page_ana(struct nvmet_req *req)
 
 static void nvmet_execute_get_log_page(struct nvmet_req *req)
 {
+	u16 status;
+
 	if (!nvmet_check_transfer_len(req, nvmet_get_log_page_len(req->cmd)))
 		return;
 
@@ -335,11 +337,13 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 		return nvmet_execute_get_log_cmd_effects_ns(req);
 	case NVME_LOG_ANA:
 		return nvmet_execute_get_log_page_ana(req);
+	default:
+		pr_err("unhandled lid %d on qid %d\n",
+		       req->cmd->get_log_page.lid, req->sq->qid);
+		req->error_loc = offsetof(struct nvme_get_log_page_command, lid);
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
 	}
-	pr_debug("unhandled lid %d on qid %d\n",
-	       req->cmd->get_log_page.lid, req->sq->qid);
-	req->error_loc = offsetof(struct nvme_get_log_page_command, lid);
-	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
+	nvmet_req_complete(req, status);
 }
 
 static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:05 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:05 -0700
Subject: [PATCH 02/14] nvmet: remove repeated call for error log page
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-3-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_get_log_page_error() and complete the
request in nvmet_execute_get_log_page(). This reduces number of
repeated nvmet_req_complete() calls for log page handlers.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 03bfd8676456..66bb5b649ffd 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -45,7 +45,7 @@ static void nvmet_execute_get_log_page_noop(struct nvmet_req *req)
 	nvmet_req_complete(req, nvmet_zero_sgl(req, 0, req->transfer_len));
 }
 
-static void nvmet_execute_get_log_page_error(struct nvmet_req *req)
+static u16 nvmet_execute_get_log_page_error(struct nvmet_req *req)
 {
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	unsigned long flags;
@@ -68,7 +68,7 @@ static void nvmet_execute_get_log_page_error(struct nvmet_req *req)
 		offset += sizeof(struct nvme_error_slot);
 	}
 	spin_unlock_irqrestore(&ctrl->error_lock, flags);
-	nvmet_req_complete(req, 0);
+	return NVME_SC_SUCCESS;
 }
 
 static u16 nvmet_get_smart_log_nsid(struct nvmet_req *req,
@@ -321,7 +321,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 
 	switch (req->cmd->get_log_page.lid) {
 	case NVME_LOG_ERROR:
-		return nvmet_execute_get_log_page_error(req);
+		status = nvmet_execute_get_log_page_error(req);
+		break;
 	case NVME_LOG_SMART:
 		return nvmet_execute_get_log_page_smart(req);
 	case NVME_LOG_FW_SLOT:
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:06 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:06 -0700
Subject: [PATCH 03/14] nvmet: remove repeated call for smart log page
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-4-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_get_log_page_smart() and complete the
request in nvmet_execute_get_log_page(). This reduces number of
repeated nvmet_req_complete() calls for log page handlers.

Also, this allows us to remove the goto out label.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 66bb5b649ffd..079fe04d0ee6 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -130,18 +130,18 @@ static u16 nvmet_get_smart_log_all(struct nvmet_req *req,
 	return NVME_SC_SUCCESS;
 }
 
-static void nvmet_execute_get_log_page_smart(struct nvmet_req *req)
+static u16 nvmet_execute_get_log_page_smart(struct nvmet_req *req)
 {
 	struct nvme_smart_log *log;
 	u16 status = NVME_SC_INTERNAL;
 	unsigned long flags;
 
 	if (req->transfer_len != sizeof(*log))
-		goto out;
+		return status;
 
 	log = kzalloc(sizeof(*log), GFP_KERNEL);
 	if (!log)
-		goto out;
+		return status;
 
 	if (req->cmd->get_log_page.nsid == cpu_to_le32(NVME_NSID_ALL))
 		status = nvmet_get_smart_log_all(req, log);
@@ -158,8 +158,8 @@ static void nvmet_execute_get_log_page_smart(struct nvmet_req *req)
 	status = nvmet_copy_to_sgl(req, 0, log, sizeof(*log));
 out_free_log:
 	kfree(log);
-out:
-	nvmet_req_complete(req, status);
+
+	return status;
 }
 
 static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log)
@@ -324,7 +324,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 		status = nvmet_execute_get_log_page_error(req);
 		break;
 	case NVME_LOG_SMART:
-		return nvmet_execute_get_log_page_smart(req);
+		status = nvmet_execute_get_log_page_smart(req);
+		break;
 	case NVME_LOG_FW_SLOT:
 		/*
 		 * We only support a single firmware slot which always is
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:07 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:07 -0700
Subject: [PATCH 04/14] nvmet: remove repeated call for fw slot log page
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-5-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_get_log_page_noop() and complete the
request in nvmet_execute_get_log_page(). This reduces number of
repeated nvmet_req_complete() calls for log page handlers.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 079fe04d0ee6..8c798559f1cc 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -40,9 +40,9 @@ u64 nvmet_get_log_page_offset(struct nvme_command *cmd)
 	return le64_to_cpu(cmd->get_log_page.lpo);
 }
 
-static void nvmet_execute_get_log_page_noop(struct nvmet_req *req)
+static u16 nvmet_execute_get_log_page_noop(struct nvmet_req *req)
 {
-	nvmet_req_complete(req, nvmet_zero_sgl(req, 0, req->transfer_len));
+	return nvmet_zero_sgl(req, 0, req->transfer_len);
 }
 
 static u16 nvmet_execute_get_log_page_error(struct nvmet_req *req)
@@ -332,7 +332,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 		 * active, so we can zero out the whole firmware slot log and
 		 * still claim to fully implement this mandatory log page.
 		 */
-		return nvmet_execute_get_log_page_noop(req);
+		status = nvmet_execute_get_log_page_noop(req);
+		break;
 	case NVME_LOG_CHANGED_NS:
 		return nvmet_execute_get_log_changed_ns(req);
 	case NVME_LOG_CMD_EFFECTS:
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:08 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:08 -0700
Subject: [PATCH 05/14] nvmet: remove repeated call for changed-ns log page
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-6-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_get_log_page_changed_ns() and complete
the request in nvmet_execute_get_log_page(). This reduces number of
repeated nvmet_req_complete() calls for log page handlers.

Also, now we can get rid of the goto and out label needed for request
completion and directly return.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 8c798559f1cc..2b018b84bf3f 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -221,14 +221,14 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
 	nvmet_req_complete(req, status);
 }
 
-static void nvmet_execute_get_log_changed_ns(struct nvmet_req *req)
+static u16 nvmet_execute_get_log_changed_ns(struct nvmet_req *req)
 {
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	u16 status = NVME_SC_INTERNAL;
 	size_t len;
 
 	if (req->transfer_len != NVME_MAX_CHANGED_NAMESPACES * sizeof(__le32))
-		goto out;
+		return status;
 
 	mutex_lock(&ctrl->lock);
 	if (ctrl->nr_changed_ns == U32_MAX)
@@ -241,8 +241,8 @@ static void nvmet_execute_get_log_changed_ns(struct nvmet_req *req)
 	ctrl->nr_changed_ns = 0;
 	nvmet_clear_aen_bit(req, NVME_AEN_BIT_NS_ATTR);
 	mutex_unlock(&ctrl->lock);
-out:
-	nvmet_req_complete(req, status);
+
+	return status;
 }
 
 static u32 nvmet_format_ana_group(struct nvmet_req *req, u32 grpid,
@@ -335,7 +335,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 		status = nvmet_execute_get_log_page_noop(req);
 		break;
 	case NVME_LOG_CHANGED_NS:
-		return nvmet_execute_get_log_changed_ns(req);
+		status = nvmet_execute_get_log_changed_ns(req);
+		break;
 	case NVME_LOG_CMD_EFFECTS:
 		return nvmet_execute_get_log_cmd_effects_ns(req);
 	case NVME_LOG_ANA:
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:09 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:09 -0700
Subject: [PATCH 06/14] nvmet: remove repeated call for effects log page
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-7-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_get_log_page_cmd_effects_ns() and
complete the request in nvmet_execute_get_log_page(). This reduces
number of repeated nvmet_req_complete() calls for log page
handlers.

Also, now we can get rid of the goto and out label needed for request
completion and directly return.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 2b018b84bf3f..cf02d70dedeb 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -186,16 +186,14 @@ static void nvmet_get_cmd_effects_zns(struct nvme_effects_log *log)
 	log->iocs[nvme_cmd_zone_mgmt_recv]	= cpu_to_le32(1 << 0);
 }
 
-static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
+static u16 nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
 {
 	struct nvme_effects_log *log;
 	u16 status = NVME_SC_SUCCESS;
 
 	log = kzalloc(sizeof(*log), GFP_KERNEL);
-	if (!log) {
-		status = NVME_SC_INTERNAL;
-		goto out;
-	}
+	if (!log)
+		return NVME_SC_INTERNAL;
 
 	switch (req->cmd->get_log_page.csi) {
 	case NVME_CSI_NVM:
@@ -217,8 +215,8 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)
 	status = nvmet_copy_to_sgl(req, 0, log, sizeof(*log));
 free:
 	kfree(log);
-out:
-	nvmet_req_complete(req, status);
+
+	return status;
 }
 
 static u16 nvmet_execute_get_log_changed_ns(struct nvmet_req *req)
@@ -338,7 +336,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 		status = nvmet_execute_get_log_changed_ns(req);
 		break;
 	case NVME_LOG_CMD_EFFECTS:
-		return nvmet_execute_get_log_cmd_effects_ns(req);
+		status = nvmet_execute_get_log_cmd_effects_ns(req);
+		break;
 	case NVME_LOG_ANA:
 		return nvmet_execute_get_log_page_ana(req);
 	default:
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:10 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:10 -0700
Subject: [PATCH 07/14] nvmet: remove repeated call for ana log page
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-8-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_get_log_page_ana() and complete the
request in nvmet_execute_get_log_page(). This reduces number of
repeated nvmet_req_complete() calls for log page handlers.

Also, now we can get rid of the goto and out label needed for request
completion and directly return.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index cf02d70dedeb..4e58dfbfbbd5 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -265,7 +265,7 @@ static u32 nvmet_format_ana_group(struct nvmet_req *req, u32 grpid,
 	return sizeof(struct nvme_ana_group_desc) + count * sizeof(__le32);
 }
 
-static void nvmet_execute_get_log_page_ana(struct nvmet_req *req)
+static u16 nvmet_execute_get_log_page_ana(struct nvmet_req *req)
 {
 	struct nvme_ana_rsp_hdr hdr = { 0, };
 	struct nvme_ana_group_desc *desc;
@@ -275,11 +275,10 @@ static void nvmet_execute_get_log_page_ana(struct nvmet_req *req)
 	u16 ngrps = 0;
 	u16 status;
 
-	status = NVME_SC_INTERNAL;
 	desc = kmalloc(sizeof(struct nvme_ana_group_desc) +
 			NVMET_MAX_NAMESPACES * sizeof(__le32), GFP_KERNEL);
 	if (!desc)
-		goto out;
+		return NVME_SC_INTERNAL;
 
 	down_read(&nvmet_ana_sem);
 	for (grpid = 1; grpid <= NVMET_MAX_ANAGRPS; grpid++) {
@@ -305,9 +304,9 @@ static void nvmet_execute_get_log_page_ana(struct nvmet_req *req)
 	kfree(desc);
 
 	/* copy the header last once we know the number of groups */
-	status = nvmet_copy_to_sgl(req, 0, &hdr, sizeof(hdr));
-out:
-	nvmet_req_complete(req, status);
+	if (!status)
+		status = nvmet_copy_to_sgl(req, 0, &hdr, sizeof(hdr));
+	return status;
 }
 
 static void nvmet_execute_get_log_page(struct nvmet_req *req)
@@ -339,7 +338,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 		status = nvmet_execute_get_log_cmd_effects_ns(req);
 		break;
 	case NVME_LOG_ANA:
-		return nvmet_execute_get_log_page_ana(req);
+		status = nvmet_execute_get_log_page_ana(req);
+		break;
 	default:
 		pr_err("unhandled lid %d on qid %d\n",
 		       req->cmd->get_log_page.lid, req->sq->qid);
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:11 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:11 -0700
Subject: [PATCH 08/14] nvmet: add default case for nvmet_execute_identify
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-9-chaitanya.kulkarni@wdc.com>

Just like on what we have done earlier in the series for the
nvmet_execute_get_log_page() repeat the same pattern for the
nvmet_execute_identify(), add a default case to print the error & move
the request completion call outside of the switch so it can be used by
all the switch cases.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 10 +++++++++-
 drivers/nvme/target/nvmet.h     |  8 --------
 2 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 4e58dfbfbbd5..28c24c08bd99 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -682,6 +682,8 @@ static bool nvmet_handle_identify_desclist(struct nvmet_req *req)
 
 static void nvmet_execute_identify(struct nvmet_req *req)
 {
+	u16 status;
+
 	if (!nvmet_check_transfer_len(req, NVME_IDENTIFY_DATA_SIZE))
 		return;
 
@@ -732,9 +734,15 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 		if (nvmet_handle_identify_desclist(req) == true)
 			return;
 		break;
+	default:
+		pr_debug("unhandled identify cns %d on qid %d\n",
+			 req->cmd->identify.cns, req->sq->qid);
+		req->error_loc = offsetof(struct nvme_identify, cns);
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		break;
 	}
 
-	nvmet_req_cns_error_complete(req);
+	nvmet_req_complete(req, status);
 }
 
 /*
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index d719a1cd5dda..2119abc2a9b9 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -645,14 +645,6 @@ static inline bool nvmet_use_inline_bvec(struct nvmet_req *req)
 	       req->sg_cnt <= NVMET_MAX_INLINE_BIOVEC;
 }
 
-static inline void nvmet_req_cns_error_complete(struct nvmet_req *req)
-{
-	pr_debug("unhandled identify cns %d on qid %d\n",
-	       req->cmd->identify.cns, req->sq->qid);
-	req->error_loc = offsetof(struct nvme_identify, cns);
-	nvmet_req_complete(req, NVME_SC_INVALID_FIELD | NVME_SC_DNR);
-}
-
 static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio)
 {
 	if (bio != &req->b.inline_bio)
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:12 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:12 -0700
Subject: [PATCH 09/14] nvmet: remove repeated call for id-ctrl
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-10-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_identidy_ctrl() and complete request
in nvmet_execute_identify(). This reduces number of repeated
 nvmet_req_complete() calls for log page handlers.

Also, now we can get rid of the goto and out label needed for request
completion and directly return.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 28c24c08bd99..4f88276e0c48 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -349,7 +349,7 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 	nvmet_req_complete(req, status);
 }
 
-static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
+static u16 nvmet_execute_identify_ctrl(struct nvmet_req *req)
 {
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	struct nvmet_subsys *subsys = ctrl->subsys;
@@ -364,10 +364,8 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	}
 
 	id = kzalloc(sizeof(*id), GFP_KERNEL);
-	if (!id) {
-		status = NVME_SC_INTERNAL;
-		goto out;
-	}
+	if (!id)
+		return NVME_SC_INTERNAL;
 
 	/* XXX: figure out how to assign real vendors IDs. */
 	id->vid = 0;
@@ -484,8 +482,8 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	status = nvmet_copy_to_sgl(req, 0, id, sizeof(*id));
 
 	kfree(id);
-out:
-	nvmet_req_complete(req, status);
+
+	return status;
 }
 
 static void nvmet_execute_identify_ns(struct nvmet_req *req)
@@ -709,7 +707,8 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 	case NVME_ID_CNS_CTRL:
 		switch (req->cmd->identify.csi) {
 		case NVME_CSI_NVM:
-			return nvmet_execute_identify_ctrl(req);
+			status = nvmet_execute_identify_ctrl(req);
+			break;
 		}
 		break;
 	case NVME_ID_CNS_CS_CTRL:
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:13 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:13 -0700
Subject: [PATCH 10/14] nvmet: remove repeated call for id-ns
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-11-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_identidy_ns() and complete the request
in nvmet_execute_identify(). This reduces number of repeated
nvmet_req_complete() calls for identify handlers.

Also, now we can get rid of the goto and out label needed for request
completion and directly return.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 19 +++++++++----------
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 4f88276e0c48..e625a33d0861 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -486,22 +486,20 @@ static u16 nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	return status;
 }
 
-static void nvmet_execute_identify_ns(struct nvmet_req *req)
+static u16 nvmet_execute_identify_ns(struct nvmet_req *req)
 {
 	struct nvme_id_ns *id;
 	u16 status;
 
 	if (le32_to_cpu(req->cmd->identify.nsid) == NVME_NSID_ALL) {
 		req->error_loc = offsetof(struct nvme_identify, nsid);
-		status = NVME_SC_INVALID_NS | NVME_SC_DNR;
-		goto out;
+		return NVME_SC_INVALID_NS | NVME_SC_DNR;
 	}
 
 	id = kzalloc(sizeof(*id), GFP_KERNEL);
-	if (!id) {
-		status = NVME_SC_INTERNAL;
-		goto out;
-	}
+	if (!id)
+		return NVME_SC_INTERNAL;
+
 
 	/* return an all zeroed buffer if we can't find an active namespace */
 	status = nvmet_req_find_ns(req);
@@ -565,8 +563,8 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req)
 		status = nvmet_copy_to_sgl(req, 0, id, sizeof(*id));
 
 	kfree(id);
-out:
-	nvmet_req_complete(req, status);
+
+	return status;
 }
 
 static void nvmet_execute_identify_nslist(struct nvmet_req *req)
@@ -689,7 +687,8 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 	case NVME_ID_CNS_NS:
 		switch (req->cmd->identify.csi) {
 		case NVME_CSI_NVM:
-			return nvmet_execute_identify_ns(req);
+			status = nvmet_execute_identify_ns(req);
+			break;
 		default:
 			break;
 		}
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:14 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:14 -0700
Subject: [PATCH 11/14] nvmet: remove repeated call for nslist
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-12-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_identidy_nslist() and complete the
request in nvmet_execute_identify(). This reduces number of repeated
nvmet_req_complete() calls for identify handlers.

Also, now we can get rid of the goto and out label needed for request
completion and directly return.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index e625a33d0861..749d7e695fa2 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -567,7 +567,7 @@ static u16 nvmet_execute_identify_ns(struct nvmet_req *req)
 	return status;
 }
 
-static void nvmet_execute_identify_nslist(struct nvmet_req *req)
+static u16 nvmet_execute_identify_nslist(struct nvmet_req *req)
 {
 	static const int buf_size = NVME_IDENTIFY_DATA_SIZE;
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
@@ -575,14 +575,12 @@ static void nvmet_execute_identify_nslist(struct nvmet_req *req)
 	unsigned long idx;
 	u32 min_nsid = le32_to_cpu(req->cmd->identify.nsid);
 	__le32 *list;
-	u16 status = 0;
+	u16 status;
 	int i = 0;
 
 	list = kzalloc(buf_size, GFP_KERNEL);
-	if (!list) {
-		status = NVME_SC_INTERNAL;
-		goto out;
-	}
+	if (!list)
+		return NVME_SC_INTERNAL;
 
 	xa_for_each(&ctrl->subsys->namespaces, idx, ns) {
 		if (ns->nsid <= min_nsid)
@@ -595,8 +593,8 @@ static void nvmet_execute_identify_nslist(struct nvmet_req *req)
 	status = nvmet_copy_to_sgl(req, 0, list, buf_size);
 
 	kfree(list);
-out:
-	nvmet_req_complete(req, status);
+
+	return status;
 }
 
 static u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
@@ -723,7 +721,8 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 	case NVME_ID_CNS_NS_ACTIVE_LIST:
 		switch (req->cmd->identify.csi) {
 		case NVME_CSI_NVM:
-			return nvmet_execute_identify_nslist(req);
+			status = nvmet_execute_identify_nslist(req);
+			break;
 		default:
 			break;
 		}
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:15 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:15 -0700
Subject: [PATCH 12/14] nvmet: remove repeated call for desclist
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-13-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_identidy_desclist() and complete
the request in nvmet_execute_identify(). This reduces number of
repeated nvmet_req_complete() calls for identify handlers.

Also, now we can get rid of the goto and out label needed for request
completion and directly return.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c | 33 ++++++++++++++-------------------
 1 file changed, 14 insertions(+), 19 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 749d7e695fa2..02d28158e9f0 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -619,58 +619,54 @@ static u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
 	return 0;
 }
 
-static void nvmet_execute_identify_desclist(struct nvmet_req *req)
+static u16 nvmet_execute_identify_desclist(struct nvmet_req *req)
 {
 	off_t off = 0;
 	u16 status;
 
 	status = nvmet_req_find_ns(req);
 	if (status)
-		goto out;
+		return status;
 
 	if (memchr_inv(&req->ns->uuid, 0, sizeof(req->ns->uuid))) {
 		status = nvmet_copy_ns_identifier(req, NVME_NIDT_UUID,
 						  NVME_NIDT_UUID_LEN,
 						  &req->ns->uuid, &off);
 		if (status)
-			goto out;
+			return status;
 	}
 	if (memchr_inv(req->ns->nguid, 0, sizeof(req->ns->nguid))) {
 		status = nvmet_copy_ns_identifier(req, NVME_NIDT_NGUID,
 						  NVME_NIDT_NGUID_LEN,
 						  &req->ns->nguid, &off);
 		if (status)
-			goto out;
+			return status;
 	}
 
 	status = nvmet_copy_ns_identifier(req, NVME_NIDT_CSI,
 					  NVME_NIDT_CSI_LEN,
 					  &req->ns->csi, &off);
 	if (status)
-		goto out;
+		return status;
 
 	if (sg_zero_buffer(req->sg, req->sg_cnt, NVME_IDENTIFY_DATA_SIZE - off,
 			off) != NVME_IDENTIFY_DATA_SIZE - off)
-		status = NVME_SC_INTERNAL | NVME_SC_DNR;
+		return NVME_SC_INTERNAL | NVME_SC_DNR;
 
-out:
-	nvmet_req_complete(req, status);
+	return status;
 }
 
-static bool nvmet_handle_identify_desclist(struct nvmet_req *req)
+static u16 nvmet_handle_identify_desclist(struct nvmet_req *req)
 {
 	switch (req->cmd->identify.csi) {
 	case NVME_CSI_NVM:
-		nvmet_execute_identify_desclist(req);
-		return true;
+		return nvmet_execute_identify_desclist(req);
 	case NVME_CSI_ZNS:
-		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
-			nvmet_execute_identify_desclist(req);
-			return true;
-		}
-		return false;
+		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED))
+			return nvmet_execute_identify_desclist(req);
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
 	default:
-		return false;
+		return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
 	}
 }
 
@@ -728,8 +724,7 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 		}
 		break;
 	case NVME_ID_CNS_NS_DESC_LIST:
-		if (nvmet_handle_identify_desclist(req) == true)
-			return;
+		status = nvmet_handle_identify_desclist(req);
 		break;
 	default:
 		pr_debug("unhandled identify cns %d on qid %d\n",
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:16 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:16 -0700
Subject: [PATCH 13/14] nvmet: remove repeated call for id-cns-ns
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-14-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_identify_cns_cs_ns() and complete the
request in nvmet_execute_identify(). This reduces number of repeated
nvmet_req_complete() calls for identify handlers.

Also, now we can get rid of the goto and out label needed for request
completion and directly return.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c |  3 ++-
 drivers/nvme/target/nvmet.h     |  2 +-
 drivers/nvme/target/zns.c       | 15 ++++++---------
 3 files changed, 9 insertions(+), 11 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 02d28158e9f0..ef6982beb6b0 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -691,7 +691,8 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
 			switch (req->cmd->identify.csi) {
 			case NVME_CSI_ZNS:
-				return nvmet_execute_identify_cns_cs_ns(req);
+				status = nvmet_execute_identify_cns_cs_ns(req);
+				break;
 			default:
 				break;
 			}
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 2119abc2a9b9..f1c9cd445c74 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -546,7 +546,7 @@ u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts);
 
 bool nvmet_bdev_zns_enable(struct nvmet_ns *ns);
 void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req);
-void nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req);
+u16 nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req);
 void nvmet_bdev_execute_zone_mgmt_recv(struct nvmet_req *req);
 void nvmet_bdev_execute_zone_mgmt_send(struct nvmet_req *req);
 void nvmet_bdev_execute_zone_append(struct nvmet_req *req);
diff --git a/drivers/nvme/target/zns.c b/drivers/nvme/target/zns.c
index 17f8b7a45f21..2e1f01ec0dd1 100644
--- a/drivers/nvme/target/zns.c
+++ b/drivers/nvme/target/zns.c
@@ -96,7 +96,7 @@ void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req)
 	nvmet_req_complete(req, status);
 }
 
-void nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req)
+u16 nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req)
 {
 	struct nvme_id_ns_zns *id_zns;
 	u64 zsze;
@@ -104,15 +104,12 @@ void nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req)
 
 	if (le32_to_cpu(req->cmd->identify.nsid) == NVME_NSID_ALL) {
 		req->error_loc = offsetof(struct nvme_identify, nsid);
-		status = NVME_SC_INVALID_NS | NVME_SC_DNR;
-		goto out;
+		return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
 	}
 
 	id_zns = kzalloc(sizeof(*id_zns), GFP_KERNEL);
-	if (!id_zns) {
-		status = NVME_SC_INTERNAL;
-		goto out;
-	}
+	if (!id_zns)
+		return NVME_SC_INTERNAL;
 
 	status = nvmet_req_find_ns(req);
 	if (status) {
@@ -136,8 +133,8 @@ void nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req)
 done:
 	status = nvmet_copy_to_sgl(req, 0, id_zns, sizeof(*id_zns));
 	kfree(id_zns);
-out:
-	nvmet_req_complete(req, status);
+
+	return status;
 }
 
 static u16 nvmet_bdev_validate_zone_mgmt_recv(struct nvmet_req *req)
-- 
2.22.1



From chaitanya.kulkarni at wdc.com  Sun Jun 27 22:08:17 2021
From: chaitanya.kulkarni at wdc.com (Chaitanya Kulkarni)
Date: Sun, 27 Jun 2021 22:08:17 -0700
Subject: [PATCH 14/14] nvmet: remove repeated call for id-cns-ctrl
In-Reply-To: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
References: <20210628050817.19788-1-chaitanya.kulkarni@wdc.com>
Message-ID: <20210628050817.19788-15-chaitanya.kulkarni@wdc.com>

Return status from nvmet_execute_identify_cns_cs_ctrl() and complete
the request in nvmet_execute_identify(). This reduces number of
repeated nvmet_req_complete() calls for identify handlers.

Also, now we can get rid of the goto and out label needed for request
completion and directly return.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
---
 drivers/nvme/target/admin-cmd.c |  4 +++-
 drivers/nvme/target/nvmet.h     |  2 +-
 drivers/nvme/target/zns.c       | 12 +++++-------
 3 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index ef6982beb6b0..1a8ca93b6bb9 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -709,7 +709,9 @@ static void nvmet_execute_identify(struct nvmet_req *req)
 		if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
 			switch (req->cmd->identify.csi) {
 			case NVME_CSI_ZNS:
-				return nvmet_execute_identify_cns_cs_ctrl(req);
+				status =
+				nvmet_execute_identify_cns_cs_ctrl(req);
+				break;
 			default:
 				break;
 			}
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index f1c9cd445c74..b2eeb296cc37 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -545,7 +545,7 @@ void nvmet_ns_revalidate(struct nvmet_ns *ns);
 u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts);
 
 bool nvmet_bdev_zns_enable(struct nvmet_ns *ns);
-void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req);
+u16 nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req);
 u16 nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req);
 void nvmet_bdev_execute_zone_mgmt_recv(struct nvmet_req *req);
 void nvmet_bdev_execute_zone_mgmt_send(struct nvmet_req *req);
diff --git a/drivers/nvme/target/zns.c b/drivers/nvme/target/zns.c
index 2e1f01ec0dd1..28a1de2e0fbf 100644
--- a/drivers/nvme/target/zns.c
+++ b/drivers/nvme/target/zns.c
@@ -71,7 +71,7 @@ bool nvmet_bdev_zns_enable(struct nvmet_ns *ns)
 	return true;
 }
 
-void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req)
+u16 nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req)
 {
 	u8 zasl = req->sq->ctrl->subsys->zasl;
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
@@ -79,10 +79,8 @@ void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req)
 	u16 status;
 
 	id = kzalloc(sizeof(*id), GFP_KERNEL);
-	if (!id) {
-		status = NVME_SC_INTERNAL;
-		goto out;
-	}
+	if (!id)
+		return NVME_SC_INTERNAL;
 
 	if (ctrl->ops->get_mdts)
 		id->zasl = min_t(u8, ctrl->ops->get_mdts(ctrl), zasl);
@@ -92,8 +90,8 @@ void nvmet_execute_identify_cns_cs_ctrl(struct nvmet_req *req)
 	status = nvmet_copy_to_sgl(req, 0, id, sizeof(*id));
 
 	kfree(id);
-out:
-	nvmet_req_complete(req, status);
+
+	return status;
 }
 
 u16 nvmet_execute_identify_cns_cs_ns(struct nvmet_req *req)
-- 
2.22.1



From gerlitz.or at gmail.com  Mon Jun 28 00:10:36 2021
From: gerlitz.or at gmail.com (Or Gerlitz)
Date: Mon, 28 Jun 2021 10:10:36 +0300
Subject: [PATCH v3 08/20] nvme-tcp-offload: Add IO level implementation
In-Reply-To: <20210624172410.14225-9-smalin@marvell.com>
References: <20210624172410.14225-1-smalin@marvell.com>
 <20210624172410.14225-9-smalin@marvell.com>
Message-ID: <CAJ3xEMhtrEEMLbkWuna_fvH_oinaCZmj3xr4y7gzdRawA1gmNA@mail.gmail.com>

On Thu, Jun 24, 2021 at 8:41 PM Shai Malin <smalin at marvell.com> wrote:
>
> From: Dean Balandin <dbalandin at marvell.com>
> In this patch, we present the IO level functionality.
[..]

> +static void nvme_tcp_ofld_set_sg_null(struct nvme_command *c)
> +{
> +       struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
> +       sg->addr = 0;

ok

> +       sg->length = 0;
> +       sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) |
> +                       NVME_SGL_FMT_TRANSPORT_A;

> +inline void nvme_tcp_ofld_set_sg_inline(struct nvme_tcp_ofld_queue *queue,
> +                                       struct nvme_command *c, u32 data_len)
> +{
> +       struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
> +       sg->addr = cpu_to_le64(queue->ctrl->nctrl.icdoff);

ok, what about dma mapping of the address?

> +       sg->length = cpu_to_le32(data_len);
> +       sg->type = (NVME_SGL_FMT_DATA_DESC << 4) | NVME_SGL_FMT_OFFSET;

> +static void nvme_tcp_ofld_map_data(struct nvme_command *c, u32 data_len)
> +{
> +       struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
> +
> +       sg->addr = 0;

???

> +       sg->length = cpu_to_le32(data_len);
> +       sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) |
> +                       NVME_SGL_FMT_TRANSPORT_A;


From malin1024 at gmail.com  Mon Jun 28 03:42:22 2021
From: malin1024 at gmail.com (Shai Malin)
Date: Mon, 28 Jun 2021 13:42:22 +0300
Subject: [PATCH v3 08/20] nvme-tcp-offload: Add IO level implementation
In-Reply-To: <CAJ3xEMhtrEEMLbkWuna_fvH_oinaCZmj3xr4y7gzdRawA1gmNA@mail.gmail.com>
References: <20210624172410.14225-1-smalin@marvell.com>
 <20210624172410.14225-9-smalin@marvell.com>
 <CAJ3xEMhtrEEMLbkWuna_fvH_oinaCZmj3xr4y7gzdRawA1gmNA@mail.gmail.com>
Message-ID: <CAKKgK4wJDrNcSbaPERPg2EyrSNvi8KY9UHyM3=T9bL_d3WBSDQ@mail.gmail.com>

On Mon, 28 Jun 2021 at 10:10, Or Gerlitz <gerlitz.or at gmail.com> wrote:
> On Thu, Jun 24, 2021 at 8:41 PM Shai Malin <smalin at marvell.com> wrote:
> >
> > From: Dean Balandin <dbalandin at marvell.com>
> > In this patch, we present the IO level functionality.
> [..]
>
> > +static void nvme_tcp_ofld_set_sg_null(struct nvme_command *c)
> > +{
> > +       struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
> > +       sg->addr = 0;
>
> ok
>
> > +       sg->length = 0;
> > +       sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) |
> > +                       NVME_SGL_FMT_TRANSPORT_A;
>
> > +inline void nvme_tcp_ofld_set_sg_inline(struct nvme_tcp_ofld_queue *queue,
> > +                                       struct nvme_command *c, u32 data_len)
> > +{
> > +       struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
> > +       sg->addr = cpu_to_le64(queue->ctrl->nctrl.icdoff);
>
> ok, what about dma mapping of the address?

The dma mapping is done by the offload device driver.
patch 18 - "qedn: Add IO level fastpath functionality", in qedn_init_sgl().

>
> > +       sg->length = cpu_to_le32(data_len);
> > +       sg->type = (NVME_SGL_FMT_DATA_DESC << 4) | NVME_SGL_FMT_OFFSET;
>
> > +static void nvme_tcp_ofld_map_data(struct nvme_command *c, u32 data_len)
> > +{
> > +       struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
> > +
> > +       sg->addr = 0;
>
> ???

We will rename the function: nvme_tcp_ofld_set_sg_host_data().
The dma mapping is done by the offload device driver.

>
> > +       sg->length = cpu_to_le32(data_len);
> > +       sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) |
> > +                       NVME_SGL_FMT_TRANSPORT_A;


From ming.lei at redhat.com  Mon Jun 28 18:31:03 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Tue, 29 Jun 2021 09:31:03 +0800
Subject: [PATCH 0/2] Handle update hardware queues and queue freeze more
 carefully
In-Reply-To: <YNXTaUMAFCA84jfZ@T590>
References: <20210625101649.49296-1-dwagner@suse.de>
 <20210625122156.x5yzoobuaaec5hss@beryllium.lan>
 <YNXTaUMAFCA84jfZ@T590>
Message-ID: <CAFj5m9J9NL9qHjo1X9=PdE1-Nkgj2zV-ifdZ9aqqts2QNUpf8w@mail.gmail.com>

On Fri, Jun 25, 2021 at 9:00 PM Ming Lei <ming.lei at redhat.com> wrote:
>
> On Fri, Jun 25, 2021 at 02:21:56PM +0200, Daniel Wagner wrote:
> > On Fri, Jun 25, 2021 at 12:16:47PM +0200, Daniel Wagner wrote:
> > > this is a followup on the crash I reported in
> > >
> > >   https://lore.kernel.org/linux-block/20210608183339.70609-1-dwagner at suse.de/
> > >
> > > By moving the hardware check up the crash was gone. Unfortuntatly, I
> > > don't understand why this fixes the crash. The per-cpu access is
> > > crashing but I can't see why the blk_mq_update_nr_hw_queues() is
> > > fixing this problem.
> > >
> > > Even though I can't explain why it fixes it, I think it makes sense to
> > > update the hardware queue mapping bevore we recreate the IO
> > > queues. Thus I avoided in the commit message to say it fixes
> > > something.
> >
> > I just discussed this with Hannes and we figured out how the crash is
> > fixed by moving the blk_mq_update_nr_hw_queues() before the
> > nvme_fc_create_hw_io_queues()/nvme_fc_connect_io_queues().
> >
> > First of all, blk_mq_update_nr_hw_queues() operates on the normal
> > tag_set and not the admin_tag_set. That means when we move the
> > blk_mq_update_nr_hw_queues() before the nvme_fc_connect_io_queues(), we
> > update the mapping to only CPUs and hwctx which are available. When we
> > then do the connect call nvmf_connect_io_queue() we will only allocate
> > tags from queues which are not in the BLK_MQ_S_INACTIVE anymore. Hence
> > we skip the blk_mq_put_tag() call.
>
> Your patch just reduces the race window, what if all cpus in
> hctx->cpumask become offline when calling blk_mq_alloc_request_hctx()?

connect io queues after updating nr_hw_queues can cause correct hctx_idx
to be passed to blk_mq_alloc_request_hctx(), so this patch is good, so the patch
looks good.

Yeah, there is still other issue not covered during cpu hotplug, but
that is different
with this one.

Thanks,



From ming.lei at redhat.com  Mon Jun 28 18:32:11 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Tue, 29 Jun 2021 09:32:11 +0800
Subject: [PATCH 1/2] nvme-fc: Update hardware queues before using them
In-Reply-To: <20210625101649.49296-2-dwagner@suse.de>
References: <20210625101649.49296-1-dwagner@suse.de>
 <20210625101649.49296-2-dwagner@suse.de>
Message-ID: <YNp4GyXwOlJeqtby@T590>

On Fri, Jun 25, 2021 at 12:16:48PM +0200, Daniel Wagner wrote:
> In case the number of hardware queues changes, do the update the
> tagset and ctx to hctx first before using the mapping to recreate and
> connnect the IO queues.
> 
> Signed-off-by: Daniel Wagner <dwagner at suse.de>
> ---
>  drivers/nvme/host/fc.c | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index 8a3c4814d21b..a9645cd89eca 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -2951,14 +2951,6 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
>  	if (ctrl->ctrl.queue_count == 1)
>  		return 0;
>  
> -	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
> -	if (ret)
> -		goto out_free_io_queues;
> -
> -	ret = nvme_fc_connect_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
> -	if (ret)
> -		goto out_delete_hw_queues;
> -
>  	if (prior_ioq_cnt != nr_io_queues) {
>  		dev_info(ctrl->ctrl.device,
>  			"reconnect: revising io queue count from %d to %d\n",
> @@ -2968,6 +2960,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
>  		nvme_unfreeze(&ctrl->ctrl);
>  	}
>  
> +	ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
> +	if (ret)
> +		goto out_free_io_queues;
> +
> +	ret = nvme_fc_connect_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
> +	if (ret)
> +		goto out_delete_hw_queues;
> +
>  	return 0;
>  
>  out_delete_hw_queues:
> -- 
> 2.29.2
> 

This way may cause correct hctx_idx to be passed to blk_mq_alloc_request_hctx(), so:

Reviewed-by: Ming Lei <ming.lei at redhat.com>


Thanks,
Ming



From ming.lei at redhat.com  Mon Jun 28 18:39:30 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Tue, 29 Jun 2021 09:39:30 +0800
Subject: [PATCH 2/2] nvme-fc: Wait with a timeout for queue to freeze
In-Reply-To: <20210625101649.49296-3-dwagner@suse.de>
References: <20210625101649.49296-1-dwagner@suse.de>
 <20210625101649.49296-3-dwagner@suse.de>
Message-ID: <YNp50pmlzN6M0kNX@T590>

On Fri, Jun 25, 2021 at 12:16:49PM +0200, Daniel Wagner wrote:
> Do not wait indifinitly for all queues to freeze. Instead use a
> timeout and abort the operation if we get stuck.
> 
> Signed-off-by: Daniel Wagner <dwagner at suse.de>
> ---
>  drivers/nvme/host/fc.c | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index a9645cd89eca..d8db85aa5417 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -2955,7 +2955,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
>  		dev_info(ctrl->ctrl.device,
>  			"reconnect: revising io queue count from %d to %d\n",
>  			prior_ioq_cnt, nr_io_queues);
> -		nvme_wait_freeze(&ctrl->ctrl);
> +		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
> +			/*
> +			 * If we timed out waiting for freeze we are likely to
> +			 * be stuck.  Fail the controller initialization just
> +			 * to be safe.
> +			 */
> +			return -ENODEV;
> +		}
>  		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
>  		nvme_unfreeze(&ctrl->ctrl);

Can you investigate a bit on why there is the hang? FC shouldn't use
managed IRQ, so the interrupt won't be shutdown.

blk-mq debugfs may help to dump the requests after the hang is triggered,
or you still can add debug code in nvme_wait_freeze_timeout() to dump
all requests if blk-mq debugfs doesn't work.


Thanks,
Ming



From dwagner at suse.de  Tue Jun 29 00:48:25 2021
From: dwagner at suse.de (Daniel Wagner)
Date: Tue, 29 Jun 2021 09:48:25 +0200
Subject: [PATCH 2/2] nvme-fc: Wait with a timeout for queue to freeze
In-Reply-To: <YNp50pmlzN6M0kNX@T590>
References: <20210625101649.49296-1-dwagner@suse.de>
 <20210625101649.49296-3-dwagner@suse.de> <YNp50pmlzN6M0kNX@T590>
Message-ID: <20210629074825.s5f2d3ihuyscktg3@beryllium.lan>

On Tue, Jun 29, 2021 at 09:39:30AM +0800, Ming Lei wrote:
> On Fri, Jun 25, 2021 at 12:16:49PM +0200, Daniel Wagner wrote:
> > Do not wait indifinitly for all queues to freeze. Instead use a
> > timeout and abort the operation if we get stuck.
> > 
> > Signed-off-by: Daniel Wagner <dwagner at suse.de>
> > ---
> >  drivers/nvme/host/fc.c | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> > index a9645cd89eca..d8db85aa5417 100644
> > --- a/drivers/nvme/host/fc.c
> > +++ b/drivers/nvme/host/fc.c
> > @@ -2955,7 +2955,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
> >  		dev_info(ctrl->ctrl.device,
> >  			"reconnect: revising io queue count from %d to %d\n",
> >  			prior_ioq_cnt, nr_io_queues);
> > -		nvme_wait_freeze(&ctrl->ctrl);
> > +		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
> > +			/*
> > +			 * If we timed out waiting for freeze we are likely to
> > +			 * be stuck.  Fail the controller initialization just
> > +			 * to be safe.
> > +			 */
> > +			return -ENODEV;
> > +		}
> >  		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
> >  		nvme_unfreeze(&ctrl->ctrl);
> 
> Can you investigate a bit on why there is the hang? FC shouldn't use
> managed IRQ, so the interrupt won't be shutdown.
> 
> blk-mq debugfs may help to dump the requests after the hang is triggered,
> or you still can add debug code in nvme_wait_freeze_timeout() to dump
> all requests if blk-mq debugfs doesn't work.

Sure thing, I'll try to find out why it hangs. The good thing is that I
was able to reliable reproduce it. So let's see.


From ming.lei at redhat.com  Tue Jun 29 00:49:49 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Tue, 29 Jun 2021 15:49:49 +0800
Subject: [PATCH 0/2] blk-mq: fix blk_mq_alloc_request_hctx
Message-ID: <20210629074951.1981284-1-ming.lei@redhat.com>

Hi,

blk_mq_alloc_request_hctx() is used by NVMe fc/rdma/tcp/loop to connect
io queue. Also the sw ctx is chosen as the 1st online cpu in hctx->cpumask.
However, all cpus in hctx->cpumask may be offline.

This usage model isn't well supported by blk-mq which supposes allocator is
always done on one online CPU in hctx->cpumask. This assumption is
related with managed irq, which also requires blk-mq to drain inflight
request in this hctx when the last cpu in hctx->cpumask is going to
offline.

However, NVMe fc/rdma/tcp/loop don't use managed irq, so we should allow
them to ask for request allocation when the specified hctx is inactive
(all cpus in hctx->cpumask are offline).

Fix blk_mq_alloc_request_hctx() by adding/passing flag of
BLK_MQ_F_NOT_USE_MANAGED_IRQ. 


Ming Lei (2):
  blk-mq: not deactivate hctx if the device doesn't use managed irq
  nvme: pass BLK_MQ_F_NOT_USE_MANAGED_IRQ for fc/rdma/tcp/loop

 block/blk-mq.c             | 6 +++++-
 drivers/nvme/host/fc.c     | 3 ++-
 drivers/nvme/host/rdma.c   | 3 ++-
 drivers/nvme/host/tcp.c    | 3 ++-
 drivers/nvme/target/loop.c | 3 ++-
 include/linux/blk-mq.h     | 1 +
 6 files changed, 14 insertions(+), 5 deletions(-)

Cc: Sagi Grimberg <sagi at grimberg.me>
Cc: Daniel Wagner <dwagner at suse.de>
Cc: Wen Xiong <wenxiong at us.ibm.com>
Cc: John Garry <john.garry at huawei.com>


-- 
2.31.1



From ming.lei at redhat.com  Tue Jun 29 00:49:50 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Tue, 29 Jun 2021 15:49:50 +0800
Subject: [PATCH 1/2] blk-mq: not deactivate hctx if the device doesn't use
 managed irq
In-Reply-To: <20210629074951.1981284-1-ming.lei@redhat.com>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
Message-ID: <20210629074951.1981284-2-ming.lei@redhat.com>

hctx is deactivated when all CPU in hctx->cpumask become offline by
draining all requests originated from this hctx and moving new
allocation to active hctx. This way is for avoiding inflight IO when
the managed irq is shutdown.

Some drivers(nvme fc, rdma, tcp, loop) doesn't use managed irq, so
they needn't to deactivate hctx. Also, they are the only user of
blk_mq_alloc_request_hctx() which is used for connecting io queue.
And their requirement is that the connect request can be submitted
via one specified hctx on which all CPU in its hctx->cpumask may have
become offline.

Address the requirement for nvme fc/rdma/loop, so the reported kernel
panic on the following line in blk_mq_alloc_request_hctx() can be fixed.

	data.ctx = __blk_mq_get_ctx(q, cpu)

Cc: Sagi Grimberg <sagi at grimberg.me>
Cc: Daniel Wagner <dwagner at suse.de>
Cc: Wen Xiong <wenxiong at us.ibm.com>
Cc: John Garry <john.garry at huawei.com>
Signed-off-by: Ming Lei <ming.lei at redhat.com>
---
 block/blk-mq.c         | 6 +++++-
 include/linux/blk-mq.h | 1 +
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index df5dc3b756f5..74632f50d969 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -494,7 +494,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
 	data.hctx = q->queue_hw_ctx[hctx_idx];
 	if (!blk_mq_hw_queue_mapped(data.hctx))
 		goto out_queue_exit;
-	cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
+	cpu = cpumask_first(data.hctx->cpumask);
 	data.ctx = __blk_mq_get_ctx(q, cpu);
 
 	if (!q->elevator)
@@ -2570,6 +2570,10 @@ static int blk_mq_hctx_notify_offline(unsigned int cpu, struct hlist_node *node)
 	    !blk_mq_last_cpu_in_hctx(cpu, hctx))
 		return 0;
 
+	/* Controller doesn't use managed IRQ, no need to deactivate hctx */
+	if (hctx->flags & BLK_MQ_F_NOT_USE_MANAGED_IRQ)
+		return 0;
+
 	/*
 	 * Prevent new request from being allocated on the current hctx.
 	 *
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 21140132a30d..600c5dd1a069 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -403,6 +403,7 @@ enum {
 	 */
 	BLK_MQ_F_STACKING	= 1 << 2,
 	BLK_MQ_F_TAG_HCTX_SHARED = 1 << 3,
+	BLK_MQ_F_NOT_USE_MANAGED_IRQ = 1 << 4,
 	BLK_MQ_F_BLOCKING	= 1 << 5,
 	BLK_MQ_F_NO_SCHED	= 1 << 6,
 	BLK_MQ_F_ALLOC_POLICY_START_BIT = 8,
-- 
2.31.1



From ming.lei at redhat.com  Tue Jun 29 00:49:51 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Tue, 29 Jun 2021 15:49:51 +0800
Subject: [PATCH 2/2] nvme: pass BLK_MQ_F_NOT_USE_MANAGED_IRQ for
 fc/rdma/tcp/loop
In-Reply-To: <20210629074951.1981284-1-ming.lei@redhat.com>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
Message-ID: <20210629074951.1981284-3-ming.lei@redhat.com>

All the 4 host drivers don't use managed irq for completing request, so
it is correct to pass the flag to blk-mq.

Secondly with this flag, blk-mq will help us dispatch connect request
allocated via blk_mq_alloc_request_hctx() to driver even though all
CPU in the specified hctx's cpumask are offline.

Cc: Sagi Grimberg <sagi at grimberg.me>
Cc: Daniel Wagner <dwagner at suse.de>
Cc: Wen Xiong <wenxiong at us.ibm.com>
Cc: John Garry <john.garry at huawei.com>
Signed-off-by: Ming Lei <ming.lei at redhat.com>
---
 drivers/nvme/host/fc.c     | 3 ++-
 drivers/nvme/host/rdma.c   | 3 ++-
 drivers/nvme/host/tcp.c    | 3 ++-
 drivers/nvme/target/loop.c | 3 ++-
 4 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 256e87721a01..c563a2b6e9fc 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2876,7 +2876,8 @@ nvme_fc_create_io_queues(struct nvme_fc_ctrl *ctrl)
 	ctrl->tag_set.queue_depth = ctrl->ctrl.opts->queue_size;
 	ctrl->tag_set.reserved_tags = NVMF_RESERVED_TAGS;
 	ctrl->tag_set.numa_node = ctrl->ctrl.numa_node;
-	ctrl->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
+	ctrl->tag_set.flags = BLK_MQ_F_SHOULD_MERGE |
+		BLK_MQ_F_NOT_USE_MANAGED_IRQ;
 	ctrl->tag_set.cmd_size =
 		struct_size((struct nvme_fcp_op_w_sgl *)NULL, priv,
 			    ctrl->lport->ops->fcprqst_priv_sz);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 37943dc4c2c1..4b7bdc829109 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -817,7 +817,8 @@ static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl,
 		set->queue_depth = nctrl->sqsize + 1;
 		set->reserved_tags = NVMF_RESERVED_TAGS;
 		set->numa_node = nctrl->numa_node;
-		set->flags = BLK_MQ_F_SHOULD_MERGE;
+		set->flags = BLK_MQ_F_SHOULD_MERGE |
+			BLK_MQ_F_NOT_USE_MANAGED_IRQ;
 		set->cmd_size = sizeof(struct nvme_rdma_request) +
 				NVME_RDMA_DATA_SGL_SIZE;
 		if (nctrl->max_integrity_segments)
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 34f4b3402f7c..0125463b7d77 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1600,7 +1600,8 @@ static struct blk_mq_tag_set *nvme_tcp_alloc_tagset(struct nvme_ctrl *nctrl,
 		set->queue_depth = nctrl->sqsize + 1;
 		set->reserved_tags = NVMF_RESERVED_TAGS;
 		set->numa_node = nctrl->numa_node;
-		set->flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_BLOCKING;
+		set->flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_BLOCKING |
+			BLK_MQ_F_NOT_USE_MANAGED_IRQ;
 		set->cmd_size = sizeof(struct nvme_tcp_request);
 		set->driver_data = ctrl;
 		set->nr_hw_queues = nctrl->queue_count - 1;
diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index cb30cb942e1d..bf032249e010 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -524,7 +524,8 @@ static int nvme_loop_create_io_queues(struct nvme_loop_ctrl *ctrl)
 	ctrl->tag_set.queue_depth = ctrl->ctrl.opts->queue_size;
 	ctrl->tag_set.reserved_tags = NVMF_RESERVED_TAGS;
 	ctrl->tag_set.numa_node = ctrl->ctrl.numa_node;
-	ctrl->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
+	ctrl->tag_set.flags = BLK_MQ_F_SHOULD_MERGE |
+		BLK_MQ_F_NOT_USE_MANAGED_IRQ;
 	ctrl->tag_set.cmd_size = sizeof(struct nvme_loop_iod) +
 		NVME_INLINE_SG_CNT * sizeof(struct scatterlist);
 	ctrl->tag_set.driver_data = ctrl;
-- 
2.31.1



From hare at suse.de  Tue Jun 29 04:00:06 2021
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 29 Jun 2021 13:00:06 +0200
Subject: [PATCH 0/2] nvme-fabrics: short-circuit connect retries
In-Reply-To: <28ec79af-beb0-8ef9-ab6a-d608a632ee2c@huawei.com>
References: <20210623143250.82445-1-hare@suse.de>
 <8d7c79d9-0928-8c71-74fd-7185d67e7895@grimberg.me>
 <204a6656-ef83-b688-4979-26fe17f57974@suse.de>
 <cdf40226-12ed-5f87-8549-10b4defceae5@huawei.com>
 <91ff9961-6537-05d1-c552-18a02d13bf0c@suse.de>
 <984b2d1a-f319-1a09-7347-6c4edd1dbab6@huawei.com>
 <d20b23fc-dd08-cb51-eb29-aa033810b0a4@suse.de>
 <caa3759f-9a72-5e3b-9097-700132e4ab63@gmail.com>
 <28ec79af-beb0-8ef9-ab6a-d608a632ee2c@huawei.com>
Message-ID: <0af5901a-4dab-70de-abda-00ed7f8cf45f@suse.de>

On 6/28/21 3:10 AM, Chao Leng wrote:
> 
> 
> On 2021/6/27 21:39, James Smart wrote:
>> On 6/26/2021 5:09 AM, Hannes Reinecke wrote:
>>> On 6/26/21 3:03 AM, Chao Leng wrote:
>>>>
>>>>
>>>> On 2021/6/24 16:10, Hannes Reinecke wrote:
>>>>> On 6/24/21 9:29 AM, Chao Leng wrote:
>>>>>>
>>>>>>
>>>>>> On 2021/6/24 13:51, Hannes Reinecke wrote:
>>>>>>> On 6/23/21 11:38 PM, Sagi Grimberg wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> commit f25f8ef70ce2 ("nvme-fc: short-circuit reconnect retries")
>>>>>>>>> allowed the fc transport to honour the DNR bit during reconnect
>>>>>>>>> retries, allowing to speed up error recovery.
>>>>>>>>
>>>>>>>> How does this speed up error recovery?
>>>>>>>
>>>>>>> Well, not exactly error recovery (as there is nothing to recover).
>>>>>>> But we won't attempt pointless retries, thereby reducing the 
>>>>>>> noise in
>>>>>>> the message log.
>>>>>> This conflict with the tcp and rdma target.
>>>>>> You may need to delete the improper NVME_SC_DNR at the target.
>>>>>> However, this will cause compatibility issues between different 
>>>>>> versions.
>>>>>
>>>>> Which ones?
>>>> In many scenarios, the destination sets DNR for abnormal packets,
>>>> but each new connection may not have the same error.
>>>
>>> This patch series is only for the DNR bit set in response to the 
>>> 'connect' command.
>>> If the target is not able to process the 'connect' command, but may 
>>> be so in the future it really should not set the DNR bit.
>>>
>>>>> I checked the DNR usage in the target code, and they seem to set it
>>>>> correctly (ie the result would not change when the command is 
>>>>> retried).
>>>>> With the possible exception of ENOSPC handling, as this is arguably
>>>>> dynamic and might change with a retry.
>>>> The DNR status of the old connection may not be relevant to the 
>>>> re-established connection.
>>>
>>> See above.
>>> We are just checking the DNR settings for the 'connect' command (or 
>>> any other commands being sent during initial controller configuration).
>>> If that fails the connect never was properly initialized; if the 
>>> controller would return a different status after reconnect it simply 
>>> should not set the DNR bit ...
>>>
>>> Cheers,
>>>
>>> Hannes
>>
>> Agreed. Since 1.3 spec says: "If set to ?1?, indicates that if the 
>> same command is re-submitted to any controller in the NVM subsystem, 
>> then that re-submitted command is expected to fail."
> According to the definition of the protocol, this is not strictly 
> implemented on the target side.
> In nvme/target, there are many external errors, the DNR is set to 1.
> For example, abormal fabrics cmd.

Yes; but an abnormal fabrics cmd will be abnormal even after a reconnect.

>>
>> Thus if the initial connect fails in this manner, any new association 
>> will be on a different controller, where it is now expected connect on 
>> that controller will fail too.? Thus - why continue to connect when 
>> it's expected each will fail.
> Agree.
> We should not attempt to re-establish the connection if target can not 
> work due to target inner error .
> However, now? target does not behave exactly like this, so there are 
> conflicts and compatibility issues.

Note: we never said the target should _not_ send a DNR state. We only 
said the target should only set the DNR bit if a retry (even after 
reconnect or on a different controller) will not change the result.
Which from what I can see _is_ the case, so I do not see any issues here.

Please give details for the issues you are concerned with.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From hare at suse.de  Tue Jun 29 05:31:25 2021
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 29 Jun 2021 14:31:25 +0200
Subject: [PATCH 1/2] nvme-fc: Update hardware queues before using them
In-Reply-To: <20210625101649.49296-2-dwagner@suse.de>
References: <20210625101649.49296-1-dwagner@suse.de>
 <20210625101649.49296-2-dwagner@suse.de>
Message-ID: <11737f53-8959-3d83-6be8-feca7fa40be7@suse.de>

On 6/25/21 12:16 PM, Daniel Wagner wrote:
> In case the number of hardware queues changes, do the update the
> tagset and ctx to hctx first before using the mapping to recreate and
> connnect the IO queues.
> 
> Signed-off-by: Daniel Wagner <dwagner at suse.de>
> ---
>   drivers/nvme/host/fc.c | 16 ++++++++--------
>   1 file changed, 8 insertions(+), 8 deletions(-)
> 
Reviewed-by: Hannes Reinecke <hare at suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From hare at suse.de  Tue Jun 29 05:31:49 2021
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 29 Jun 2021 14:31:49 +0200
Subject: [PATCH 2/2] nvme-fc: Wait with a timeout for queue to freeze
In-Reply-To: <20210625101649.49296-3-dwagner@suse.de>
References: <20210625101649.49296-1-dwagner@suse.de>
 <20210625101649.49296-3-dwagner@suse.de>
Message-ID: <aef60901-b1d0-0499-211c-b87dc0745e2d@suse.de>

On 6/25/21 12:16 PM, Daniel Wagner wrote:
> Do not wait indifinitly for all queues to freeze. Instead use a
> timeout and abort the operation if we get stuck.
> 
> Signed-off-by: Daniel Wagner <dwagner at suse.de>
> ---
>   drivers/nvme/host/fc.c | 9 ++++++++-
>   1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index a9645cd89eca..d8db85aa5417 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -2955,7 +2955,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
>   		dev_info(ctrl->ctrl.device,
>   			"reconnect: revising io queue count from %d to %d\n",
>   			prior_ioq_cnt, nr_io_queues);
> -		nvme_wait_freeze(&ctrl->ctrl);
> +		if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
> +			/*
> +			 * If we timed out waiting for freeze we are likely to
> +			 * be stuck.  Fail the controller initialization just
> +			 * to be safe.
> +			 */
> +			return -ENODEV;
> +		}
>   		blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
>   		nvme_unfreeze(&ctrl->ctrl);
>   	}
> 
Reviewed-by: Hannes Reinecke <hare at suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From hare at suse.de  Tue Jun 29 05:39:14 2021
From: hare at suse.de (Hannes Reinecke)
Date: Tue, 29 Jun 2021 14:39:14 +0200
Subject: [PATCH 1/2] blk-mq: not deactivate hctx if the device doesn't use
 managed irq
In-Reply-To: <20210629074951.1981284-2-ming.lei@redhat.com>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <20210629074951.1981284-2-ming.lei@redhat.com>
Message-ID: <1a14a397-6244-928e-5aaa-85c2ccbe0e40@suse.de>

On 6/29/21 9:49 AM, Ming Lei wrote:
> hctx is deactivated when all CPU in hctx->cpumask become offline by
> draining all requests originated from this hctx and moving new
> allocation to active hctx. This way is for avoiding inflight IO when
> the managed irq is shutdown.
> 
> Some drivers(nvme fc, rdma, tcp, loop) doesn't use managed irq, so
> they needn't to deactivate hctx. Also, they are the only user of
> blk_mq_alloc_request_hctx() which is used for connecting io queue.
> And their requirement is that the connect request can be submitted
> via one specified hctx on which all CPU in its hctx->cpumask may have
> become offline.
> 

How can you submit a connect request for a hctx on which all CPUs are 
offline? That hctx will be unusable as it'll never be able to receive 
interrupts ...

> Address the requirement for nvme fc/rdma/loop, so the reported kernel
> panic on the following line in blk_mq_alloc_request_hctx() can be fixed.
> 
> 	data.ctx = __blk_mq_get_ctx(q, cpu)
> 
> Cc: Sagi Grimberg <sagi at grimberg.me>
> Cc: Daniel Wagner <dwagner at suse.de>
> Cc: Wen Xiong <wenxiong at us.ibm.com>
> Cc: John Garry <john.garry at huawei.com>
> Signed-off-by: Ming Lei <ming.lei at redhat.com>
> ---
>   block/blk-mq.c         | 6 +++++-
>   include/linux/blk-mq.h | 1 +
>   2 files changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index df5dc3b756f5..74632f50d969 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -494,7 +494,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
>   	data.hctx = q->queue_hw_ctx[hctx_idx];
>   	if (!blk_mq_hw_queue_mapped(data.hctx))
>   		goto out_queue_exit;
> -	cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
> +	cpu = cpumask_first(data.hctx->cpumask);
>   	data.ctx = __blk_mq_get_ctx(q, cpu);

I don't get it.
Doesn't this allow us to allocate a request on a dead cpu, ie the very 
thing we try to prevent?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From pkushwaha at marvell.com  Tue Jun 29 05:47:23 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:23 +0300
Subject: [PATCH v4 00/20] NVMeTCP Offload ULP 
Message-ID: <20210629124743.6898-1-pkushwaha@marvell.com>

With the goal of enabling a generic infrastructure that allows NVMe/TCP
offload devices like NICs to seamlessly plug into the NVMe-oF stack, this
patch series introduces the nvme-tcp-offload ULP host layer, which will
be a new transport type called "tcp-offload" and will serve as an 
abstraction layer to work with device specific nvme-tcp offload drivers.

NVMeTCP offload is a full offload of the NVMeTCP protocol, this includes 
both the TCP level and the NVMeTCP level.

The nvme-tcp-offload transport can co-exist with the existing tcp and 
other transports. The tcp offload was designed so that stack changes are 
kept to a bare minimum: only registering new transports.
All other APIs, ops etc. are identical to the regular tcp transport.
Representing the TCP offload as a new transport allows clear and manageable
differentiation between the connections which should use the offload path 
and those that are not offloaded (even on the same device).

The nvme-tcp-offload layers and API compared to nvme-tcp and nvme-rdma:

* NVMe layer: *

       [ nvme/nvme-fabrics/blk-mq ]
             |
        (nvme API and blk-mq API)
             |
             |			 
* Transport layer: *

      [ nvme-rdma ] [ nvme-tcp ] [ nvme-tcp-offload ]
             |        |             |
           (Verbs) 
             |        |             |
             |     (Socket)
             |        |             |
             |        |        (nvme-tcp-offload API)
             |        |             |
             |        |             |
* Transport Driver: *

             |        |             |
      [ RDMA driver ]       
                      |             |
             [ Network driver ]
                                    |
                       [ NVMeTCP Offload driver ]

Upstream plan:
==============
As discussed in RFCV7, "NVMeTCP Offload ULP and QEDN Device Driver" 
contains 3 parts:
https://lore.kernel.org/linux-nvme/20210531225222.16992-1-smalin at marvell.com/

This series contains part1 and part3, intended for linux-nvme:
- Part 1: The nvme-tcp-offload patches
- Part 3: Marvell's Offload device driver(qedn) patches. 
          It has "compilation dependency" on both Part 1 and Part 2. 

Part 2 is already accepted in net-next.git:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=eda1bc65b0dc1b03006e427430ba23746ec44714


Usage:
======
The user will interact with the network-device in order to configure 
the ip/vlan - Logically similar to the RDMA model.
The NVMeTCP configuration is populated as part of the 
nvme connect command.

Example:
Assign IP to the net-device (from any existing Linux tool):

    ip addr add 100.100.0.101/24 dev p1p1

This IP will be used by both net-device and offload-device.

In order to connect from "sw" nvme-tcp through the net-device:

    nvme connect -t tcp -s 4420 -a 100.100.0.100 -n testnqn

In order to connect from "offload" nvme-tcp through the offload-device:

    nvme connect -t tcp_offload -s 4420 -a 100.100.0.100 -n testnqn
	
An alternative approach, and as a future enhancement that will not impact this 
series will be to modify nvme-cli with a new flag that will determine 
if "-t tcp" should be the regular nvme-tcp (which will be the default) 
or nvme-tcp-offload.
Exmaple:
    nvme connect -t tcp -s 4420 -a 100.100.0.100 -n testnqn -[new flag]


Queue Initialization Design:
============================
The nvme-tcp-offload ULP module shall register with the existing 
nvmf_transport_ops (.name = "tcp_offload"), nvme_ctrl_ops and blk_mq_ops.
The nvme-tcp-offload driver shall register to nvme-tcp-offload ULP
with the following ops:
- claim_dev() - in order to resolve the route to the target according to
                the paired net_dev.
- create_queue() - in order to create offloaded nvme-tcp queue.

The nvme-tcp-offload ULP module shall manage all the controller level
functionalities, call claim_dev and based on the return values shall call
the relevant module create_queue in order to create the admin queue and
the IO queues.


IO-path Design:
===============
The nvme-tcp-offload shall work at the IO-level - the nvme-tcp-offload 
ULP module shall pass the request (the IO) to the nvme-tcp-offload 
driver and later, the nvme-tcp-offload driver returns the request
completion (the IO completion).
No additional handling is needed in between; this design will reduce the
CPU utilization as we will describe below.

The nvme-tcp-offload driver shall register to nvme-tcp-offload ULP
with the following IO-path ops:
- send_req() - in order to pass the request to the handling of the
               offload driver that shall pass it to the offload device 
			   specific device.
- poll_queue()

Once the IO completes, the nvme-tcp-offload driver shall call 
command.done() that will invoke the nvme-tcp-offload ULP layer to
complete the request.


Teardown and errors:
====================
In case of NVMeTCP queue error the nvme-tcp-offload driver shall
call the nvme_tcp_ofld_report_queue_err.
The nvme-tcp-offload driver shall register to nvme-tcp-offload ULP
with the following teardown ops:
- drain_queue()
- destroy_queue()


The Marvell qedn driver:
========================
The new driver will be added under "drivers/nvme/hw" and will be enabled
by the Kconfig "Marvell NVM Express over Fabrics TCP offload".
As part of the qedn init, the driver will register as a pci device driver 
and will work with the Marvell fastlinQ NIC.
As part of the probe, the driver will register to the nvme_tcp_offload
(ULP) and to the qed module (qed_nvmetcp_ops) - similar to other
"qed_*_ops" which are used by the qede, qedr, qedf and qedi device
drivers.


Changes since RFC v1:
=====================
- nvme-tcp-offload: Fix nvme_tcp_ofld_ops return values.
- nvme-tcp-offload: Remove NVMF_TRTYPE_TCP_OFFLOAD.
- nvme-tcp-offload: Add nvme_tcp_ofld_poll() implementation.
- nvme-tcp-offload: Fix nvme_tcp_ofld_queue_rq() to check map_sg() and 
  send_req() return values.

Changes since RFC v2:
=====================
- nvme-tcp-offload: Fixes in controller and queue level (patches 3-6).
- qedn: Add the Marvell's NVMeTCP HW offload device driver init and probe
  (patches 8-11).
  
Changes since RFC v3:
=====================
- nvme-tcp-offload: Add the full implementation of the nvme-tcp-offload layer 
  including the new ops: setup_ctrl(), release_ctrl(), commit_rqs() and new 
  flows (ASYNC and timeout).
- nvme-tcp-offload: Add device maximums: max_hw_sectors, max_segments.
- nvme-tcp-offload: layer design and optimization changes.
- qedn: Add full implementation for the conn level, IO path and error handling.

Changes since RFC v4:
=====================
(Many thanks to Hannes Reinecke for his feedback)
- nvme_tcp_offload: Add num_hw_vectors in order to limit the number of queues.
- nvme_tcp_offload: Add per device private_data.
- nvme_tcp_offload: Fix header digest, data digest and tos initialization.
- qedn: Remove the qedn_global list.
- qedn: Remove the workqueue flow from send_req.
- qedn: Add db_recovery support.

Changes since RFC v5:
=====================
(Many thanks to Sagi Grimberg for his feedback)
- nvme-fabrics: Expose nvmf_check_required_opts() globally (as a new patch).
- nvme_tcp_offload: Remove io-queues BLK_MQ_F_BLOCKING.
- nvme_tcp_offload: Fix the nvme_tcp_ofld_stop_queue (drain_queue) flow.
- nvme_tcp_offload: Fix the nvme_tcp_ofld_free_queue (destroy_queue) flow.
- nvme_tcp_offload: Change rwsem to mutex.
- nvme_tcp_offload: remove redundant fields.
- nvme_tcp_offload: Remove the "new" from setup_ctrl().
- nvme_tcp_offload: Remove the init_req() and commit_rqs() ops.
- nvme_tcp_offload: Minor fixes in nvme_tcp_ofld_create_ctrl() ansd 
  nvme_tcp_ofld_free_queue().
- nvme_tcp_offload: Patch 8 (timeout and async) was squeashed into 
  patch 7 (io level).
- qedn: Fix the free_queue flow and the destroy_queue flow.
- qedn: Remove version number.

Changes since RFC v6:
=====================
- No changes in nvme_tcp_offload
- qedn: Remove redundant logic in the io-queues core affinity initialization.
- qedn: Remove qedn_validate_cccid_in_range().

Changes since v1:
=====================
- nvme_tcp_offload: Add support for NVME_OPT_HOST_IFACE.
- nvme_tcp_offload: Kconfig fix (thanks to Petr Mladek).
- nvme_tcp_offload: return code fix (thanks to Dan Carpenter).

Changes since v2:
=====================
- nvme_tcp_offload: Fix overly long lines.
- nvme_tcp_offload: use correct terminology for vendor driver.
- qedn: Added qedn driver as part of series.

Changes since v3:
=====================
- nvme_tcp_offload: Rename nvme_tcp_ofld_map_data() to 
  nvme_tcp_ofld_set_sg_host_data().


Arie Gershberg (2):
  nvme-tcp-offload: Add controller level implementation
  nvme-tcp-offload: Add controller level error recovery implementation

Dean Balandin (3):
  nvme-tcp-offload: Add device scan implementation
  nvme-tcp-offload: Add queue level implementation
  nvme-tcp-offload: Add IO level implementation

Nikolay Assa (1):
  qedn: Add qedn_claim_dev API support

Prabhakar Kushwaha (7):
  nvme-fabrics: Move NVMF_ALLOWED_OPTS and NVMF_REQUIRED_OPTS
    definitions
  nvme-fabrics: Expose nvmf_check_required_opts() globally
  qedn: Add connection-level slowpath functionality
  qedn: Add support of configuring HW filter block
  qedn: Add support of Task and SGL
  qedn: Add support of NVME ICReq & ICResp
  qedn: Add support of ASYNC

Shai Malin (7):
  nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW offload ULP
  qedn: Add qedn - Marvell's NVMeTCP HW offload device driver
  qedn: Add qedn probe
  qedn: Add IRQ and fast-path resources initializations
  qedn: Add IO level qedn_send_req and fw_cq workqueue
  qedn: Add IO level fastpath functionality
  qedn: Add Connection and IO level recovery flows

 MAINTAINERS                      |   18 +
 drivers/nvme/Kconfig             |    1 +
 drivers/nvme/Makefile            |    1 +
 drivers/nvme/host/Kconfig        |   15 +
 drivers/nvme/host/Makefile       |    3 +
 drivers/nvme/host/fabrics.c      |   12 +-
 drivers/nvme/host/fabrics.h      |    9 +
 drivers/nvme/host/tcp-offload.c  | 1346 ++++++++++++++++++++++++++++++
 drivers/nvme/host/tcp-offload.h  |  207 +++++
 drivers/nvme/hw/Kconfig          |    9 +
 drivers/nvme/hw/Makefile         |    3 +
 drivers/nvme/hw/qedn/Makefile    |    4 +
 drivers/nvme/hw/qedn/qedn.h      |  402 +++++++++
 drivers/nvme/hw/qedn/qedn_conn.c | 1076 ++++++++++++++++++++++++
 drivers/nvme/hw/qedn/qedn_main.c | 1109 ++++++++++++++++++++++++
 drivers/nvme/hw/qedn/qedn_task.c |  873 +++++++++++++++++++
 16 files changed, 5079 insertions(+), 9 deletions(-)
 create mode 100644 drivers/nvme/host/tcp-offload.c
 create mode 100644 drivers/nvme/host/tcp-offload.h
 create mode 100644 drivers/nvme/hw/Kconfig
 create mode 100644 drivers/nvme/hw/Makefile
 create mode 100644 drivers/nvme/hw/qedn/Makefile
 create mode 100644 drivers/nvme/hw/qedn/qedn.h
 create mode 100644 drivers/nvme/hw/qedn/qedn_conn.c
 create mode 100644 drivers/nvme/hw/qedn/qedn_main.c
 create mode 100644 drivers/nvme/hw/qedn/qedn_task.c

-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:24 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:24 +0300
Subject: [PATCH v4 01/20] nvme-tcp-offload: Add nvme-tcp-offload - NVMeTCP HW
 offload ULP
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-2-pkushwaha@marvell.com>

From: Shai Malin <smalin at marvell.com>

This patch will present the structure for the NVMeTCP offload common
layer driver. This module is added under "drivers/nvme/host/" and future
offload drivers which will register to it will be placed under
"drivers/nvme/hw".
This new driver will be enabled by the Kconfig "NVM Express over Fabrics
TCP offload commmon layer".
In order to support the new transport type, for host mode, no change is
needed.

Each new offload device specific driver will register to this ULP during
its probe function, by filling out the nvme_tcp_ofld_dev->ops and
nvme_tcp_ofld_dev->private_data and calling nvme_tcp_ofld_register_dev
with the initialized struct.

The internal implementation:
- tcp-offload.h:
  Includes all common structs and ops to be used and shared by offload
  drivers.

- tcp-offload.c:
  Includes the init function which registers as a NVMf transport just
  like any other transport.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 MAINTAINERS                     |   8 ++
 drivers/nvme/host/Kconfig       |  15 +++
 drivers/nvme/host/Makefile      |   3 +
 drivers/nvme/host/tcp-offload.c | 124 ++++++++++++++++++++
 drivers/nvme/host/tcp-offload.h | 199 ++++++++++++++++++++++++++++++++
 5 files changed, 349 insertions(+)
 create mode 100644 drivers/nvme/host/tcp-offload.c
 create mode 100644 drivers/nvme/host/tcp-offload.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 81e1edeceae4..01fbebdc7722 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13093,6 +13093,14 @@ F:	drivers/nvme/host/
 F:	include/linux/nvme.h
 F:	include/uapi/linux/nvme_ioctl.h
 
+NVM EXPRESS TCP OFFLOAD TRANSPORT DRIVERS
+M:	Shai Malin <smalin at marvell.com>
+M:	Ariel Elior <aelior at marvell.com>
+L:	linux-nvme at lists.infradead.org
+S:	Supported
+F:	drivers/nvme/host/tcp-offload.c
+F:	drivers/nvme/host/tcp-offload.h
+
 NVM EXPRESS FC TRANSPORT DRIVERS
 M:	James Smart <james.smart at broadcom.com>
 L:	linux-nvme at lists.infradead.org
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 102292289cdf..1993734d0104 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -84,3 +84,18 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_TCP_OFFLOAD
+	tristate "NVM Express over Fabrics TCP offload common layer"
+	depends on INET
+	depends on BLK_DEV_NVME
+	select NVME_FABRICS
+	help
+	  This provides support for the NVMe over Fabrics protocol using
+	  the TCP offload transport. This allows you to use remote block devices
+	  exported using the NVMe protocol set.
+
+	  To configure a NVMe over Fabrics controller use the nvme-cli tool
+	  from https://github.com/linux-nvme/nvme-cli.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index cbc509784b2e..3c3fdf83ce38 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -8,6 +8,7 @@ obj-$(CONFIG_NVME_FABRICS)		+= nvme-fabrics.o
 obj-$(CONFIG_NVME_RDMA)			+= nvme-rdma.o
 obj-$(CONFIG_NVME_FC)			+= nvme-fc.o
 obj-$(CONFIG_NVME_TCP)			+= nvme-tcp.o
+obj-$(CONFIG_NVME_TCP_OFFLOAD)	+= nvme-tcp-offload.o
 
 nvme-core-y				:= core.o ioctl.o
 nvme-core-$(CONFIG_TRACING)		+= trace.o
@@ -26,3 +27,5 @@ nvme-rdma-y				+= rdma.o
 nvme-fc-y				+= fc.o
 
 nvme-tcp-y				+= tcp.o
+
+nvme-tcp-offload-y		+= tcp-offload.o
diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
new file mode 100644
index 000000000000..10b87f5b875b
--- /dev/null
+++ b/drivers/nvme/host/tcp-offload.c
@@ -0,0 +1,124 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+/* Kernel includes */
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+/* Driver includes */
+#include "tcp-offload.h"
+
+static LIST_HEAD(nvme_tcp_ofld_devices);
+static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
+
+/**
+ * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
+ * function.
+ * @dev:	NVMeTCP offload device instance to be registered to the
+ *		common tcp offload instance.
+ *
+ * API function that registers the type of offload device specific driver
+ * being implemented to the common NVMe over TCP offload library. Part of
+ * the overall init sequence of starting up an offload driver.
+ */
+int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev)
+{
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+
+	if (!ops->claim_dev ||
+	    !ops->setup_ctrl ||
+	    !ops->release_ctrl ||
+	    !ops->create_queue ||
+	    !ops->drain_queue ||
+	    !ops->destroy_queue ||
+	    !ops->poll_queue ||
+	    !ops->send_req)
+		return -EINVAL;
+
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_add_tail(&dev->entry, &nvme_tcp_ofld_devices);
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_register_dev);
+
+/**
+ * nvme_tcp_ofld_unregister_dev() - NVMeTCP Offload Library unregistration
+ * function.
+ * @dev:	NVMeTCP offload device instance to be unregistered from the
+ *		common tcp offload instance.
+ *
+ * API function that unregisters the type of offload device specific driver
+ * being implemented from the common NVMe over TCP offload library.
+ * Part of the overall exit sequence of unloading the implemented driver.
+ */
+void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev)
+{
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_del(&dev->entry);
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
+
+/**
+ * nvme_tcp_ofld_report_queue_err() - NVMeTCP Offload report error event
+ * callback function. Pointed to by nvme_tcp_ofld_queue->report_err.
+ * @queue:	NVMeTCP offload queue instance on which the error has occurred.
+ *
+ * API function that allows the offload device specific driver to reports
+ * errors to the common offload layer, to invoke error recovery.
+ */
+int nvme_tcp_ofld_report_queue_err(struct nvme_tcp_ofld_queue *queue)
+{
+	/* Placeholder - invoke error recovery flow */
+
+	return 0;
+}
+
+/**
+ * nvme_tcp_ofld_req_done() - NVMeTCP Offload request done callback
+ * function. Pointed to by nvme_tcp_ofld_req->done.
+ * Handles both NVME_TCP_F_DATA_SUCCESS flag and NVMe CQ.
+ * @req:	NVMeTCP offload request to complete.
+ * @result:     The nvme_result.
+ * @status:     The completion status.
+ *
+ * API function that allows the offload device specific driver to report
+ * request completions to the common offload layer.
+ */
+void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
+			    union nvme_result *result,
+			    __le16 status)
+{
+	/* Placeholder - complete request with/without error */
+}
+
+static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
+	.name		= "tcp_offload",
+	.module		= THIS_MODULE,
+	.required_opts	= NVMF_OPT_TRADDR,
+	.allowed_opts	= NVMF_OPT_TRSVCID | NVMF_OPT_NR_WRITE_QUEUES  |
+			  NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO |
+			  NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HDR_DIGEST |
+			  NVMF_OPT_DATA_DIGEST | NVMF_OPT_NR_POLL_QUEUES |
+			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE,
+};
+
+static int __init nvme_tcp_ofld_init_module(void)
+{
+	nvmf_register_transport(&nvme_tcp_ofld_transport);
+
+	return 0;
+}
+
+static void __exit nvme_tcp_ofld_cleanup_module(void)
+{
+	nvmf_unregister_transport(&nvme_tcp_ofld_transport);
+}
+
+module_init(nvme_tcp_ofld_init_module);
+module_exit(nvme_tcp_ofld_cleanup_module);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
new file mode 100644
index 000000000000..bfa759177a07
--- /dev/null
+++ b/drivers/nvme/host/tcp-offload.h
@@ -0,0 +1,199 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+/* Linux includes */
+#include <linux/dma-mapping.h>
+#include <linux/scatterlist.h>
+#include <linux/types.h>
+#include <linux/nvme-tcp.h>
+
+/* Driver includes */
+#include "nvme.h"
+#include "fabrics.h"
+
+/* Forward declarations */
+struct nvme_tcp_ofld_ops;
+
+/* Representation of a offload device. This is the struct used to register
+ * to the offload layer by the offload device specific driver, during its probe
+ * function.
+ * Allocated by offload device specific driver.
+ */
+struct nvme_tcp_ofld_dev {
+	struct list_head entry;
+	struct nvme_tcp_ofld_ops *ops;
+
+	/* Offload device specific driver context */
+	int num_hw_vectors;
+};
+
+/* Per IO struct holding the nvme_request and command
+ * Allocated by blk-mq.
+ */
+struct nvme_tcp_ofld_req {
+	struct nvme_request req;
+	struct nvme_command nvme_cmd;
+	struct list_head queue_entry;
+	struct nvme_tcp_ofld_queue *queue;
+
+	/* Offload device specific driver context */
+	void *private_data;
+
+	/* async flag is used to distinguish between async and IO flow
+	 * in common send_req() of nvme_tcp_ofld_ops.
+	 */
+	bool async;
+
+	void (*done)(struct nvme_tcp_ofld_req *req,
+		     union nvme_result *result,
+		     __le16 status);
+};
+
+enum nvme_tcp_ofld_queue_flags {
+	NVME_TCP_OFLD_Q_ALLOCATED = 0,
+	NVME_TCP_OFLD_Q_LIVE = 1,
+};
+
+/* Allocated by nvme_tcp_ofld */
+struct nvme_tcp_ofld_queue {
+	/* Offload device associated to this queue */
+	struct nvme_tcp_ofld_dev *dev;
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	unsigned long flags;
+	size_t cmnd_capsule_len;
+
+	u8 hdr_digest;
+	u8 data_digest;
+	u8 tos;
+
+	/* Offload device specific driver context */
+	void *private_data;
+
+	/* Error callback function */
+	int (*report_err)(struct nvme_tcp_ofld_queue *queue);
+};
+
+/* Connectivity (routing) params used for establishing a connection */
+struct nvme_tcp_ofld_ctrl_con_params {
+	struct sockaddr_storage remote_ip_addr;
+
+	/* If NVMF_OPT_HOST_TRADDR is provided it will be set in local_ip_addr
+	 * in nvme_tcp_ofld_create_ctrl().
+	 * If NVMF_OPT_HOST_TRADDR is not provided the local_ip_addr will be
+	 * initialized by claim_dev().
+	 */
+	struct sockaddr_storage local_ip_addr;
+};
+
+/* Allocated by nvme_tcp_ofld */
+struct nvme_tcp_ofld_ctrl {
+	struct nvme_ctrl nctrl;
+	struct list_head list;
+	struct net_device *ndev;
+	struct nvme_tcp_ofld_dev *dev;
+
+	/* admin and IO queues */
+	struct blk_mq_tag_set tag_set;
+	struct blk_mq_tag_set admin_tag_set;
+	struct nvme_tcp_ofld_queue *queues;
+
+	struct work_struct err_work;
+	struct delayed_work connect_work;
+
+	/*
+	 * Each entry in the array indicates the number of queues of
+	 * corresponding type.
+	 */
+	u32 io_queues[HCTX_MAX_TYPES];
+
+	/* Connectivity params */
+	struct nvme_tcp_ofld_ctrl_con_params conn_params;
+
+	/* Offload device driver context */
+	void *private_data;
+};
+
+struct nvme_tcp_ofld_ops {
+	const char *name;
+	struct module *module;
+
+	/* For offload device specific driver to report what opts it supports.
+	 * It could be different than the ULP supported opts due to hardware
+	 * limitations. Also it could be different among different offload
+	 * specific device drivers.
+	 */
+	int required_opts; /* bitmap using enum nvmf_parsing_opts */
+	int allowed_opts; /* bitmap using enum nvmf_parsing_opts */
+
+	/* For offload device specific max num of segments and IO sizes */
+	u32 max_hw_sectors;
+	u32 max_segments;
+
+	/**
+	 * claim_dev: Return True if addr is reachable via offload device.
+	 * @dev: The offload device to check.
+	 * @ctrl: The offload ctrl have the conn_params field. The
+	 * conn_params is to be filled with routing params by the
+	 * offload device specific driver.
+	 */
+	int (*claim_dev)(struct nvme_tcp_ofld_dev *dev,
+			 struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * setup_ctrl: Setup device specific controller structures.
+	 * @ctrl: The offload ctrl.
+	 */
+	int (*setup_ctrl)(struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * release_ctrl: Release/Free device specific controller structures.
+	 * @ctrl: The offload ctrl.
+	 */
+	int (*release_ctrl)(struct nvme_tcp_ofld_ctrl *ctrl);
+
+	/**
+	 * create_queue: Create offload queue and establish TCP + NVMeTCP
+	 * (icreq+icresp) connection. Return true on successful connection.
+	 * Based on nvme_tcp_alloc_queue.
+	 * @queue: The queue itself - used as input and output.
+	 * @qid: The queue ID associated with the requested queue.
+	 * @q_size: The queue depth.
+	 */
+	int (*create_queue)(struct nvme_tcp_ofld_queue *queue, int qid,
+			    size_t queue_size);
+
+	/**
+	 * drain_queue: Drain a given queue - blocking function call.
+	 * Return from this function ensures that no additional
+	 * completions will arrive on this queue and that the HW will
+	 * not access host memory.
+	 * @queue: The queue to drain.
+	 */
+	void (*drain_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * destroy_queue: Close the TCP + NVMeTCP connection of a given queue
+	 * and make sure its no longer active (no completions will arrive on the
+	 * queue).
+	 * @queue: The queue to destroy.
+	 */
+	void (*destroy_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * poll_queue: Poll a given queue for completions.
+	 * @queue: The queue to poll.
+	 */
+	int (*poll_queue)(struct nvme_tcp_ofld_queue *queue);
+
+	/**
+	 * send_req: Dispatch a request. Returns the execution status.
+	 * @req: Ptr to request to be sent.
+	 */
+	int (*send_req)(struct nvme_tcp_ofld_req *req);
+};
+
+/* Exported functions for offload device specific drivers */
+int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
+void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:26 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:26 +0300
Subject: [PATCH v4 03/20] nvme-fabrics: Expose nvmf_check_required_opts()
 globally
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-4-pkushwaha@marvell.com>

nvmf_check_required_opts() is used to check if user provided opts has
the required_opts or not. if not, it will log which options are not
provided.

It can be leveraged by nvme-tcp-offload to check if provided opts are
supported by this specific offload driver or not.

So expose nvmf_check_required_opts() globally.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/fabrics.c | 5 +++--
 drivers/nvme/host/fabrics.h | 2 ++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 21d67775d2af..7830d2540492 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -864,8 +864,8 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	return ret;
 }
 
-static int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
-		unsigned int required_opts)
+int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
+			     unsigned int required_opts)
 {
 	if ((opts->mask & required_opts) != required_opts) {
 		int i;
@@ -883,6 +883,7 @@ static int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(nvmf_check_required_opts);
 
 bool nvmf_ip_options_match(struct nvme_ctrl *ctrl,
 		struct nvmf_ctrl_options *opts)
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 38ac7b757d78..15d9c15ef8a6 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -197,5 +197,7 @@ int nvmf_get_address(struct nvme_ctrl *ctrl, char *buf, int size);
 bool nvmf_should_reconnect(struct nvme_ctrl *ctrl);
 bool nvmf_ip_options_match(struct nvme_ctrl *ctrl,
 		struct nvmf_ctrl_options *opts);
+int nvmf_check_required_opts(struct nvmf_ctrl_options *opts,
+			     unsigned int required_opts);
 
 #endif /* _NVME_FABRICS_H */
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:25 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:25 +0300
Subject: [PATCH v4 02/20] nvme-fabrics: Move NVMF_ALLOWED_OPTS and
 NVMF_REQUIRED_OPTS definitions
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-3-pkushwaha@marvell.com>

Move NVMF_ALLOWED_OPTS and NVMF_REQUIRED_OPTS definitions
to header file, so it can be used by the different HW devices.

NVMeTCP offload devices might have different limitations of the
allowed options, for example, a device that does not support all the
queue types. With tcp and rdma, only the nvme-tcp and nvme-rdma layers
handle those attributes and the HW devices do not create any limitations
for the allowed options.

An alternative design could be to add separate fields in
nvme_tcp_ofld_ops such as max_hw_sectors and max_segments that
we already have in this series.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Acked-by: Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/fabrics.c | 7 -------
 drivers/nvme/host/fabrics.h | 7 +++++++
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 1e6a7cc056ca..21d67775d2af 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -947,13 +947,6 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
 
-#define NVMF_REQUIRED_OPTS	(NVMF_OPT_TRANSPORT | NVMF_OPT_NQN)
-#define NVMF_ALLOWED_OPTS	(NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \
-				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
-				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
-				 NVMF_OPT_DISABLE_SQFLOW |\
-				 NVMF_OPT_FAIL_FAST_TMO)
-
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
 {
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index c31dad69a773..38ac7b757d78 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -69,6 +69,13 @@ enum {
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 };
 
+#define NVMF_REQUIRED_OPTS	(NVMF_OPT_TRANSPORT | NVMF_OPT_NQN)
+#define NVMF_ALLOWED_OPTS	(NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \
+				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
+				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
+				 NVMF_OPT_DISABLE_SQFLOW |\
+				 NVMF_OPT_FAIL_FAST_TMO)
+
 /**
  * struct nvmf_ctrl_options - Used to hold the options specified
  *			      with the parsing opts enum.
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:27 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:27 +0300
Subject: [PATCH v4 04/20] nvme-tcp-offload: Add device scan implementation
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-5-pkushwaha@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

As part of create_ctrl(), it scans the registered devices and calls
the claim_dev op on each of them, to find the first devices that matches
the connection params. Once the correct devices is found (claim_dev
returns true), we raise the refcnt of that device and return that device
as the device to be used for ctrl currently being created.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/tcp-offload.c | 91 +++++++++++++++++++++++++++++++++
 drivers/nvme/host/tcp-offload.h |  1 +
 2 files changed, 92 insertions(+)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 10b87f5b875b..d0f4b83549b9 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -13,6 +13,11 @@
 static LIST_HEAD(nvme_tcp_ofld_devices);
 static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
 
+static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nc)
+{
+	return container_of(nc, struct nvme_tcp_ofld_ctrl, nctrl);
+}
+
 /**
  * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
  * function.
@@ -96,6 +101,91 @@ void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
 	/* Placeholder - complete request with/without error */
 }
 
+static struct nvme_tcp_ofld_dev *
+nvme_tcp_ofld_lookup_dev(struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+	struct nvme_tcp_ofld_dev *dev;
+	char *iface;
+
+	if (nctrl->opts->mask & NVMF_OPT_HOST_IFACE) {
+		iface = nctrl->opts->host_iface;
+		ctrl->ndev = __dev_get_by_name(&init_net, iface);
+		if (!ctrl->ndev) {
+			pr_err("invalid interface passed: %s\n", iface);
+			return NULL;
+		}
+	}
+
+	mutex_lock(&nvme_tcp_ofld_devices_mutex);
+	list_for_each_entry(dev, &nvme_tcp_ofld_devices, entry) {
+	/* ctrl includes the destination ip, source ip (if provided) and
+	 * network interface (if provided).
+	 */
+		if (dev->ops->claim_dev(dev, ctrl))
+			goto out;
+	}
+
+	dev = NULL;
+out:
+	mutex_unlock(&nvme_tcp_ofld_devices_mutex);
+
+	return dev;
+}
+
+static struct nvme_ctrl *
+nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	struct nvme_tcp_ofld_dev *dev;
+	struct nvme_ctrl *nctrl;
+	int rc = 0;
+
+	ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
+	if (!ctrl)
+		return ERR_PTR(-ENOMEM);
+
+	nctrl = &ctrl->nctrl;
+
+	/* Init nvme_tcp_ofld_ctrl & nvme_ctrl params based on received opts */
+
+	/* Find device that can reach the dest addr */
+	dev = nvme_tcp_ofld_lookup_dev(ctrl);
+	if (!dev) {
+		pr_info("no device found for addr %s:%s.\n",
+			opts->traddr, opts->trsvcid);
+		rc = -EINVAL;
+		goto out_free_ctrl;
+	}
+
+	/* Increase driver refcnt */
+	if (!try_module_get(dev->ops->module)) {
+		pr_err("try_module_get failed\n");
+		rc = -ENODEV;
+		goto out_free_ctrl;
+	}
+
+	ctrl->dev = dev;
+
+	if (ctrl->dev->ops->max_hw_sectors)
+		nctrl->max_hw_sectors = ctrl->dev->ops->max_hw_sectors;
+	if (ctrl->dev->ops->max_segments)
+		nctrl->max_segments = ctrl->dev->ops->max_segments;
+
+	/* Init queues */
+
+	/* Call nvme_init_ctrl */
+
+	/* Setup ctrl */
+
+	return nctrl;
+
+out_free_ctrl:
+	kfree(ctrl);
+
+	return ERR_PTR(rc);
+}
+
 static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
 	.name		= "tcp_offload",
 	.module		= THIS_MODULE,
@@ -105,6 +195,7 @@ static struct nvmf_transport_ops nvme_tcp_ofld_transport = {
 			  NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HDR_DIGEST |
 			  NVMF_OPT_DATA_DIGEST | NVMF_OPT_NR_POLL_QUEUES |
 			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE,
+	.create_ctrl	= nvme_tcp_ofld_create_ctrl,
 };
 
 static int __init nvme_tcp_ofld_init_module(void)
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index bfa759177a07..d1c2c6171897 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -8,6 +8,7 @@
 #include <linux/scatterlist.h>
 #include <linux/types.h>
 #include <linux/nvme-tcp.h>
+#include <linux/netdevice.h>
 
 /* Driver includes */
 #include "nvme.h"
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:28 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:28 +0300
Subject: [PATCH v4 05/20] nvme-tcp-offload: Add controller level implementation
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-6-pkushwaha@marvell.com>

From: Arie Gershberg <agershberg at marvell.com>

In this patch we implement controller level functionality including:
- create_ctrl.
- delete_ctrl.
- free_ctrl.

The implementation is similar to other nvme fabrics modules, the main
difference being that the nvme-tcp-offload ULP calls the offload specific
claim_dev() op with the given TCP/IP parameters to determine which device
will be used for this controller.
Once found, the offload specific device and controller will be paired and
kept in a controller list managed by the ULP.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/tcp-offload.c | 487 +++++++++++++++++++++++++++++++-
 1 file changed, 482 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index d0f4b83549b9..806247722a35 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -12,6 +12,10 @@
 
 static LIST_HEAD(nvme_tcp_ofld_devices);
 static DEFINE_MUTEX(nvme_tcp_ofld_devices_mutex);
+static LIST_HEAD(nvme_tcp_ofld_ctrl_list);
+static DEFINE_MUTEX(nvme_tcp_ofld_ctrl_mutex);
+static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops;
+static struct blk_mq_ops nvme_tcp_ofld_mq_ops;
 
 static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nc)
 {
@@ -133,21 +137,441 @@ nvme_tcp_ofld_lookup_dev(struct nvme_tcp_ofld_ctrl *ctrl)
 	return dev;
 }
 
+static struct blk_mq_tag_set *
+nvme_tcp_ofld_alloc_tagset(struct nvme_ctrl *nctrl, bool admin)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct blk_mq_tag_set *set;
+	int rc;
+
+	if (admin) {
+		set = &ctrl->admin_tag_set;
+		memset(set, 0, sizeof(*set));
+		set->ops = &nvme_tcp_ofld_admin_mq_ops;
+		set->queue_depth = NVME_AQ_MQ_TAG_DEPTH;
+		set->reserved_tags = NVMF_RESERVED_TAGS;
+		set->numa_node = nctrl->numa_node;
+		set->flags = BLK_MQ_F_BLOCKING;
+		set->cmd_size = sizeof(struct nvme_tcp_ofld_req);
+		set->driver_data = ctrl;
+		set->nr_hw_queues = 1;
+		set->timeout = NVME_ADMIN_TIMEOUT;
+	} else {
+		set = &ctrl->tag_set;
+		memset(set, 0, sizeof(*set));
+		set->ops = &nvme_tcp_ofld_mq_ops;
+		set->queue_depth = nctrl->sqsize + 1;
+		set->reserved_tags = NVMF_RESERVED_TAGS;
+		set->numa_node = nctrl->numa_node;
+		set->flags = BLK_MQ_F_SHOULD_MERGE;
+		set->cmd_size = sizeof(struct nvme_tcp_ofld_req);
+		set->driver_data = ctrl;
+		set->nr_hw_queues = nctrl->queue_count - 1;
+		set->timeout = NVME_IO_TIMEOUT;
+		set->nr_maps = nctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2;
+	}
+
+	rc = blk_mq_alloc_tag_set(set);
+	if (rc)
+		return ERR_PTR(rc);
+
+	return set;
+}
+
+static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
+					       bool new)
+{
+	int rc;
+
+	/* Placeholder - alloc_admin_queue */
+	if (new) {
+		nctrl->admin_tagset =
+				nvme_tcp_ofld_alloc_tagset(nctrl, true);
+		if (IS_ERR(nctrl->admin_tagset)) {
+			rc = PTR_ERR(nctrl->admin_tagset);
+			nctrl->admin_tagset = NULL;
+			goto out_destroy_queue;
+		}
+
+		nctrl->fabrics_q = blk_mq_init_queue(nctrl->admin_tagset);
+		if (IS_ERR(nctrl->fabrics_q)) {
+			rc = PTR_ERR(nctrl->fabrics_q);
+			nctrl->fabrics_q = NULL;
+			goto out_free_tagset;
+		}
+
+		nctrl->admin_q = blk_mq_init_queue(nctrl->admin_tagset);
+		if (IS_ERR(nctrl->admin_q)) {
+			rc = PTR_ERR(nctrl->admin_q);
+			nctrl->admin_q = NULL;
+			goto out_cleanup_fabrics_q;
+		}
+	}
+
+	/* Placeholder - nvme_tcp_ofld_start_queue */
+
+	rc = nvme_enable_ctrl(nctrl);
+	if (rc)
+		goto out_stop_queue;
+
+	blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	rc = nvme_init_ctrl_finish(nctrl);
+	if (rc)
+		goto out_quiesce_queue;
+
+	return 0;
+
+out_quiesce_queue:
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+
+out_stop_queue:
+	/* Placeholder - stop offload queue */
+	nvme_cancel_admin_tagset(nctrl);
+
+out_cleanup_fabrics_q:
+	if (new)
+		blk_cleanup_queue(nctrl->fabrics_q);
+out_free_tagset:
+	if (new)
+		blk_mq_free_tag_set(nctrl->admin_tagset);
+out_destroy_queue:
+	/* Placeholder - free admin queue */
+
+	return rc;
+}
+
+static int
+nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
+{
+	int rc;
+
+	/* Placeholder - alloc_io_queues */
+
+	if (new) {
+		nctrl->tagset = nvme_tcp_ofld_alloc_tagset(nctrl, false);
+		if (IS_ERR(nctrl->tagset)) {
+			rc = PTR_ERR(nctrl->tagset);
+			nctrl->tagset = NULL;
+			goto out_free_io_queues;
+		}
+
+		nctrl->connect_q = blk_mq_init_queue(nctrl->tagset);
+		if (IS_ERR(nctrl->connect_q)) {
+			rc = PTR_ERR(nctrl->connect_q);
+			nctrl->connect_q = NULL;
+			goto out_free_tag_set;
+		}
+	}
+
+	/* Placeholder - start_io_queues */
+
+	if (!new) {
+		nvme_start_queues(nctrl);
+		if (!nvme_wait_freeze_timeout(nctrl, NVME_IO_TIMEOUT)) {
+			/*
+			 * If we timed out waiting for freeze we are likely to
+			 * be stuck.  Fail the controller initialization just
+			 * to be safe.
+			 */
+			rc = -ENODEV;
+			goto out_wait_freeze_timed_out;
+		}
+		blk_mq_update_nr_hw_queues(nctrl->tagset,
+					   nctrl->queue_count - 1);
+		nvme_unfreeze(nctrl);
+	}
+
+	return 0;
+
+out_wait_freeze_timed_out:
+	nvme_stop_queues(nctrl);
+	nvme_sync_io_queues(nctrl);
+
+	/* Placeholder - Stop IO queues */
+
+	if (new)
+		blk_cleanup_queue(nctrl->connect_q);
+out_free_tag_set:
+	if (new)
+		blk_mq_free_tag_set(nctrl->tagset);
+out_free_io_queues:
+	/* Placeholder - free_io_queues */
+
+	return rc;
+}
+
+static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvmf_ctrl_options *opts = nctrl->opts;
+	int rc = 0;
+
+	rc = ctrl->dev->ops->setup_ctrl(ctrl);
+	if (rc)
+		return rc;
+
+	rc = nvme_tcp_ofld_configure_admin_queue(nctrl, new);
+	if (rc)
+		goto out_release_ctrl;
+
+	if (nctrl->icdoff) {
+		dev_err(nctrl->device, "icdoff is not supported!\n");
+		rc = -EOPNOTSUPP;
+		goto destroy_admin;
+	}
+
+	if (!nvme_ctrl_sgl_supported(nctrl)) {
+		dev_err(nctrl->device, "Mandatory sgls are not supported!\n");
+		rc = -EOPNOTSUPP;
+		goto destroy_admin;
+	}
+
+	if (opts->queue_size > nctrl->sqsize + 1)
+		dev_warn(nctrl->device,
+			 "queue_size %zu > ctrl sqsize %u, clamping down\n",
+			 opts->queue_size, nctrl->sqsize + 1);
+
+	if (nctrl->sqsize + 1 > nctrl->maxcmd) {
+		dev_warn(nctrl->device,
+			 "sqsize %u > ctrl maxcmd %u, clamping down\n",
+			 nctrl->sqsize + 1, nctrl->maxcmd);
+		nctrl->sqsize = nctrl->maxcmd - 1;
+	}
+
+	if (nctrl->queue_count > 1) {
+		rc = nvme_tcp_ofld_configure_io_queues(nctrl, new);
+		if (rc)
+			goto destroy_admin;
+	}
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_LIVE)) {
+		/*
+		 * state change failure is ok if we started ctrl delete,
+		 * unless we're during creation of a new controller to
+		 * avoid races with teardown flow.
+		 */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+		WARN_ON_ONCE(new);
+		rc = -EINVAL;
+		goto destroy_io;
+	}
+
+	nvme_start_ctrl(nctrl);
+
+	return 0;
+
+destroy_io:
+	/* Placeholder - stop and destroy io queues*/
+destroy_admin:
+	/* Placeholder - stop and destroy admin queue*/
+out_release_ctrl:
+	ctrl->dev->ops->release_ctrl(ctrl);
+
+	return rc;
+}
+
+static int
+nvme_tcp_ofld_check_dev_opts(struct nvmf_ctrl_options *opts,
+			     struct nvme_tcp_ofld_ops *ofld_ops)
+{
+	unsigned int nvme_tcp_ofld_opt_mask = NVMF_ALLOWED_OPTS |
+			ofld_ops->allowed_opts | ofld_ops->required_opts;
+	struct nvmf_ctrl_options dev_opts_mask;
+
+	if (opts->mask & ~nvme_tcp_ofld_opt_mask) {
+		pr_warn("One or more nvmf options missing from ofld drvr %s.\n",
+			ofld_ops->name);
+
+		dev_opts_mask.mask = nvme_tcp_ofld_opt_mask;
+
+		return nvmf_check_required_opts(&dev_opts_mask, opts->mask);
+	}
+
+	return 0;
+}
+
+static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+
+	if (list_empty(&ctrl->list))
+		goto free_ctrl;
+
+	ctrl->dev->ops->release_ctrl(ctrl);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_del(&ctrl->list);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+
+	nvmf_free_options(nctrl->opts);
+free_ctrl:
+	module_put(dev->ops->module);
+	kfree(ctrl->queues);
+	kfree(ctrl);
+}
+
+static void
+nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove)
+{
+	/* Placeholder - teardown_admin_queue */
+}
+
+static void
+nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
+{
+	/* Placeholder - teardown_io_queues */
+}
+
+static void
+nvme_tcp_ofld_teardown_ctrl(struct nvme_ctrl *nctrl, bool shutdown)
+{
+	/* Placeholder - err_work and connect_work */
+	nvme_tcp_ofld_teardown_io_queues(nctrl, shutdown);
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	if (shutdown)
+		nvme_shutdown_ctrl(nctrl);
+	else
+		nvme_disable_ctrl(nctrl);
+	nvme_tcp_ofld_teardown_admin_queue(nctrl, shutdown);
+}
+
+static void nvme_tcp_ofld_delete_ctrl(struct nvme_ctrl *nctrl)
+{
+	nvme_tcp_ofld_teardown_ctrl(nctrl, true);
+}
+
+static int
+nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
+			   struct request *rq,
+			   unsigned int hctx_idx,
+			   unsigned int numa_node)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+
+	/* Placeholder - init request */
+
+	req->done = nvme_tcp_ofld_req_done;
+
+	return 0;
+}
+
+static blk_status_t
+nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
+		       const struct blk_mq_queue_data *bd)
+{
+	/* Call nvme_setup_cmd(...) */
+
+	/* Call ops->send_req(...) */
+
+	return BLK_STS_OK;
+}
+
+static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
+	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.init_request	= nvme_tcp_ofld_init_request,
+	/*
+	 * All additional ops will be also implemented and registered similar to
+	 * tcp.c
+	 */
+};
+
+static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
+	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.init_request	= nvme_tcp_ofld_init_request,
+	/*
+	 * All additional ops will be also implemented and registered similar to
+	 * tcp.c
+	 */
+};
+
+static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
+	.name			= "tcp_offload",
+	.module			= THIS_MODULE,
+	.flags			= NVME_F_FABRICS,
+	.reg_read32		= nvmf_reg_read32,
+	.reg_read64		= nvmf_reg_read64,
+	.reg_write32		= nvmf_reg_write32,
+	.free_ctrl		= nvme_tcp_ofld_free_ctrl,
+	.delete_ctrl		= nvme_tcp_ofld_delete_ctrl,
+	.get_address		= nvmf_get_address,
+};
+
+static bool
+nvme_tcp_ofld_existing_controller(struct nvmf_ctrl_options *opts)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	bool found = false;
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_for_each_entry(ctrl, &nvme_tcp_ofld_ctrl_list, list) {
+		found = nvmf_ip_options_match(&ctrl->nctrl, opts);
+		if (found)
+			break;
+	}
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+
+	return found;
+}
+
 static struct nvme_ctrl *
 nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 {
+	struct nvme_tcp_ofld_queue *queue;
 	struct nvme_tcp_ofld_ctrl *ctrl;
 	struct nvme_tcp_ofld_dev *dev;
 	struct nvme_ctrl *nctrl;
-	int rc = 0;
+	int i, rc = 0;
 
 	ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
 	if (!ctrl)
 		return ERR_PTR(-ENOMEM);
 
+	INIT_LIST_HEAD(&ctrl->list);
 	nctrl = &ctrl->nctrl;
+	nctrl->opts = opts;
+	nctrl->queue_count = opts->nr_io_queues + opts->nr_write_queues +
+			     opts->nr_poll_queues + 1;
+	nctrl->sqsize = opts->queue_size - 1;
+	nctrl->kato = opts->kato;
+	if (!(opts->mask & NVMF_OPT_TRSVCID)) {
+		opts->trsvcid =
+			kstrdup(__stringify(NVME_TCP_DISC_PORT), GFP_KERNEL);
+		if (!opts->trsvcid) {
+			rc = -ENOMEM;
+			goto out_free_ctrl;
+		}
+		opts->mask |= NVMF_OPT_TRSVCID;
+	}
+
+	rc = inet_pton_with_scope(&init_net, AF_UNSPEC, opts->traddr,
+				  opts->trsvcid,
+				  &ctrl->conn_params.remote_ip_addr);
+	if (rc) {
+		pr_err("malformed address passed: %s:%s\n",
+		       opts->traddr, opts->trsvcid);
+		goto out_free_ctrl;
+	}
 
-	/* Init nvme_tcp_ofld_ctrl & nvme_ctrl params based on received opts */
+	if (opts->mask & NVMF_OPT_HOST_TRADDR) {
+		rc = inet_pton_with_scope(&init_net, AF_UNSPEC,
+					  opts->host_traddr, NULL,
+					  &ctrl->conn_params.local_ip_addr);
+		if (rc) {
+			pr_err("malformed src address passed: %s\n",
+			       opts->host_traddr);
+			goto out_free_ctrl;
+		}
+	}
+
+	if (!opts->duplicate_connect &&
+	    nvme_tcp_ofld_existing_controller(opts)) {
+		rc = -EALREADY;
+		goto out_free_ctrl;
+	}
 
 	/* Find device that can reach the dest addr */
 	dev = nvme_tcp_ofld_lookup_dev(ctrl);
@@ -165,6 +589,10 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 		goto out_free_ctrl;
 	}
 
+	rc = nvme_tcp_ofld_check_dev_opts(opts, dev->ops);
+	if (rc)
+		goto out_module_put;
+
 	ctrl->dev = dev;
 
 	if (ctrl->dev->ops->max_hw_sectors)
@@ -172,14 +600,55 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 	if (ctrl->dev->ops->max_segments)
 		nctrl->max_segments = ctrl->dev->ops->max_segments;
 
-	/* Init queues */
+	ctrl->queues = kcalloc(nctrl->queue_count,
+			       sizeof(struct nvme_tcp_ofld_queue),
+			       GFP_KERNEL);
+	if (!ctrl->queues) {
+		rc = -ENOMEM;
+		goto out_module_put;
+	}
+
+	for (i = 0; i < nctrl->queue_count; ++i) {
+		queue = &ctrl->queues[i];
+		queue->ctrl = ctrl;
+		queue->dev = dev;
+		queue->report_err = nvme_tcp_ofld_report_queue_err;
+	}
+
+	rc = nvme_init_ctrl(nctrl, ndev, &nvme_tcp_ofld_ctrl_ops, 0);
+	if (rc)
+		goto out_free_queues;
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		WARN_ON_ONCE(1);
+		rc = -EINTR;
+		goto out_uninit_ctrl;
+	}
 
-	/* Call nvme_init_ctrl */
+	rc = nvme_tcp_ofld_setup_ctrl(nctrl, true);
+	if (rc)
+		goto out_uninit_ctrl;
 
-	/* Setup ctrl */
+	dev_info(nctrl->device, "new ctrl: NQN \"%s\", addr %pISp\n",
+		 opts->subsysnqn, &ctrl->conn_params.remote_ip_addr);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_add_tail(&ctrl->list, &nvme_tcp_ofld_ctrl_list);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
 
 	return nctrl;
 
+out_uninit_ctrl:
+	nvme_uninit_ctrl(nctrl);
+	nvme_put_ctrl(nctrl);
+	if (rc > 0)
+		rc = -EIO;
+
+	return ERR_PTR(rc);
+out_free_queues:
+	kfree(ctrl->queues);
+out_module_put:
+	module_put(dev->ops->module);
 out_free_ctrl:
 	kfree(ctrl);
 
@@ -207,7 +676,15 @@ static int __init nvme_tcp_ofld_init_module(void)
 
 static void __exit nvme_tcp_ofld_cleanup_module(void)
 {
+	struct nvme_tcp_ofld_ctrl *ctrl;
+
 	nvmf_unregister_transport(&nvme_tcp_ofld_transport);
+
+	mutex_lock(&nvme_tcp_ofld_ctrl_mutex);
+	list_for_each_entry(ctrl, &nvme_tcp_ofld_ctrl_list, list)
+		nvme_delete_ctrl(&ctrl->nctrl);
+	mutex_unlock(&nvme_tcp_ofld_ctrl_mutex);
+	flush_workqueue(nvme_delete_wq);
 }
 
 module_init(nvme_tcp_ofld_init_module);
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:29 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:29 +0300
Subject: [PATCH v4 06/20] nvme-tcp-offload: Add controller level error
 recovery implementation
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-7-pkushwaha@marvell.com>

From: Arie Gershberg <agershberg at marvell.com>

In this patch, we implement controller level error handling and recovery.
Upon an error discovered by the ULP or reset controller initiated by the
nvme-core (using reset_ctrl workqueue), the ULP will initiate a controller
recovery which includes teardown and re-connect of all queues.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 drivers/nvme/host/tcp-offload.c | 127 +++++++++++++++++++++++++++++++-
 drivers/nvme/host/tcp-offload.h |   1 +
 2 files changed, 126 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 806247722a35..3af89a938d1b 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -72,6 +72,23 @@ void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev)
 }
 EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
 
+/**
+ * nvme_tcp_ofld_error_recovery() - NVMeTCP Offload library error recovery.
+ * function.
+ * @nctrl:	NVMe controller instance to change to resetting.
+ *
+ * API function that change the controller state to resseting.
+ * Part of the overall controller reset sequence.
+ */
+void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl)
+{
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_RESETTING))
+		return;
+
+	queue_work(nvme_reset_wq, &to_tcp_ofld_ctrl(nctrl)->err_work);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_error_recovery);
+
 /**
  * nvme_tcp_ofld_report_queue_err() - NVMeTCP Offload report error event
  * callback function. Pointed to by nvme_tcp_ofld_queue->report_err.
@@ -82,7 +99,8 @@ EXPORT_SYMBOL_GPL(nvme_tcp_ofld_unregister_dev);
  */
 int nvme_tcp_ofld_report_queue_err(struct nvme_tcp_ofld_queue *queue)
 {
-	/* Placeholder - invoke error recovery flow */
+	pr_err("nvme-tcp-offload queue error\n");
+	nvme_tcp_ofld_error_recovery(&queue->ctrl->nctrl);
 
 	return 0;
 }
@@ -302,6 +320,28 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 	return rc;
 }
 
+static void nvme_tcp_ofld_reconnect_or_remove(struct nvme_ctrl *nctrl)
+{
+	/* If we are resetting/deleting then do nothing */
+	if (nctrl->state != NVME_CTRL_CONNECTING) {
+		WARN_ON_ONCE(nctrl->state == NVME_CTRL_NEW ||
+			     nctrl->state == NVME_CTRL_LIVE);
+
+		return;
+	}
+
+	if (nvmf_should_reconnect(nctrl)) {
+		dev_info(nctrl->device, "Reconnecting in %d seconds...\n",
+			 nctrl->opts->reconnect_delay);
+		queue_delayed_work(nvme_wq,
+				   &to_tcp_ofld_ctrl(nctrl)->connect_work,
+				   nctrl->opts->reconnect_delay * HZ);
+	} else {
+		dev_info(nctrl->device, "Removing controller...\n");
+		nvme_delete_ctrl(nctrl);
+	}
+}
+
 static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 {
 	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
@@ -426,10 +466,63 @@ nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
 	/* Placeholder - teardown_io_queues */
 }
 
+static void nvme_tcp_ofld_reconnect_ctrl_work(struct work_struct *work)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl =
+				container_of(to_delayed_work(work),
+					     struct nvme_tcp_ofld_ctrl,
+					     connect_work);
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+
+	++nctrl->nr_reconnects;
+
+	if (nvme_tcp_ofld_setup_ctrl(nctrl, false))
+		goto requeue;
+
+	dev_info(nctrl->device, "Successfully reconnected (%d attempt)\n",
+		 nctrl->nr_reconnects);
+
+	nctrl->nr_reconnects = 0;
+
+	return;
+
+requeue:
+	dev_info(nctrl->device, "Failed reconnect attempt %d\n",
+		 nctrl->nr_reconnects);
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
+static void nvme_tcp_ofld_error_recovery_work(struct work_struct *work)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl =
+		container_of(work, struct nvme_tcp_ofld_ctrl, err_work);
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+
+	nvme_stop_keep_alive(nctrl);
+	nvme_tcp_ofld_teardown_io_queues(nctrl, false);
+	/* unquiesce to fail fast pending requests */
+	nvme_start_queues(nctrl);
+	nvme_tcp_ofld_teardown_admin_queue(nctrl, false);
+	blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		/* state change failure is ok if we started nctrl delete */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+
+		return;
+	}
+
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
 static void
 nvme_tcp_ofld_teardown_ctrl(struct nvme_ctrl *nctrl, bool shutdown)
 {
-	/* Placeholder - err_work and connect_work */
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+
+	cancel_work_sync(&ctrl->err_work);
+	cancel_delayed_work_sync(&ctrl->connect_work);
 	nvme_tcp_ofld_teardown_io_queues(nctrl, shutdown);
 	blk_mq_quiesce_queue(nctrl->admin_q);
 	if (shutdown)
@@ -444,6 +537,32 @@ static void nvme_tcp_ofld_delete_ctrl(struct nvme_ctrl *nctrl)
 	nvme_tcp_ofld_teardown_ctrl(nctrl, true);
 }
 
+static void nvme_tcp_ofld_reset_ctrl_work(struct work_struct *work)
+{
+	struct nvme_ctrl *nctrl =
+		container_of(work, struct nvme_ctrl, reset_work);
+
+	nvme_stop_ctrl(nctrl);
+	nvme_tcp_ofld_teardown_ctrl(nctrl, false);
+
+	if (!nvme_change_ctrl_state(nctrl, NVME_CTRL_CONNECTING)) {
+		/* state change failure is ok if we started ctrl delete */
+		WARN_ON_ONCE(nctrl->state != NVME_CTRL_DELETING &&
+			     nctrl->state != NVME_CTRL_DELETING_NOIO);
+
+		return;
+	}
+
+	if (nvme_tcp_ofld_setup_ctrl(nctrl, false))
+		goto out_fail;
+
+	return;
+
+out_fail:
+	++nctrl->nr_reconnects;
+	nvme_tcp_ofld_reconnect_or_remove(nctrl);
+}
+
 static int
 nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 			   struct request *rq,
@@ -537,6 +656,10 @@ nvme_tcp_ofld_create_ctrl(struct device *ndev, struct nvmf_ctrl_options *opts)
 			     opts->nr_poll_queues + 1;
 	nctrl->sqsize = opts->queue_size - 1;
 	nctrl->kato = opts->kato;
+	INIT_DELAYED_WORK(&ctrl->connect_work,
+			  nvme_tcp_ofld_reconnect_ctrl_work);
+	INIT_WORK(&ctrl->err_work, nvme_tcp_ofld_error_recovery_work);
+	INIT_WORK(&nctrl->reset_work, nvme_tcp_ofld_reset_ctrl_work);
 	if (!(opts->mask & NVMF_OPT_TRSVCID)) {
 		opts->trsvcid =
 			kstrdup(__stringify(NVME_TCP_DISC_PORT), GFP_KERNEL);
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index d1c2c6171897..51fec632b72b 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -198,3 +198,4 @@ struct nvme_tcp_ofld_ops {
 /* Exported functions for offload device specific drivers */
 int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
+void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl);
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:30 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:30 +0300
Subject: [PATCH v4 07/20] nvme-tcp-offload: Add queue level implementation
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-8-pkushwaha@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

In this patch we implement queue level functionality.
The implementation is similar to the nvme-tcp module, the main
difference being that we call the offload device specific create_queue
op which creates the TCP connection, and NVMeTPC connection including
icreq+icresp negotiation.
Once create_queue returns successfully, we can move on to the fabrics
connect.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/host/tcp-offload.c | 420 +++++++++++++++++++++++++++++---
 drivers/nvme/host/tcp-offload.h |   4 +
 2 files changed, 396 insertions(+), 28 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 3af89a938d1b..26253b107db2 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -22,6 +22,11 @@ static inline struct nvme_tcp_ofld_ctrl *to_tcp_ofld_ctrl(struct nvme_ctrl *nc)
 	return container_of(nc, struct nvme_tcp_ofld_ctrl, nctrl);
 }
 
+static inline int nvme_tcp_ofld_qid(struct nvme_tcp_ofld_queue *queue)
+{
+	return queue - queue->ctrl->queues;
+}
+
 /**
  * nvme_tcp_ofld_register_dev() - NVMeTCP Offload Library registration
  * function.
@@ -196,19 +201,127 @@ nvme_tcp_ofld_alloc_tagset(struct nvme_ctrl *nctrl, bool admin)
 	return set;
 }
 
+static void __nvme_tcp_ofld_stop_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	queue->dev->ops->drain_queue(queue);
+}
+
+static void nvme_tcp_ofld_stop_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+
+	mutex_lock(&queue->queue_lock);
+	if (test_and_clear_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags))
+		__nvme_tcp_ofld_stop_queue(queue);
+	mutex_unlock(&queue->queue_lock);
+}
+
+static void nvme_tcp_ofld_stop_io_queues(struct nvme_ctrl *ctrl)
+{
+	int i;
+
+	for (i = 1; i < ctrl->queue_count; i++)
+		nvme_tcp_ofld_stop_queue(ctrl, i);
+}
+
+static void __nvme_tcp_ofld_free_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	queue->dev->ops->destroy_queue(queue);
+}
+
+static void nvme_tcp_ofld_free_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+
+	if (test_and_clear_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags)) {
+		__nvme_tcp_ofld_free_queue(queue);
+		mutex_destroy(&queue->queue_lock);
+	}
+}
+
+static void
+nvme_tcp_ofld_free_io_queues(struct nvme_ctrl *nctrl)
+{
+	int i;
+
+	for (i = 1; i < nctrl->queue_count; i++)
+		nvme_tcp_ofld_free_queue(nctrl, i);
+}
+
+static void nvme_tcp_ofld_destroy_io_queues(struct nvme_ctrl *nctrl,
+					    bool remove)
+{
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+	if (remove) {
+		blk_cleanup_queue(nctrl->connect_q);
+		blk_mq_free_tag_set(nctrl->tagset);
+	}
+	nvme_tcp_ofld_free_io_queues(nctrl);
+}
+
+static void nvme_tcp_ofld_destroy_admin_queue(struct nvme_ctrl *nctrl,
+					      bool remove)
+{
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	if (remove) {
+		blk_cleanup_queue(nctrl->admin_q);
+		blk_cleanup_queue(nctrl->fabrics_q);
+		blk_mq_free_tag_set(nctrl->admin_tagset);
+	}
+	nvme_tcp_ofld_free_queue(nctrl, 0);
+}
+
+static int nvme_tcp_ofld_start_queue(struct nvme_ctrl *nctrl, int qid)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[qid];
+	int rc;
+
+	queue = &ctrl->queues[qid];
+	if (qid) {
+		queue->cmnd_capsule_len = nctrl->ioccsz * 16;
+		rc = nvmf_connect_io_queue(nctrl, qid, false);
+	} else {
+		queue->cmnd_capsule_len = sizeof(struct nvme_command) +
+						NVME_TCP_ADMIN_CCSZ;
+		rc = nvmf_connect_admin_queue(nctrl);
+	}
+
+	if (!rc) {
+		set_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags);
+	} else {
+		if (test_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags))
+			__nvme_tcp_ofld_stop_queue(queue);
+		dev_err(nctrl->device,
+			"failed to connect queue: %d ret=%d\n", qid, rc);
+	}
+
+	return rc;
+}
+
 static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 					       bool new)
 {
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[0];
 	int rc;
 
-	/* Placeholder - alloc_admin_queue */
+	mutex_init(&queue->queue_lock);
+
+	rc = ctrl->dev->ops->create_queue(queue, 0, NVME_AQ_DEPTH);
+	if (rc)
+		return rc;
+
+	set_bit(NVME_TCP_OFLD_Q_ALLOCATED, &queue->flags);
 	if (new) {
 		nctrl->admin_tagset =
 				nvme_tcp_ofld_alloc_tagset(nctrl, true);
 		if (IS_ERR(nctrl->admin_tagset)) {
 			rc = PTR_ERR(nctrl->admin_tagset);
 			nctrl->admin_tagset = NULL;
-			goto out_destroy_queue;
+			goto out_free_queue;
 		}
 
 		nctrl->fabrics_q = blk_mq_init_queue(nctrl->admin_tagset);
@@ -226,7 +339,9 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 		}
 	}
 
-	/* Placeholder - nvme_tcp_ofld_start_queue */
+	rc = nvme_tcp_ofld_start_queue(nctrl, 0);
+	if (rc)
+		goto out_cleanup_queue;
 
 	rc = nvme_enable_ctrl(nctrl);
 	if (rc)
@@ -243,19 +358,143 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 out_quiesce_queue:
 	blk_mq_quiesce_queue(nctrl->admin_q);
 	blk_sync_queue(nctrl->admin_q);
-
 out_stop_queue:
-	/* Placeholder - stop offload queue */
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
 	nvme_cancel_admin_tagset(nctrl);
-
+out_cleanup_queue:
+	if (new)
+		blk_cleanup_queue(nctrl->admin_q);
 out_cleanup_fabrics_q:
 	if (new)
 		blk_cleanup_queue(nctrl->fabrics_q);
 out_free_tagset:
 	if (new)
 		blk_mq_free_tag_set(nctrl->admin_tagset);
-out_destroy_queue:
-	/* Placeholder - free admin queue */
+out_free_queue:
+	nvme_tcp_ofld_free_queue(nctrl, 0);
+
+	return rc;
+}
+
+static unsigned int nvme_tcp_ofld_nr_io_queues(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+	u32 hw_vectors = dev->num_hw_vectors;
+	u32 nr_write_queues, nr_poll_queues;
+	u32 nr_io_queues, nr_total_queues;
+
+	nr_io_queues = min3(nctrl->opts->nr_io_queues, num_online_cpus(),
+			    hw_vectors);
+	nr_write_queues = min3(nctrl->opts->nr_write_queues, num_online_cpus(),
+			       hw_vectors);
+	nr_poll_queues = min3(nctrl->opts->nr_poll_queues, num_online_cpus(),
+			      hw_vectors);
+
+	nr_total_queues = nr_io_queues + nr_write_queues + nr_poll_queues;
+
+	return nr_total_queues;
+}
+
+static void
+nvme_tcp_ofld_set_io_queues(struct nvme_ctrl *nctrl, unsigned int nr_io_queues)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	struct nvmf_ctrl_options *opts = nctrl->opts;
+
+	if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) {
+		/*
+		 * separate read/write queues
+		 * hand out dedicated default queues only after we have
+		 * sufficient read queues.
+		 */
+		ctrl->io_queues[HCTX_TYPE_READ] = opts->nr_io_queues;
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_READ];
+		ctrl->io_queues[HCTX_TYPE_DEFAULT] =
+			min(opts->nr_write_queues, nr_io_queues);
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	} else {
+		/*
+		 * shared read/write queues
+		 * either no write queues were requested, or we don't have
+		 * sufficient queue count to have dedicated default queues.
+		 */
+		ctrl->io_queues[HCTX_TYPE_DEFAULT] =
+			min(opts->nr_io_queues, nr_io_queues);
+		nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	}
+
+	if (opts->nr_poll_queues && nr_io_queues) {
+		/* map dedicated poll queues only if we have queues left */
+		ctrl->io_queues[HCTX_TYPE_POLL] =
+			min(opts->nr_poll_queues, nr_io_queues);
+	}
+}
+
+static int nvme_tcp_ofld_create_io_queues(struct nvme_ctrl *nctrl)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
+	int i, rc;
+
+	for (i = 1; i < nctrl->queue_count; i++) {
+		mutex_init(&ctrl->queues[i].queue_lock);
+
+		rc = ctrl->dev->ops->create_queue(&ctrl->queues[i],
+						  i, nctrl->sqsize + 1);
+		if (rc)
+			goto out_free_queues;
+
+		set_bit(NVME_TCP_OFLD_Q_ALLOCATED, &ctrl->queues[i].flags);
+	}
+
+	return 0;
+
+out_free_queues:
+	for (i--; i >= 1; i--)
+		nvme_tcp_ofld_free_queue(nctrl, i);
+
+	return rc;
+}
+
+static int nvme_tcp_ofld_alloc_io_queues(struct nvme_ctrl *nctrl)
+{
+	unsigned int nr_io_queues;
+	int rc;
+
+	nr_io_queues = nvme_tcp_ofld_nr_io_queues(nctrl);
+	rc = nvme_set_queue_count(nctrl, &nr_io_queues);
+	if (rc)
+		return rc;
+
+	nctrl->queue_count = nr_io_queues + 1;
+	if (nctrl->queue_count < 2) {
+		dev_err(nctrl->device,
+			"unable to set any I/O queues\n");
+
+		return -ENOMEM;
+	}
+
+	dev_info(nctrl->device, "creating %d I/O queues.\n", nr_io_queues);
+	nvme_tcp_ofld_set_io_queues(nctrl, nr_io_queues);
+
+	return nvme_tcp_ofld_create_io_queues(nctrl);
+}
+
+static int nvme_tcp_ofld_start_io_queues(struct nvme_ctrl *nctrl)
+{
+	int i, rc = 0;
+
+	for (i = 1; i < nctrl->queue_count; i++) {
+		rc = nvme_tcp_ofld_start_queue(nctrl, i);
+		if (rc)
+			goto out_stop_queues;
+	}
+
+	return 0;
+
+out_stop_queues:
+	for (i--; i >= 1; i--)
+		nvme_tcp_ofld_stop_queue(nctrl, i);
 
 	return rc;
 }
@@ -263,9 +502,10 @@ static int nvme_tcp_ofld_configure_admin_queue(struct nvme_ctrl *nctrl,
 static int
 nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 {
-	int rc;
+	int rc = nvme_tcp_ofld_alloc_io_queues(nctrl);
 
-	/* Placeholder - alloc_io_queues */
+	if (rc)
+		return rc;
 
 	if (new) {
 		nctrl->tagset = nvme_tcp_ofld_alloc_tagset(nctrl, false);
@@ -283,7 +523,9 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 		}
 	}
 
-	/* Placeholder - start_io_queues */
+	rc = nvme_tcp_ofld_start_io_queues(nctrl);
+	if (rc)
+		goto out_cleanup_connect_q;
 
 	if (!new) {
 		nvme_start_queues(nctrl);
@@ -306,16 +548,16 @@ nvme_tcp_ofld_configure_io_queues(struct nvme_ctrl *nctrl, bool new)
 out_wait_freeze_timed_out:
 	nvme_stop_queues(nctrl);
 	nvme_sync_io_queues(nctrl);
-
-	/* Placeholder - Stop IO queues */
-
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+out_cleanup_connect_q:
+	nvme_cancel_tagset(nctrl);
 	if (new)
 		blk_cleanup_queue(nctrl->connect_q);
 out_free_tag_set:
 	if (new)
 		blk_mq_free_tag_set(nctrl->tagset);
 out_free_io_queues:
-	/* Placeholder - free_io_queues */
+	nvme_tcp_ofld_free_io_queues(nctrl);
 
 	return rc;
 }
@@ -342,6 +584,17 @@ static void nvme_tcp_ofld_reconnect_or_remove(struct nvme_ctrl *nctrl)
 	}
 }
 
+static int
+nvme_tcp_ofld_init_admin_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			      unsigned int hctx_idx)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = data;
+
+	hctx->driver_data = &ctrl->queues[0];
+
+	return 0;
+}
+
 static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 {
 	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(nctrl);
@@ -404,9 +657,19 @@ static int nvme_tcp_ofld_setup_ctrl(struct nvme_ctrl *nctrl, bool new)
 	return 0;
 
 destroy_io:
-	/* Placeholder - stop and destroy io queues*/
+	if (nctrl->queue_count > 1) {
+		nvme_stop_queues(nctrl);
+		nvme_sync_io_queues(nctrl);
+		nvme_tcp_ofld_stop_io_queues(nctrl);
+		nvme_cancel_tagset(nctrl);
+		nvme_tcp_ofld_destroy_io_queues(nctrl, new);
+	}
 destroy_admin:
-	/* Placeholder - stop and destroy admin queue*/
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	nvme_cancel_admin_tagset(nctrl);
+	nvme_tcp_ofld_destroy_admin_queue(nctrl, new);
 out_release_ctrl:
 	ctrl->dev->ops->release_ctrl(ctrl);
 
@@ -455,15 +718,37 @@ static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
 }
 
 static void
-nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove)
+nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *nctrl, bool remove)
 {
-	/* Placeholder - teardown_admin_queue */
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	blk_sync_queue(nctrl->admin_q);
+
+	nvme_tcp_ofld_stop_queue(nctrl, 0);
+	nvme_cancel_admin_tagset(nctrl);
+
+	if (remove)
+		blk_mq_unquiesce_queue(nctrl->admin_q);
+
+	nvme_tcp_ofld_destroy_admin_queue(nctrl, remove);
 }
 
 static void
 nvme_tcp_ofld_teardown_io_queues(struct nvme_ctrl *nctrl, bool remove)
 {
-	/* Placeholder - teardown_io_queues */
+	if (nctrl->queue_count <= 1)
+		return;
+
+	blk_mq_quiesce_queue(nctrl->admin_q);
+	nvme_start_freeze(nctrl);
+	nvme_stop_queues(nctrl);
+	nvme_sync_io_queues(nctrl);
+	nvme_tcp_ofld_stop_io_queues(nctrl);
+	nvme_cancel_tagset(nctrl);
+
+	if (remove)
+		nvme_start_queues(nctrl);
+
+	nvme_tcp_ofld_destroy_io_queues(nctrl, remove);
 }
 
 static void nvme_tcp_ofld_reconnect_ctrl_work(struct work_struct *work)
@@ -578,6 +863,12 @@ nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 	return 0;
 }
 
+inline size_t nvme_tcp_ofld_inline_data_size(struct nvme_tcp_ofld_queue *queue)
+{
+	return queue->cmnd_capsule_len - sizeof(struct nvme_command);
+}
+EXPORT_SYMBOL_GPL(nvme_tcp_ofld_inline_data_size);
+
 static blk_status_t
 nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 		       const struct blk_mq_queue_data *bd)
@@ -589,22 +880,95 @@ nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 	return BLK_STS_OK;
 }
 
+static void
+nvme_tcp_ofld_exit_request(struct blk_mq_tag_set *set,
+			   struct request *rq, unsigned int hctx_idx)
+{
+	/*
+	 * Nothing is allocated in nvme_tcp_ofld_init_request,
+	 * hence empty.
+	 */
+}
+
+static int
+nvme_tcp_ofld_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
+			unsigned int hctx_idx)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = data;
+
+	hctx->driver_data = &ctrl->queues[hctx_idx + 1];
+
+	return 0;
+}
+
+static int nvme_tcp_ofld_map_queues(struct blk_mq_tag_set *set)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = set->driver_data;
+	struct nvmf_ctrl_options *opts = ctrl->nctrl.opts;
+
+	if (opts->nr_write_queues && ctrl->io_queues[HCTX_TYPE_READ]) {
+		/* separate read/write queues */
+		set->map[HCTX_TYPE_DEFAULT].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_DEFAULT].queue_offset = 0;
+		set->map[HCTX_TYPE_READ].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_READ];
+		set->map[HCTX_TYPE_READ].queue_offset =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+	} else {
+		/* shared read/write queues */
+		set->map[HCTX_TYPE_DEFAULT].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_DEFAULT].queue_offset = 0;
+		set->map[HCTX_TYPE_READ].nr_queues =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT];
+		set->map[HCTX_TYPE_READ].queue_offset = 0;
+	}
+	blk_mq_map_queues(&set->map[HCTX_TYPE_DEFAULT]);
+	blk_mq_map_queues(&set->map[HCTX_TYPE_READ]);
+
+	if (opts->nr_poll_queues && ctrl->io_queues[HCTX_TYPE_POLL]) {
+		/* map dedicated poll queues only if we have queues left */
+		set->map[HCTX_TYPE_POLL].nr_queues =
+				ctrl->io_queues[HCTX_TYPE_POLL];
+		set->map[HCTX_TYPE_POLL].queue_offset =
+			ctrl->io_queues[HCTX_TYPE_DEFAULT] +
+			ctrl->io_queues[HCTX_TYPE_READ];
+		blk_mq_map_queues(&set->map[HCTX_TYPE_POLL]);
+	}
+
+	dev_info(ctrl->nctrl.device,
+		 "mapped %d/%d/%d default/read/poll queues.\n",
+		 ctrl->io_queues[HCTX_TYPE_DEFAULT],
+		 ctrl->io_queues[HCTX_TYPE_READ],
+		 ctrl->io_queues[HCTX_TYPE_POLL]);
+
+	return 0;
+}
+
+static int nvme_tcp_ofld_poll(struct blk_mq_hw_ctx *hctx)
+{
+	/* Placeholder - Implement polling mechanism */
+
+	return 0;
+}
+
 static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
 	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.complete	= nvme_complete_rq,
 	.init_request	= nvme_tcp_ofld_init_request,
-	/*
-	 * All additional ops will be also implemented and registered similar to
-	 * tcp.c
-	 */
+	.exit_request	= nvme_tcp_ofld_exit_request,
+	.init_hctx	= nvme_tcp_ofld_init_hctx,
+	.map_queues	= nvme_tcp_ofld_map_queues,
+	.poll		= nvme_tcp_ofld_poll,
 };
 
 static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
 	.queue_rq	= nvme_tcp_ofld_queue_rq,
+	.complete	= nvme_complete_rq,
 	.init_request	= nvme_tcp_ofld_init_request,
-	/*
-	 * All additional ops will be also implemented and registered similar to
-	 * tcp.c
-	 */
+	.exit_request	= nvme_tcp_ofld_exit_request,
+	.init_hctx	= nvme_tcp_ofld_init_admin_hctx,
 };
 
 static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index 51fec632b72b..b3502c01394e 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -65,6 +65,9 @@ struct nvme_tcp_ofld_queue {
 	unsigned long flags;
 	size_t cmnd_capsule_len;
 
+	/* mutex used during stop_queue */
+	struct mutex queue_lock;
+
 	u8 hdr_digest;
 	u8 data_digest;
 	u8 tos;
@@ -199,3 +202,4 @@ struct nvme_tcp_ofld_ops {
 int nvme_tcp_ofld_register_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_unregister_dev(struct nvme_tcp_ofld_dev *dev);
 void nvme_tcp_ofld_error_recovery(struct nvme_ctrl *nctrl);
+inline size_t nvme_tcp_ofld_inline_data_size(struct nvme_tcp_ofld_queue *queue);
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:31 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:31 +0300
Subject: [PATCH v4 08/20] nvme-tcp-offload: Add IO level implementation
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-9-pkushwaha@marvell.com>

From: Dean Balandin <dbalandin at marvell.com>

In this patch, we present the IO level functionality.
The nvme-tcp-offload shall work on the IO-level, meaning the
nvme-tcp-offload ULP module shall pass the request to the nvme-tcp-offload
driver and shall expect for the request completion.
No additional handling is needed in between, this design will reduce the
CPU utilization as we will describe below.

The nvme-tcp-offload driver shall register to nvme-tcp-offload ULP
with the following IO-path ops:
 - send_req - in order to pass the request to the handling of the offload
   driver that shall pass it to the offload specific device
 - poll_queue

The offload device driver will manage the context from which the request
will be executed and the request aggregations.
Once the IO completed, the nvme-tcp-offload driver shall call
command.done() that shall invoke the nvme-tcp-offload ULP layer for
completing the request.

This patch also add support for the nvme-tcp-offload timeout and
nvme-tcp-offload ASYNC flow.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani at oracle.com>
---
 drivers/nvme/host/tcp-offload.c | 181 ++++++++++++++++++++++++++++++--
 drivers/nvme/host/tcp-offload.h |   2 +
 2 files changed, 176 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/tcp-offload.c b/drivers/nvme/host/tcp-offload.c
index 26253b107db2..501006ec9c97 100644
--- a/drivers/nvme/host/tcp-offload.c
+++ b/drivers/nvme/host/tcp-offload.c
@@ -125,7 +125,30 @@ void nvme_tcp_ofld_req_done(struct nvme_tcp_ofld_req *req,
 			    union nvme_result *result,
 			    __le16 status)
 {
-	/* Placeholder - complete request with/without error */
+	struct request *rq = blk_mq_rq_from_pdu(req);
+
+	if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), *result))
+		nvme_complete_rq(rq);
+}
+
+/**
+ * nvme_tcp_ofld_async_req_done() - NVMeTCP Offload request done callback
+ * function for async request. Pointed to by nvme_tcp_ofld_req->done.
+ * Handles both NVME_TCP_F_DATA_SUCCESS flag and NVMe CQ.
+ * @req:	NVMeTCP offload request to complete.
+ * @result:     The nvme_result.
+ * @status:     The completion status.
+ *
+ * API function that allows the offload device specific driver to report
+ * request completions to the common offload layer.
+ */
+void nvme_tcp_ofld_async_req_done(struct nvme_tcp_ofld_req *req,
+				  union nvme_result *result, __le16 status)
+{
+	struct nvme_tcp_ofld_queue *queue = req->queue;
+	struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl;
+
+	nvme_complete_async_event(&ctrl->nctrl, status, result);
 }
 
 static struct nvme_tcp_ofld_dev *
@@ -717,6 +740,57 @@ static void nvme_tcp_ofld_free_ctrl(struct nvme_ctrl *nctrl)
 	kfree(ctrl);
 }
 
+static void nvme_tcp_ofld_set_sg_null(struct nvme_command *c)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = 0;
+	sg->length = 0;
+	sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) |
+			NVME_SGL_FMT_TRANSPORT_A;
+}
+
+inline void nvme_tcp_ofld_set_sg_inline(struct nvme_tcp_ofld_queue *queue,
+					struct nvme_command *c, u32 data_len)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = cpu_to_le64(queue->ctrl->nctrl.icdoff);
+	sg->length = cpu_to_le32(data_len);
+	sg->type = (NVME_SGL_FMT_DATA_DESC << 4) | NVME_SGL_FMT_OFFSET;
+}
+
+static void nvme_tcp_ofld_set_sg_host_data(struct nvme_command *c,
+					   u32 data_len)
+{
+	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;
+
+	sg->addr = 0;
+	sg->length = cpu_to_le32(data_len);
+	sg->type = (NVME_TRANSPORT_SGL_DATA_DESC << 4) |
+			NVME_SGL_FMT_TRANSPORT_A;
+}
+
+static void nvme_tcp_ofld_submit_async_event(struct nvme_ctrl *arg)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl = to_tcp_ofld_ctrl(arg);
+	struct nvme_tcp_ofld_queue *queue = &ctrl->queues[0];
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+
+	ctrl->async_req.nvme_cmd.common.opcode = nvme_admin_async_event;
+	ctrl->async_req.nvme_cmd.common.command_id = NVME_AQ_BLK_MQ_DEPTH;
+	ctrl->async_req.nvme_cmd.common.flags |= NVME_CMD_SGL_METABUF;
+
+	nvme_tcp_ofld_set_sg_null(&ctrl->async_req.nvme_cmd);
+
+	ctrl->async_req.async = true;
+	ctrl->async_req.queue = queue;
+	ctrl->async_req.done = nvme_tcp_ofld_async_req_done;
+
+	ops->send_req(&ctrl->async_req);
+}
+
 static void
 nvme_tcp_ofld_teardown_admin_queue(struct nvme_ctrl *nctrl, bool remove)
 {
@@ -855,9 +929,13 @@ nvme_tcp_ofld_init_request(struct blk_mq_tag_set *set,
 			   unsigned int numa_node)
 {
 	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_ofld_ctrl *ctrl = set->driver_data;
+	int qid;
 
-	/* Placeholder - init request */
-
+	qid = (set == &ctrl->tag_set) ? hctx_idx + 1 : 0;
+	req->queue = &ctrl->queues[qid];
+	nvme_req(rq)->ctrl = &ctrl->nctrl;
+	nvme_req(rq)->cmd = &req->nvme_cmd;
 	req->done = nvme_tcp_ofld_req_done;
 
 	return 0;
@@ -873,9 +951,46 @@ static blk_status_t
 nvme_tcp_ofld_queue_rq(struct blk_mq_hw_ctx *hctx,
 		       const struct blk_mq_queue_data *bd)
 {
-	/* Call nvme_setup_cmd(...) */
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(bd->rq);
+	struct nvme_tcp_ofld_queue *queue = hctx->driver_data;
+	struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl;
+	struct nvme_ns *ns = hctx->queue->queuedata;
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
+	struct nvme_command *nvme_cmd;
+	struct request *rq = bd->rq;
+	bool queue_ready;
+	u32 data_len;
+	int rc;
+
+	queue_ready = test_bit(NVME_TCP_OFLD_Q_LIVE, &queue->flags);
+
+	req->async = false;
+
+	if (!nvme_check_ready(&ctrl->nctrl, rq, queue_ready))
+		return nvme_fail_nonready_command(&ctrl->nctrl, rq);
+
+	rc = nvme_setup_cmd(ns, rq);
+	if (unlikely(rc))
+		return rc;
 
-	/* Call ops->send_req(...) */
+	blk_mq_start_request(rq);
+
+	nvme_cmd = &req->nvme_cmd;
+	nvme_cmd->common.flags |= NVME_CMD_SGL_METABUF;
+
+	data_len = blk_rq_nr_phys_segments(rq) ? blk_rq_payload_bytes(rq) : 0;
+	if (!data_len)
+		nvme_tcp_ofld_set_sg_null(&req->nvme_cmd);
+	else if ((rq_data_dir(rq) == WRITE) &&
+		 data_len <= nvme_tcp_ofld_inline_data_size(queue))
+		nvme_tcp_ofld_set_sg_inline(queue, nvme_cmd, data_len);
+	else
+		nvme_tcp_ofld_set_sg_host_data(nvme_cmd, data_len);
+
+	rc = ops->send_req(req);
+	if (unlikely(rc))
+		return rc;
 
 	return BLK_STS_OK;
 }
@@ -948,9 +1063,58 @@ static int nvme_tcp_ofld_map_queues(struct blk_mq_tag_set *set)
 
 static int nvme_tcp_ofld_poll(struct blk_mq_hw_ctx *hctx)
 {
-	/* Placeholder - Implement polling mechanism */
+	struct nvme_tcp_ofld_queue *queue = hctx->driver_data;
+	struct nvme_tcp_ofld_dev *dev = queue->dev;
+	struct nvme_tcp_ofld_ops *ops = dev->ops;
 
-	return 0;
+	return ops->poll_queue(queue);
+}
+
+static void nvme_tcp_ofld_complete_timed_out(struct request *rq)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_ctrl *nctrl = &req->queue->ctrl->nctrl;
+
+	nvme_tcp_ofld_stop_queue(nctrl, nvme_tcp_ofld_qid(req->queue));
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq)) {
+		nvme_req(rq)->status = NVME_SC_HOST_ABORTED_CMD;
+		blk_mq_complete_request(rq);
+	}
+}
+
+static enum blk_eh_timer_return nvme_tcp_ofld_timeout(struct request *rq,
+						      bool reserved)
+{
+	struct nvme_tcp_ofld_req *req = blk_mq_rq_to_pdu(rq);
+	struct nvme_tcp_ofld_ctrl *ctrl = req->queue->ctrl;
+
+	dev_warn(ctrl->nctrl.device,
+		 "queue %d: timeout request %#x type %d\n",
+		 nvme_tcp_ofld_qid(req->queue), rq->tag,
+		 req->nvme_cmd.common.opcode);
+
+	if (ctrl->nctrl.state != NVME_CTRL_LIVE) {
+		/*
+		 * If we are resetting, connecting or deleting we should
+		 * complete immediately because we may block controller
+		 * teardown or setup sequence
+		 * - ctrl disable/shutdown fabrics requests
+		 * - connect requests
+		 * - initialization admin requests
+		 * - I/O requests that entered after unquiescing and
+		 *   the controller stopped responding
+		 *
+		 * All other requests should be cancelled by the error
+		 * recovery work, so it's fine that we fail it here.
+		 */
+		nvme_tcp_ofld_complete_timed_out(rq);
+
+		return BLK_EH_DONE;
+	}
+
+	nvme_tcp_ofld_error_recovery(&ctrl->nctrl);
+
+	return BLK_EH_RESET_TIMER;
 }
 
 static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
@@ -959,6 +1123,7 @@ static struct blk_mq_ops nvme_tcp_ofld_mq_ops = {
 	.init_request	= nvme_tcp_ofld_init_request,
 	.exit_request	= nvme_tcp_ofld_exit_request,
 	.init_hctx	= nvme_tcp_ofld_init_hctx,
+	.timeout	= nvme_tcp_ofld_timeout,
 	.map_queues	= nvme_tcp_ofld_map_queues,
 	.poll		= nvme_tcp_ofld_poll,
 };
@@ -969,6 +1134,7 @@ static struct blk_mq_ops nvme_tcp_ofld_admin_mq_ops = {
 	.init_request	= nvme_tcp_ofld_init_request,
 	.exit_request	= nvme_tcp_ofld_exit_request,
 	.init_hctx	= nvme_tcp_ofld_init_admin_hctx,
+	.timeout	= nvme_tcp_ofld_timeout,
 };
 
 static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
@@ -979,6 +1145,7 @@ static const struct nvme_ctrl_ops nvme_tcp_ofld_ctrl_ops = {
 	.reg_read64		= nvmf_reg_read64,
 	.reg_write32		= nvmf_reg_write32,
 	.free_ctrl		= nvme_tcp_ofld_free_ctrl,
+	.submit_async_event     = nvme_tcp_ofld_submit_async_event,
 	.delete_ctrl		= nvme_tcp_ofld_delete_ctrl,
 	.get_address		= nvmf_get_address,
 };
diff --git a/drivers/nvme/host/tcp-offload.h b/drivers/nvme/host/tcp-offload.h
index b3502c01394e..a4c28ddaf3ab 100644
--- a/drivers/nvme/host/tcp-offload.h
+++ b/drivers/nvme/host/tcp-offload.h
@@ -115,6 +115,8 @@ struct nvme_tcp_ofld_ctrl {
 	/* Connectivity params */
 	struct nvme_tcp_ofld_ctrl_con_params conn_params;
 
+	struct nvme_tcp_ofld_req async_req;
+
 	/* Offload device driver context */
 	void *private_data;
 };
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:32 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:32 +0300
Subject: [PATCH v4 09/20] qedn: Add qedn - Marvell's NVMeTCP HW offload device
 driver
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-10-pkushwaha@marvell.com>

From: Shai Malin <smalin at marvell.com>

This patch will present the skeleton of the qedn driver.
The new driver will be added under "drivers/nvme/hw/qedn" and will be
enabled by the Kconfig "Marvell NVM Express over Fabrics TCP offload".

The internal implementation:
- qedn.h:
  Includes all common structs to be used by the qedn offload device driver.

- qedn_main.c
  Includes the qedn_init and qedn_cleanup implementation.
  As part of the qedn init, the driver will register as a pci device and
  will work with the Marvell fastlinQ NICs.
  As part of the probe, the driver will register to the nvme_tcp_offload
  (ULP).

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Arie Gershberg <agershberg at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 MAINTAINERS                      |  10 ++
 drivers/nvme/Kconfig             |   1 +
 drivers/nvme/Makefile            |   1 +
 drivers/nvme/hw/Kconfig          |   8 ++
 drivers/nvme/hw/Makefile         |   3 +
 drivers/nvme/hw/qedn/Makefile    |   5 +
 drivers/nvme/hw/qedn/qedn.h      |  19 +++
 drivers/nvme/hw/qedn/qedn_main.c | 200 +++++++++++++++++++++++++++++++
 8 files changed, 247 insertions(+)
 create mode 100644 drivers/nvme/hw/Kconfig
 create mode 100644 drivers/nvme/hw/Makefile
 create mode 100644 drivers/nvme/hw/qedn/Makefile
 create mode 100644 drivers/nvme/hw/qedn/qedn.h
 create mode 100644 drivers/nvme/hw/qedn/qedn_main.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 01fbebdc7722..207a62b768c5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -15006,6 +15006,16 @@ S:	Supported
 F:	drivers/infiniband/hw/qedr/
 F:	include/uapi/rdma/qedr-abi.h
 
+QLOGIC QL4xxx NVME-TCP-OFFLOAD DRIVER
+M:	Shai Malin <smalin at marvell.com>
+M:	Ariel Elior <aelior at marvell.com>
+L:	linux-nvme at lists.infradead.org
+S:	Supported
+W:	http://git.infradead.org/nvme.git
+T:	git://git.infradead.org/nvme.git
+F:	drivers/nvme/hw/qedn/
+F:	include/linux/qed/
+
 QLOGIC QLA1280 SCSI DRIVER
 M:	Michael Reed <mdr at sgi.com>
 L:	linux-scsi at vger.kernel.org
diff --git a/drivers/nvme/Kconfig b/drivers/nvme/Kconfig
index 87ae409a32b9..827c2c9f0ad1 100644
--- a/drivers/nvme/Kconfig
+++ b/drivers/nvme/Kconfig
@@ -3,5 +3,6 @@ menu "NVME Support"
 
 source "drivers/nvme/host/Kconfig"
 source "drivers/nvme/target/Kconfig"
+source "drivers/nvme/hw/Kconfig"
 
 endmenu
diff --git a/drivers/nvme/Makefile b/drivers/nvme/Makefile
index fb42c44609a8..14c569040ef2 100644
--- a/drivers/nvme/Makefile
+++ b/drivers/nvme/Makefile
@@ -2,3 +2,4 @@
 
 obj-y		+= host/
 obj-y		+= target/
+obj-y		+= hw/
\ No newline at end of file
diff --git a/drivers/nvme/hw/Kconfig b/drivers/nvme/hw/Kconfig
new file mode 100644
index 000000000000..374f1f9dbd3d
--- /dev/null
+++ b/drivers/nvme/hw/Kconfig
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NVME_QEDN
+	tristate "Marvell NVM Express over Fabrics TCP offload"
+	depends on NVME_TCP_OFFLOAD
+	help
+	  This enables the Marvell NVMe TCP offload support (qedn).
+
+	  If unsure, say N.
diff --git a/drivers/nvme/hw/Makefile b/drivers/nvme/hw/Makefile
new file mode 100644
index 000000000000..2f38e0520795
--- /dev/null
+++ b/drivers/nvme/hw/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_NVME_QEDN)		+= qedn/
diff --git a/drivers/nvme/hw/qedn/Makefile b/drivers/nvme/hw/qedn/Makefile
new file mode 100644
index 000000000000..1422cd878680
--- /dev/null
+++ b/drivers/nvme/hw/qedn/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_NVME_QEDN) := qedn.o
+
+qedn-y := qedn_main.o
diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
new file mode 100644
index 000000000000..bcd0748a10fd
--- /dev/null
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#ifndef _QEDN_H_
+#define _QEDN_H_
+
+/* Driver includes */
+#include "../../host/tcp-offload.h"
+
+#define QEDN_MODULE_NAME "qedn"
+
+struct qedn_ctx {
+	struct pci_dev *pdev;
+	struct nvme_tcp_ofld_dev qedn_ofld_dev;
+};
+
+#endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
new file mode 100644
index 000000000000..d81b7c65acdf
--- /dev/null
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -0,0 +1,200 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+ /* Kernel includes */
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+/* Driver includes */
+#include "qedn.h"
+
+#define CHIP_NUM_NVMETCP 0x8194
+
+static struct pci_device_id qedn_pci_tbl[] = {
+	{ PCI_VDEVICE(QLOGIC, CHIP_NUM_NVMETCP), 0 },
+	{0, 0},
+};
+
+static int
+qedn_claim_dev(struct nvme_tcp_ofld_dev *dev,
+	       struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	/* Placeholder - qedn_claim_dev */
+
+	return 0;
+}
+
+static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	/* Placeholder - qedn_setup_ctrl */
+
+	return 0;
+}
+
+static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	/* Placeholder - qedn_release_ctrl */
+
+	return 0;
+}
+
+static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
+			     size_t queue_size)
+{
+	/* Placeholder - qedn_create_queue */
+
+	return 0;
+}
+
+static void qedn_drain_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	/* Placeholder - qedn_drain_queue */
+}
+
+static void qedn_destroy_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	/* Placeholder - qedn_destroy_queue */
+}
+
+static int qedn_poll_queue(struct nvme_tcp_ofld_queue *queue)
+{
+	/*
+	 * Poll queue support will be added as part of future
+	 * enhancements.
+	 */
+
+	return 0;
+}
+
+static int qedn_send_req(struct nvme_tcp_ofld_req *req)
+{
+	/* Placeholder - qedn_send_req */
+
+	return 0;
+}
+
+static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
+	.name = "qedn",
+	.module = THIS_MODULE,
+	.required_opts = NVMF_OPT_TRADDR,
+	.allowed_opts = NVMF_OPT_TRSVCID | NVMF_OPT_NR_WRITE_QUEUES |
+			NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO |
+			NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HOST_IFACE,
+		/* These flags will be as part of future enhancements
+		 *	NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST |
+		 *	NVMF_OPT_NR_POLL_QUEUES | NVMF_OPT_TOS
+		 */
+	.claim_dev = qedn_claim_dev,
+	.setup_ctrl = qedn_setup_ctrl,
+	.release_ctrl = qedn_release_ctrl,
+	.create_queue = qedn_create_queue,
+	.drain_queue = qedn_drain_queue,
+	.destroy_queue = qedn_destroy_queue,
+	.poll_queue = qedn_poll_queue,
+	.send_req = qedn_send_req,
+};
+
+static void __qedn_remove(struct pci_dev *pdev)
+{
+	struct qedn_ctx *qedn = pci_get_drvdata(pdev);
+
+	pr_notice("Starting qedn_remove\n");
+	nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev);
+	kfree(qedn);
+	pr_notice("Ending qedn_remove successfully\n");
+}
+
+static void qedn_remove(struct pci_dev *pdev)
+{
+	__qedn_remove(pdev);
+}
+
+static void qedn_shutdown(struct pci_dev *pdev)
+{
+	__qedn_remove(pdev);
+}
+
+static struct qedn_ctx *qedn_alloc_ctx(struct pci_dev *pdev)
+{
+	struct qedn_ctx *qedn = NULL;
+
+	qedn = kzalloc(sizeof(*qedn), GFP_KERNEL);
+	if (!qedn)
+		return NULL;
+
+	qedn->pdev = pdev;
+	pci_set_drvdata(pdev, qedn);
+
+	return qedn;
+}
+
+static int __qedn_probe(struct pci_dev *pdev)
+{
+	struct qedn_ctx *qedn;
+	int rc;
+
+	pr_notice("Starting qedn probe\n");
+
+	qedn = qedn_alloc_ctx(pdev);
+	if (!qedn)
+		return -ENODEV;
+
+	qedn->qedn_ofld_dev.ops = &qedn_ofld_ops;
+	INIT_LIST_HEAD(&qedn->qedn_ofld_dev.entry);
+	rc = nvme_tcp_ofld_register_dev(&qedn->qedn_ofld_dev);
+	if (rc)
+		goto release_qedn;
+
+	return 0;
+release_qedn:
+	kfree(qedn);
+
+	return rc;
+}
+
+static int qedn_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+	return __qedn_probe(pdev);
+}
+
+static struct pci_driver qedn_pci_driver = {
+	.name     = QEDN_MODULE_NAME,
+	.id_table = qedn_pci_tbl,
+	.probe    = qedn_probe,
+	.remove   = qedn_remove,
+	.shutdown = qedn_shutdown,
+};
+
+static int __init qedn_init(void)
+{
+	int rc;
+
+	rc = pci_register_driver(&qedn_pci_driver);
+	if (rc) {
+		pr_err("Failed to register pci driver\n");
+
+		return -EINVAL;
+	}
+
+	pr_notice("driver loaded successfully\n");
+
+	return 0;
+}
+
+static void __exit qedn_cleanup(void)
+{
+	pci_unregister_driver(&qedn_pci_driver);
+	pr_notice("Unloading qedn ended\n");
+}
+
+module_init(qedn_init);
+module_exit(qedn_cleanup);
+
+MODULE_LICENSE("GPL v2");
+MODULE_SOFTDEP("pre: qede nvme-fabrics nvme-tcp-offload");
+MODULE_DESCRIPTION("Marvell 25/50/100G NVMe-TCP Offload Host Driver");
+MODULE_AUTHOR("Marvell");
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:33 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:33 +0300
Subject: [PATCH v4 10/20] qedn: Add qedn probe
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-11-pkushwaha@marvell.com>

From: Shai Malin <smalin at marvell.com>

This patch introduces the functionality of loading and unloading
physical function.
qedn_probe() loads the offload device PF(physical function), and
initialize the HW and the FW with the PF parameters using the
HW ops->qed_nvmetcp_ops, which are similar to other "qed_*_ops" which
are used by the qede, qedr, qedf and qedi device drivers.
qedn_remove() unloads the offload device PF, re-initialize the HW and
the FW with the PF parameters.

The struct qedn_ctx is per PF container for PF-specific attributes and
resources.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Dean Balandin <dbalandin at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/Kconfig          |   1 +
 drivers/nvme/hw/qedn/qedn.h      |  26 ++++++
 drivers/nvme/hw/qedn/qedn_main.c | 155 ++++++++++++++++++++++++++++++-
 3 files changed, 177 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/hw/Kconfig b/drivers/nvme/hw/Kconfig
index 374f1f9dbd3d..91b1bd6f07d8 100644
--- a/drivers/nvme/hw/Kconfig
+++ b/drivers/nvme/hw/Kconfig
@@ -2,6 +2,7 @@
 config NVME_QEDN
 	tristate "Marvell NVM Express over Fabrics TCP offload"
 	depends on NVME_TCP_OFFLOAD
+	select QED_NVMETCP
 	help
 	  This enables the Marvell NVMe TCP offload support (qedn).
 
diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index bcd0748a10fd..931efc3afbaa 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -6,14 +6,40 @@
 #ifndef _QEDN_H_
 #define _QEDN_H_
 
+#include <linux/qed/qed_if.h>
+#include <linux/qed/qed_nvmetcp_if.h>
+
 /* Driver includes */
 #include "../../host/tcp-offload.h"
 
 #define QEDN_MODULE_NAME "qedn"
 
+#define QEDN_MAX_TASKS_PER_PF (16 * 1024)
+#define QEDN_MAX_CONNS_PER_PF (4 * 1024)
+#define QEDN_FW_CQ_SIZE (4 * 1024)
+#define QEDN_PROTO_CQ_PROD_IDX	0
+#define QEDN_NVMETCP_NUM_FW_CONN_QUEUE_PAGES 2
+
+enum qedn_state {
+	QEDN_STATE_CORE_PROBED = 0,
+	QEDN_STATE_CORE_OPEN,
+	QEDN_STATE_MFW_STATE,
+	QEDN_STATE_REGISTERED_OFFLOAD_DEV,
+	QEDN_STATE_MODULE_REMOVE_ONGOING,
+};
+
 struct qedn_ctx {
 	struct pci_dev *pdev;
+	struct qed_dev *cdev;
+	struct qed_dev_nvmetcp_info dev_info;
 	struct nvme_tcp_ofld_dev qedn_ofld_dev;
+	struct qed_pf_params pf_params;
+
+	/* Accessed with atomic bit ops, used with enum qedn_state */
+	unsigned long state;
+
+	/* Fast path queues */
+	u8 num_fw_cqs;
 };
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index d81b7c65acdf..97591797605e 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -14,6 +14,9 @@
 
 #define CHIP_NUM_NVMETCP 0x8194
 
+const struct qed_nvmetcp_ops *qed_ops;
+
+/* Global context instance */
 static struct pci_device_id qedn_pci_tbl[] = {
 	{ PCI_VDEVICE(QLOGIC, CHIP_NUM_NVMETCP), 0 },
 	{0, 0},
@@ -98,12 +101,109 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
 	.send_req = qedn_send_req,
 };
 
+static inline void qedn_init_pf_struct(struct qedn_ctx *qedn)
+{
+	/* Placeholder - Initialize qedn fields */
+}
+
+static inline void
+qedn_init_core_probe_params(struct qed_probe_params *probe_params)
+{
+	memset(probe_params, 0, sizeof(*probe_params));
+	probe_params->protocol = QED_PROTOCOL_NVMETCP;
+	probe_params->is_vf = false;
+	probe_params->recov_in_prog = 0;
+}
+
+static inline int qedn_core_probe(struct qedn_ctx *qedn)
+{
+	struct qed_probe_params probe_params;
+	int rc = 0;
+
+	qedn_init_core_probe_params(&probe_params);
+	pr_info("Starting QED probe\n");
+	qedn->cdev = qed_ops->common->probe(qedn->pdev, &probe_params);
+	if (!qedn->cdev) {
+		rc = -ENODEV;
+		pr_err("QED probe failed\n");
+	}
+
+	return rc;
+}
+
+static int qedn_set_nvmetcp_pf_param(struct qedn_ctx *qedn)
+{
+	u32 fw_conn_queue_pages = QEDN_NVMETCP_NUM_FW_CONN_QUEUE_PAGES;
+	struct qed_nvmetcp_pf_params *pf_params;
+
+	pf_params = &qedn->pf_params.nvmetcp_pf_params;
+	memset(pf_params, 0, sizeof(*pf_params));
+	qedn->num_fw_cqs = min_t(u8, qedn->dev_info.num_cqs, num_online_cpus());
+
+	pf_params->num_cons = QEDN_MAX_CONNS_PER_PF;
+	pf_params->num_tasks = QEDN_MAX_TASKS_PER_PF;
+
+	/* Placeholder - Initialize function level queues */
+
+	/* Placeholder - Initialize TCP params */
+
+	/* Queues */
+	pf_params->num_sq_pages_in_ring = fw_conn_queue_pages;
+	pf_params->num_r2tq_pages_in_ring = fw_conn_queue_pages;
+	pf_params->num_uhq_pages_in_ring = fw_conn_queue_pages;
+	pf_params->num_queues = qedn->num_fw_cqs;
+	pf_params->cq_num_entries = QEDN_FW_CQ_SIZE;
+
+	/* the CQ SB pi */
+	pf_params->gl_rq_pi = QEDN_PROTO_CQ_PROD_IDX;
+
+	return 0;
+}
+
+static inline int qedn_slowpath_start(struct qedn_ctx *qedn)
+{
+	struct qed_slowpath_params sp_params = {};
+	int rc = 0;
+
+	/* Start the Slowpath-process */
+	sp_params.int_mode = QED_INT_MODE_MSIX;
+	strscpy(sp_params.name, "qedn NVMeTCP", QED_DRV_VER_STR_SIZE);
+	rc = qed_ops->common->slowpath_start(qedn->cdev, &sp_params);
+	if (rc)
+		pr_err("Cannot start slowpath\n");
+
+	return rc;
+}
+
 static void __qedn_remove(struct pci_dev *pdev)
 {
 	struct qedn_ctx *qedn = pci_get_drvdata(pdev);
+	int rc;
+
+	pr_notice("Starting qedn_remove: abs PF id=%u\n",
+		  qedn->dev_info.common.abs_pf_id);
+
+	if (test_and_set_bit(QEDN_STATE_MODULE_REMOVE_ONGOING, &qedn->state)) {
+		pr_err("Remove already ongoing\n");
+
+		return;
+	}
+
+	if (test_and_clear_bit(QEDN_STATE_REGISTERED_OFFLOAD_DEV, &qedn->state))
+		nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev);
+
+	if (test_and_clear_bit(QEDN_STATE_MFW_STATE, &qedn->state)) {
+		rc = qed_ops->common->update_drv_state(qedn->cdev, false);
+		if (rc)
+			pr_err("Failed to send drv state to MFW\n");
+	}
+
+	if (test_and_clear_bit(QEDN_STATE_CORE_OPEN, &qedn->state))
+		qed_ops->common->slowpath_stop(qedn->cdev);
+
+	if (test_and_clear_bit(QEDN_STATE_CORE_PROBED, &qedn->state))
+		qed_ops->common->remove(qedn->cdev);
 
-	pr_notice("Starting qedn_remove\n");
-	nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev);
 	kfree(qedn);
 	pr_notice("Ending qedn_remove successfully\n");
 }
@@ -143,15 +243,52 @@ static int __qedn_probe(struct pci_dev *pdev)
 	if (!qedn)
 		return -ENODEV;
 
+	qedn_init_pf_struct(qedn);
+
+	/* QED probe */
+	rc = qedn_core_probe(qedn);
+	if (rc)
+		goto exit_probe_and_release_mem;
+
+	set_bit(QEDN_STATE_CORE_PROBED, &qedn->state);
+
+	rc = qed_ops->fill_dev_info(qedn->cdev, &qedn->dev_info);
+	if (rc) {
+		pr_err("fill_dev_info failed\n");
+		goto exit_probe_and_release_mem;
+	}
+
+	rc = qedn_set_nvmetcp_pf_param(qedn);
+	if (rc)
+		goto exit_probe_and_release_mem;
+
+	qed_ops->common->update_pf_params(qedn->cdev, &qedn->pf_params);
+	rc = qedn_slowpath_start(qedn);
+	if (rc)
+		goto exit_probe_and_release_mem;
+
+	set_bit(QEDN_STATE_CORE_OPEN, &qedn->state);
+
+	rc = qed_ops->common->update_drv_state(qedn->cdev, true);
+	if (rc) {
+		pr_err("Failed to send drv state to MFW\n");
+		goto exit_probe_and_release_mem;
+	}
+
+	set_bit(QEDN_STATE_MFW_STATE, &qedn->state);
+
 	qedn->qedn_ofld_dev.ops = &qedn_ofld_ops;
 	INIT_LIST_HEAD(&qedn->qedn_ofld_dev.entry);
 	rc = nvme_tcp_ofld_register_dev(&qedn->qedn_ofld_dev);
 	if (rc)
-		goto release_qedn;
+		goto exit_probe_and_release_mem;
+
+	set_bit(QEDN_STATE_REGISTERED_OFFLOAD_DEV, &qedn->state);
 
 	return 0;
-release_qedn:
-	kfree(qedn);
+exit_probe_and_release_mem:
+	__qedn_remove(pdev);
+	pr_err("probe ended with error\n");
 
 	return rc;
 }
@@ -173,6 +310,13 @@ static int __init qedn_init(void)
 {
 	int rc;
 
+	qed_ops = qed_get_nvmetcp_ops();
+	if (!qed_ops) {
+		pr_err("Failed to get QED NVMeTCP ops\n");
+
+		return -EINVAL;
+	}
+
 	rc = pci_register_driver(&qedn_pci_driver);
 	if (rc) {
 		pr_err("Failed to register pci driver\n");
@@ -188,6 +332,7 @@ static int __init qedn_init(void)
 static void __exit qedn_cleanup(void)
 {
 	pci_unregister_driver(&qedn_pci_driver);
+	qed_put_nvmetcp_ops();
 	pr_notice("Unloading qedn ended\n");
 }
 
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:34 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:34 +0300
Subject: [PATCH v4 11/20] qedn: Add qedn_claim_dev API support
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-12-pkushwaha@marvell.com>

From: Nikolay Assa <nassa at marvell.com>

This patch introduces the qedn_claim_dev() network service which the
offload device (qedn) is using through the paired net-device (qede).
qedn_claim_dev() returns true if the IP addr(IPv4 or IPv6) of the target
server is reachable via the net-device which is paired with the
offloaded device.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Nikolay Assa <nassa at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |  4 +++
 drivers/nvme/hw/qedn/qedn_main.c | 55 ++++++++++++++++++++++++++++++--
 2 files changed, 56 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 931efc3afbaa..0ce1e19d1ba8 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -8,6 +8,10 @@
 
 #include <linux/qed/qed_if.h>
 #include <linux/qed/qed_nvmetcp_if.h>
+#include <linux/qed/qed_nvmetcp_ip_services_if.h>
+#include <linux/qed/qed_chain.h>
+#include <linux/qed/storage_common.h>
+#include <linux/qed/nvmetcp_common.h>
 
 /* Driver includes */
 #include "../../host/tcp-offload.h"
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 97591797605e..78bc9fe17e7b 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -22,13 +22,62 @@ static struct pci_device_id qedn_pci_tbl[] = {
 	{0, 0},
 };
 
+static int
+qedn_find_dev(struct nvme_tcp_ofld_dev *dev,
+	      struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	struct nvme_tcp_ofld_ctrl_con_params *conn_params;
+	struct pci_dev *qede_pdev = NULL;
+	struct sockaddr remote_mac_addr;
+	struct net_device *ndev = NULL;
+	u16 vlan_id = 0;
+	int rc = 0;
+
+	conn_params = &ctrl->conn_params;
+
+	/* qedn utilizes host network stack through paired qede device for
+	 * non-offload traffic. First we verify there is valid route to remote
+	 * peer.
+	 */
+	if (conn_params->remote_ip_addr.ss_family == AF_INET) {
+		rc = qed_route_ipv4(&conn_params->local_ip_addr,
+				    &conn_params->remote_ip_addr,
+				    &remote_mac_addr, &ndev);
+	} else if (conn_params->remote_ip_addr.ss_family == AF_INET6) {
+		rc = qed_route_ipv6(&conn_params->local_ip_addr,
+				    &conn_params->remote_ip_addr,
+				    &remote_mac_addr, &ndev);
+	} else {
+		pr_err("address family %d not supported\n",
+		       conn_params->remote_ip_addr.ss_family);
+
+		return false;
+	}
+
+	if (rc)
+		return false;
+
+	if (!ctrl->private_data && ctrl->ndev &&
+	    strcmp(ctrl->ndev->name, ndev->name))
+		return false;
+
+	ctrl->ndev = ndev;
+
+	qed_vlan_get_ndev(&ctrl->ndev, &vlan_id);
+
+	/* route found through ndev - validate this is qede*/
+	qede_pdev = qed_validate_ndev(ctrl->ndev);
+	if (!qede_pdev)
+		return false;
+
+	return true;
+}
+
 static int
 qedn_claim_dev(struct nvme_tcp_ofld_dev *dev,
 	       struct nvme_tcp_ofld_ctrl *ctrl)
 {
-	/* Placeholder - qedn_claim_dev */
-
-	return 0;
+	return qedn_find_dev(dev, ctrl);
 }
 
 static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:35 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:35 +0300
Subject: [PATCH v4 12/20] qedn: Add IRQ and fast-path resources initializations
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-13-pkushwaha@marvell.com>

From: Shai Malin <smalin at marvell.com>

This patch will present the adding of qedn_fp_queue - this is a per cpu
core element which handles all of the connections on that cpu core.
The qedn_fp_queue will handle a group of connections (NVMeoF QPs) which
are handled on the same cpu core, and will only use the same FW-driver
resources with no need to be related to the same NVMeoF controller.

The per qedn_fq_queue resources are the FW CQ and FW status block:
- The FW CQ will be used for the FW to notify the driver that the
  the exchange has ended and the FW will pass the incoming NVMeoF CQE
  (if exist) to the driver.
- FW status block - which is used for the FW to notify the driver with
  the producer update of the FW CQE chain.

The FW fast-path queues are based on qed_chain.h

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |  25 +++
 drivers/nvme/hw/qedn/qedn_main.c | 289 ++++++++++++++++++++++++++++++-
 2 files changed, 311 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 0ce1e19d1ba8..edb0836bca87 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -24,17 +24,39 @@
 #define QEDN_PROTO_CQ_PROD_IDX	0
 #define QEDN_NVMETCP_NUM_FW_CONN_QUEUE_PAGES 2
 
+#define QEDN_PAGE_SIZE	4096 /* FW page size - Configurable */
+#define QEDN_IRQ_NAME_LEN 24
+#define QEDN_IRQ_NO_FLAGS 0
+
+#define QEDN_TCP_RTO_DEFAULT 280
+
 enum qedn_state {
 	QEDN_STATE_CORE_PROBED = 0,
 	QEDN_STATE_CORE_OPEN,
 	QEDN_STATE_MFW_STATE,
+	QEDN_STATE_NVMETCP_OPEN,
+	QEDN_STATE_IRQ_SET,
+	QEDN_STATE_FP_WORK_THREAD_SET,
 	QEDN_STATE_REGISTERED_OFFLOAD_DEV,
 	QEDN_STATE_MODULE_REMOVE_ONGOING,
 };
 
+/* Per CPU core params */
+struct qedn_fp_queue {
+	struct qed_chain cq_chain;
+	u16 *cq_prod;
+	struct mutex cq_mutex; /* cq handler mutex */
+	struct qedn_ctx	*qedn;
+	struct qed_sb_info *sb_info;
+	unsigned int cpu;
+	u16 sb_id;
+	char irqname[QEDN_IRQ_NAME_LEN];
+};
+
 struct qedn_ctx {
 	struct pci_dev *pdev;
 	struct qed_dev *cdev;
+	struct qed_int_info int_info;
 	struct qed_dev_nvmetcp_info dev_info;
 	struct nvme_tcp_ofld_dev qedn_ofld_dev;
 	struct qed_pf_params pf_params;
@@ -44,6 +66,9 @@ struct qedn_ctx {
 
 	/* Fast path queues */
 	u8 num_fw_cqs;
+	struct qedn_fp_queue *fp_q_arr;
+	struct nvmetcp_glbl_queue_entry *fw_cq_array_virt;
+	dma_addr_t fw_cq_array_phy; /* Physical address of fw_cq_array_virt */
 };
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 78bc9fe17e7b..6c0f36f7d9d1 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -150,6 +150,104 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
 	.send_req = qedn_send_req,
 };
 
+/* Fastpath IRQ handler */
+static irqreturn_t qedn_irq_handler(int irq, void *dev_id)
+{
+	/* Placeholder */
+
+	return IRQ_HANDLED;
+}
+
+static void qedn_sync_free_irqs(struct qedn_ctx *qedn)
+{
+	u16 vector_idx;
+	int i;
+
+	for (i = 0; i < qedn->num_fw_cqs; i++) {
+		vector_idx = i * qedn->dev_info.common.num_hwfns +
+			     qed_ops->common->get_affin_hwfn_idx(qedn->cdev);
+		synchronize_irq(qedn->int_info.msix[vector_idx].vector);
+		irq_set_affinity_hint(qedn->int_info.msix[vector_idx].vector,
+				      NULL);
+		free_irq(qedn->int_info.msix[vector_idx].vector,
+			 &qedn->fp_q_arr[i]);
+	}
+
+	qedn->int_info.used_cnt = 0;
+	qed_ops->common->set_fp_int(qedn->cdev, 0);
+}
+
+static int qedn_request_msix_irq(struct qedn_ctx *qedn)
+{
+	struct pci_dev *pdev = qedn->pdev;
+	struct qedn_fp_queue *fp_q = NULL;
+	int i, rc, cpu;
+	u16 vector_idx;
+	u32 vector;
+
+	/* numa-awareness will be added in future enhancements */
+	cpu = cpumask_first(cpu_online_mask);
+	for (i = 0; i < qedn->num_fw_cqs; i++) {
+		fp_q = &qedn->fp_q_arr[i];
+		vector_idx = i * qedn->dev_info.common.num_hwfns +
+			     qed_ops->common->get_affin_hwfn_idx(qedn->cdev);
+		vector = qedn->int_info.msix[vector_idx].vector;
+		sprintf(fp_q->irqname, "qedn_queue_%x.%x.%x_%d",
+			pdev->bus->number, PCI_SLOT(pdev->devfn),
+			PCI_FUNC(pdev->devfn), i);
+		rc = request_irq(vector, qedn_irq_handler, QEDN_IRQ_NO_FLAGS,
+				 fp_q->irqname, fp_q);
+		if (rc) {
+			pr_err("request_irq failed.\n");
+			qedn_sync_free_irqs(qedn);
+
+			return rc;
+		}
+
+		fp_q->cpu = cpu;
+		qedn->int_info.used_cnt++;
+		rc = irq_set_affinity_hint(vector, get_cpu_mask(cpu));
+		cpu = cpumask_next_wrap(cpu, cpu_online_mask, -1, false);
+	}
+
+	return 0;
+}
+
+static int qedn_setup_irq(struct qedn_ctx *qedn)
+{
+	int rc = 0;
+	u8 rval;
+
+	rval = qed_ops->common->set_fp_int(qedn->cdev, qedn->num_fw_cqs);
+	if (rval < qedn->num_fw_cqs) {
+		qedn->num_fw_cqs = rval;
+		if (rval == 0) {
+			pr_err("set_fp_int return 0 IRQs\n");
+
+			return -ENODEV;
+		}
+	}
+
+	rc = qed_ops->common->get_fp_int(qedn->cdev, &qedn->int_info);
+	if (rc) {
+		pr_err("get_fp_int failed\n");
+		goto exit_setup_int;
+	}
+
+	if (qedn->int_info.msix_cnt) {
+		rc = qedn_request_msix_irq(qedn);
+		goto exit_setup_int;
+	} else {
+		pr_err("msix_cnt = 0\n");
+		rc = -EINVAL;
+		goto exit_setup_int;
+	}
+
+exit_setup_int:
+
+	return rc;
+}
+
 static inline void qedn_init_pf_struct(struct qedn_ctx *qedn)
 {
 	/* Placeholder - Initialize qedn fields */
@@ -180,21 +278,174 @@ static inline int qedn_core_probe(struct qedn_ctx *qedn)
 	return rc;
 }
 
+static void qedn_free_function_queues(struct qedn_ctx *qedn)
+{
+	struct qed_sb_info *sb_info = NULL;
+	struct qedn_fp_queue *fp_q;
+	int i;
+
+	/* Free workqueues */
+
+	/* Free the fast path queues*/
+	for (i = 0; i < qedn->num_fw_cqs; i++) {
+		fp_q = &qedn->fp_q_arr[i];
+
+		/* Free SB */
+		sb_info = fp_q->sb_info;
+		if (sb_info->sb_virt) {
+			qed_ops->common->sb_release(qedn->cdev, sb_info,
+						    fp_q->sb_id,
+						    QED_SB_TYPE_STORAGE);
+			dma_free_coherent(&qedn->pdev->dev,
+					  sizeof(*sb_info->sb_virt),
+					  (void *)sb_info->sb_virt,
+					  sb_info->sb_phys);
+			memset(sb_info, 0, sizeof(*sb_info));
+			kfree(sb_info);
+			fp_q->sb_info = NULL;
+		}
+
+		qed_ops->common->chain_free(qedn->cdev, &fp_q->cq_chain);
+	}
+
+	if (qedn->fw_cq_array_virt)
+		dma_free_coherent(&qedn->pdev->dev,
+				  qedn->num_fw_cqs * sizeof(u64),
+				  qedn->fw_cq_array_virt,
+				  qedn->fw_cq_array_phy);
+	kfree(qedn->fp_q_arr);
+	qedn->fp_q_arr = NULL;
+}
+
+static int qedn_alloc_and_init_sb(struct qedn_ctx *qedn,
+				  struct qed_sb_info *sb_info, u16 sb_id)
+{
+	int rc = 0;
+
+	sb_info->sb_virt = dma_alloc_coherent(&qedn->pdev->dev,
+					      sizeof(struct status_block_e4),
+					      &sb_info->sb_phys, GFP_KERNEL);
+	if (!sb_info->sb_virt) {
+		pr_err("Status block allocation failed\n");
+
+		return -ENOMEM;
+	}
+
+	rc = qed_ops->common->sb_init(qedn->cdev, sb_info, sb_info->sb_virt,
+				      sb_info->sb_phys, sb_id,
+				      QED_SB_TYPE_STORAGE);
+	if (rc) {
+		pr_err("Status block initialization failed\n");
+
+		return rc;
+	}
+
+	return 0;
+}
+
+static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
+{
+	struct qed_chain_init_params chain_params = {};
+	struct status_block_e4 *sb = NULL;
+	struct qedn_fp_queue *fp_q = NULL;
+	int rc = 0, arr_size;
+	u64 cq_phy_addr;
+	int i;
+
+	/* Place holder - IO-path workqueues */
+
+	qedn->fp_q_arr = kcalloc(qedn->num_fw_cqs,
+				 sizeof(struct qedn_fp_queue), GFP_KERNEL);
+	if (!qedn->fp_q_arr)
+		return -ENOMEM;
+
+	arr_size = qedn->num_fw_cqs * sizeof(struct nvmetcp_glbl_queue_entry);
+	qedn->fw_cq_array_virt = dma_alloc_coherent(&qedn->pdev->dev,
+						    arr_size,
+						    &qedn->fw_cq_array_phy,
+						    GFP_KERNEL);
+	if (!qedn->fw_cq_array_virt) {
+		rc = -ENOMEM;
+		goto mem_alloc_failure;
+	}
+
+	/* placeholder - create task pools */
+
+	for (i = 0; i < qedn->num_fw_cqs; i++) {
+		fp_q = &qedn->fp_q_arr[i];
+		mutex_init(&fp_q->cq_mutex);
+
+		/* FW CQ */
+		chain_params.intended_use = QED_CHAIN_USE_TO_CONSUME,
+		chain_params.mode = QED_CHAIN_MODE_PBL,
+		chain_params.cnt_type = QED_CHAIN_CNT_TYPE_U16,
+		chain_params.num_elems = QEDN_FW_CQ_SIZE;
+		/* Placeholder - sizeof(struct nvmetcp_fw_cqe)*/
+		chain_params.elem_size = 64;
+
+		rc = qed_ops->common->chain_alloc(qedn->cdev,
+						  &fp_q->cq_chain,
+						  &chain_params);
+		if (rc) {
+			pr_err("CQ chain pci_alloc_consistent fail\n");
+			goto mem_alloc_failure;
+		}
+
+		cq_phy_addr = qed_chain_get_pbl_phys(&fp_q->cq_chain);
+		qedn->fw_cq_array_virt[i].cq_pbl_addr.hi = PTR_HI(cq_phy_addr);
+		qedn->fw_cq_array_virt[i].cq_pbl_addr.lo = PTR_LO(cq_phy_addr);
+
+		/* SB */
+		fp_q->sb_info = kzalloc(sizeof(*fp_q->sb_info), GFP_KERNEL);
+		if (!fp_q->sb_info)
+			goto mem_alloc_failure;
+
+		fp_q->sb_id = i;
+		rc = qedn_alloc_and_init_sb(qedn, fp_q->sb_info, fp_q->sb_id);
+		if (rc) {
+			pr_err("SB allocation and initialization failed.\n");
+			goto mem_alloc_failure;
+		}
+
+		sb = fp_q->sb_info->sb_virt;
+		fp_q->cq_prod = (u16 *)&sb->pi_array[QEDN_PROTO_CQ_PROD_IDX];
+		fp_q->qedn = qedn;
+
+		/* Placeholder - Init IO-path workqueue */
+
+		/* Placeholder - Init IO-path resources */
+	}
+
+	return 0;
+
+mem_alloc_failure:
+	pr_err("Function allocation failed\n");
+	qedn_free_function_queues(qedn);
+
+	return rc;
+}
+
 static int qedn_set_nvmetcp_pf_param(struct qedn_ctx *qedn)
 {
 	u32 fw_conn_queue_pages = QEDN_NVMETCP_NUM_FW_CONN_QUEUE_PAGES;
 	struct qed_nvmetcp_pf_params *pf_params;
+	int rc;
 
 	pf_params = &qedn->pf_params.nvmetcp_pf_params;
 	memset(pf_params, 0, sizeof(*pf_params));
 	qedn->num_fw_cqs = min_t(u8, qedn->dev_info.num_cqs, num_online_cpus());
+	pr_info("Num qedn FW CQs %u\n", qedn->num_fw_cqs);
 
 	pf_params->num_cons = QEDN_MAX_CONNS_PER_PF;
 	pf_params->num_tasks = QEDN_MAX_TASKS_PER_PF;
 
-	/* Placeholder - Initialize function level queues */
+	rc = qedn_alloc_function_queues(qedn);
+	if (rc) {
+		pr_err("Global queue allocation failed.\n");
+		goto err_alloc_mem;
+	}
 
-	/* Placeholder - Initialize TCP params */
+	set_bit(QEDN_STATE_FP_WORK_THREAD_SET, &qedn->state);
 
 	/* Queues */
 	pf_params->num_sq_pages_in_ring = fw_conn_queue_pages;
@@ -202,11 +453,14 @@ static int qedn_set_nvmetcp_pf_param(struct qedn_ctx *qedn)
 	pf_params->num_uhq_pages_in_ring = fw_conn_queue_pages;
 	pf_params->num_queues = qedn->num_fw_cqs;
 	pf_params->cq_num_entries = QEDN_FW_CQ_SIZE;
+	pf_params->glbl_q_params_addr = qedn->fw_cq_array_phy;
 
 	/* the CQ SB pi */
 	pf_params->gl_rq_pi = QEDN_PROTO_CQ_PROD_IDX;
 
-	return 0;
+err_alloc_mem:
+
+	return rc;
 }
 
 static inline int qedn_slowpath_start(struct qedn_ctx *qedn)
@@ -241,6 +495,12 @@ static void __qedn_remove(struct pci_dev *pdev)
 	if (test_and_clear_bit(QEDN_STATE_REGISTERED_OFFLOAD_DEV, &qedn->state))
 		nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev);
 
+	if (test_and_clear_bit(QEDN_STATE_IRQ_SET, &qedn->state))
+		qedn_sync_free_irqs(qedn);
+
+	if (test_and_clear_bit(QEDN_STATE_NVMETCP_OPEN, &qedn->state))
+		qed_ops->stop(qedn->cdev);
+
 	if (test_and_clear_bit(QEDN_STATE_MFW_STATE, &qedn->state)) {
 		rc = qed_ops->common->update_drv_state(qedn->cdev, false);
 		if (rc)
@@ -250,6 +510,9 @@ static void __qedn_remove(struct pci_dev *pdev)
 	if (test_and_clear_bit(QEDN_STATE_CORE_OPEN, &qedn->state))
 		qed_ops->common->slowpath_stop(qedn->cdev);
 
+	if (test_and_clear_bit(QEDN_STATE_FP_WORK_THREAD_SET, &qedn->state))
+		qedn_free_function_queues(qedn);
+
 	if (test_and_clear_bit(QEDN_STATE_CORE_PROBED, &qedn->state))
 		qed_ops->common->remove(qedn->cdev);
 
@@ -318,6 +581,25 @@ static int __qedn_probe(struct pci_dev *pdev)
 
 	set_bit(QEDN_STATE_CORE_OPEN, &qedn->state);
 
+	rc = qedn_setup_irq(qedn);
+	if (rc)
+		goto exit_probe_and_release_mem;
+
+	set_bit(QEDN_STATE_IRQ_SET, &qedn->state);
+
+	/* NVMeTCP start HW PF */
+	rc = qed_ops->start(qedn->cdev,
+			    NULL /* Placeholder for FW IO-path resources */,
+			    qedn,
+			    NULL /* Placeholder for FW Event callback */);
+	if (rc) {
+		rc = -ENODEV;
+		pr_err("Cannot start NVMeTCP Function\n");
+		goto exit_probe_and_release_mem;
+	}
+
+	set_bit(QEDN_STATE_NVMETCP_OPEN, &qedn->state);
+
 	rc = qed_ops->common->update_drv_state(qedn->cdev, true);
 	if (rc) {
 		pr_err("Failed to send drv state to MFW\n");
@@ -326,6 +608,7 @@ static int __qedn_probe(struct pci_dev *pdev)
 
 	set_bit(QEDN_STATE_MFW_STATE, &qedn->state);
 
+	qedn->qedn_ofld_dev.num_hw_vectors = qedn->num_fw_cqs;
 	qedn->qedn_ofld_dev.ops = &qedn_ofld_ops;
 	INIT_LIST_HEAD(&qedn->qedn_ofld_dev.entry);
 	rc = nvme_tcp_ofld_register_dev(&qedn->qedn_ofld_dev);
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:36 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:36 +0300
Subject: [PATCH v4 13/20] qedn: Add connection-level slowpath functionality
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-14-pkushwaha@marvell.com>

This patch will present the connection (queue) level slowpath
implementation relevant for create_queue flow.

The internal implementation:
- Add per controller slowpath workqeueue via pre_setup_ctrl

- qedn_main.c:
  Includes qedn's implementation of the create_queue op.

- qedn_conn.c will include main slowpath connection level functions,
  including:
    1. Per-queue resources allocation.
    2. Creating a new connection.
    3. Offloading the connection to the FW for TCP handshake.
    4. Destroy of a connection.
    5. Support of delete and free controller.
    6. TCP port management via qed_fetch_tcp_port, qed_return_tcp_port

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/Makefile    |   5 +-
 drivers/nvme/hw/qedn/qedn.h      | 179 ++++++++++
 drivers/nvme/hw/qedn/qedn_conn.c | 564 +++++++++++++++++++++++++++++++
 drivers/nvme/hw/qedn/qedn_main.c | 210 +++++++++++-
 4 files changed, 948 insertions(+), 10 deletions(-)
 create mode 100644 drivers/nvme/hw/qedn/qedn_conn.c

diff --git a/drivers/nvme/hw/qedn/Makefile b/drivers/nvme/hw/qedn/Makefile
index 1422cd878680..ece84772d317 100644
--- a/drivers/nvme/hw/qedn/Makefile
+++ b/drivers/nvme/hw/qedn/Makefile
@@ -1,5 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0
 
-obj-$(CONFIG_NVME_QEDN) := qedn.o
-
-qedn-y := qedn_main.o
+obj-$(CONFIG_NVME_QEDN) += qedn.o
+qedn-y := qedn_main.o qedn_conn.o
diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index edb0836bca87..38ab4ff88999 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -6,6 +6,7 @@
 #ifndef _QEDN_H_
 #define _QEDN_H_
 
+#include <linux/qed/common_hsi.h>
 #include <linux/qed/qed_if.h>
 #include <linux/qed/qed_nvmetcp_if.h>
 #include <linux/qed/qed_nvmetcp_ip_services_if.h>
@@ -28,7 +29,41 @@
 #define QEDN_IRQ_NAME_LEN 24
 #define QEDN_IRQ_NO_FLAGS 0
 
+/* Destroy connection defines */
+#define QEDN_NON_ABORTIVE_TERMINATION 0
+#define QEDN_ABORTIVE_TERMINATION 1
+
+/*
+ * TCP offload stack default configurations and defines.
+ * Future enhancements will allow controlling the configurable
+ * parameters via devlink.
+ */
 #define QEDN_TCP_RTO_DEFAULT 280
+#define QEDN_TCP_ECN_EN 0
+#define QEDN_TCP_TS_EN 0
+#define QEDN_TCP_DA_EN 0
+#define QEDN_TCP_KA_EN 0
+#define QEDN_TCP_TOS 0
+#define QEDN_TCP_TTL 0xfe
+#define QEDN_TCP_FLOW_LABEL 0
+#define QEDN_TCP_KA_TIMEOUT 7200000
+#define QEDN_TCP_KA_INTERVAL 10000
+#define QEDN_TCP_KA_MAX_PROBE_COUNT 10
+#define QEDN_TCP_MAX_RT_TIME 30000
+#define QEDN_TCP_MAX_CWND 4
+#define QEDN_TCP_RCV_WND_SCALE 2
+#define QEDN_TCP_TS_OPTION_LEN 12
+
+/* SP Work queue defines */
+#define QEDN_SP_WORKQUEUE "qedn_sp_wq"
+#define QEDN_SP_WORKQUEUE_MAX_ACTIVE 1
+
+#define QEDN_HOST_MAX_SQ_SIZE (512)
+#define QEDN_SQ_SIZE (2 * QEDN_HOST_MAX_SQ_SIZE)
+
+/* Timeouts and delay constants */
+#define QEDN_WAIT_CON_ESTABLSH_TMO 10000 /* 10 seconds */
+#define QEDN_RLS_CONS_TMO 5000 /* 5 sec */
 
 enum qedn_state {
 	QEDN_STATE_CORE_PROBED = 0,
@@ -64,6 +99,12 @@ struct qedn_ctx {
 	/* Accessed with atomic bit ops, used with enum qedn_state */
 	unsigned long state;
 
+	u8 local_mac_addr[ETH_ALEN];
+	u16 mtu;
+
+	/* Connections */
+	DECLARE_HASHTABLE(conn_ctx_hash, 16);
+
 	/* Fast path queues */
 	u8 num_fw_cqs;
 	struct qedn_fp_queue *fp_q_arr;
@@ -71,4 +112,142 @@ struct qedn_ctx {
 	dma_addr_t fw_cq_array_phy; /* Physical address of fw_cq_array_virt */
 };
 
+struct qedn_endpoint {
+	/* FW Params */
+	struct qed_chain fw_sq_chain;
+	struct nvmetcp_db_data db_data;
+	void __iomem *p_doorbell;
+
+	/* TCP Params */
+	__be32 dst_addr[4]; /* In network order */
+	__be32 src_addr[4]; /* In network order */
+	u16 src_port;
+	u16 dst_port;
+	u16 vlan_id;
+	u8 src_mac[ETH_ALEN];
+	u8 dst_mac[ETH_ALEN];
+	u8 ip_type;
+};
+
+enum sp_work_agg_action {
+	CREATE_CONNECTION = 0,
+	SEND_ICREQ,
+	HANDLE_ICRESP,
+	DESTROY_CONNECTION,
+};
+
+enum qedn_ctrl_agg_state {
+	QEDN_CTRL_SET_TO_OFLD_CTRL = 0, /* CTRL set to OFLD_CTRL */
+	QEDN_STATE_SP_WORK_THREAD_SET, /* slow patch WQ was created*/
+	LLH_FILTER, /* LLH filter added */
+	QEDN_RECOVERY,
+	ADMINQ_CONNECTED, /* At least one connection has attempted offload */
+	ERR_FLOW,
+};
+
+enum qedn_ctrl_sp_wq_state {
+	QEDN_CTRL_STATE_UNINITIALIZED = 0,
+	QEDN_CTRL_STATE_FREE_CTRL,
+	QEDN_CTRL_STATE_CTRL_ERR,
+};
+
+/* Any change to this enum requires an update of qedn_conn_state_str */
+enum qedn_conn_state {
+	CONN_STATE_CONN_IDLE = 0,
+	CONN_STATE_CREATE_CONNECTION,
+	CONN_STATE_WAIT_FOR_CONNECT_DONE,
+	CONN_STATE_OFFLOAD_COMPLETE,
+	CONN_STATE_WAIT_FOR_UPDATE_EQE,
+	CONN_STATE_WAIT_FOR_IC_COMP,
+	CONN_STATE_NVMETCP_CONN_ESTABLISHED,
+	CONN_STATE_DESTROY_CONNECTION,
+	CONN_STATE_WAIT_FOR_DESTROY_DONE,
+	CONN_STATE_DESTROY_COMPLETE
+};
+
+struct qedn_ctrl {
+	struct list_head glb_entry;
+	struct list_head pf_entry;
+
+	struct qedn_ctx *qedn;
+	struct nvme_tcp_ofld_queue *queue;
+	struct nvme_tcp_ofld_ctrl *ctrl;
+
+	struct sockaddr remote_mac_addr;
+	u16 vlan_id;
+
+	struct workqueue_struct *sp_wq;
+	enum qedn_ctrl_sp_wq_state sp_wq_state;
+
+	struct work_struct sp_wq_entry;
+
+	struct qedn_llh_filter *llh_filter;
+
+	unsigned long agg_state;
+
+	atomic_t host_num_active_conns;
+};
+
+/* Connection level struct */
+struct qedn_conn_ctx {
+	struct qedn_ctx *qedn;
+	struct nvme_tcp_ofld_queue *queue;
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	u32 conn_handle;
+	u32 fw_cid;
+
+	atomic_t est_conn_indicator;
+	atomic_t destroy_conn_indicator;
+	wait_queue_head_t conn_waitq;
+
+	struct work_struct sp_wq_entry;
+
+	/* Connection aggregative state.
+	 * Can have different states independently.
+	 */
+	unsigned long agg_work_action;
+
+	struct hlist_node hash_node;
+	struct nvmetcp_host_cccid_itid_entry *host_cccid_itid;
+	dma_addr_t host_cccid_itid_phy_addr;
+	struct qedn_endpoint ep;
+	int abrt_flag;
+
+	/* Connection resources - turned on to indicate what resource was
+	 * allocated, to that it can later be released.
+	 */
+	unsigned long resrc_state;
+
+	/* Connection state */
+	spinlock_t conn_state_lock;
+	enum qedn_conn_state state;
+
+	size_t sq_depth;
+
+	/* "dummy" socket */
+	struct socket *sock;
+};
+
+enum qedn_conn_resources_state {
+	QEDN_CONN_RESRC_FW_SQ,
+	QEDN_CONN_RESRC_ACQUIRE_CONN,
+	QEDN_CONN_RESRC_CCCID_ITID_MAP,
+	QEDN_CONN_RESRC_TCP_PORT,
+	QEDN_CONN_RESRC_DB_ADD,
+	QEDN_CONN_RESRC_MAX = 64
+};
+
+struct qedn_conn_ctx *qedn_get_conn_hash(struct qedn_ctx *qedn, u16 icid);
+int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data);
+void qedn_sp_wq_handler(struct work_struct *work);
+void qedn_set_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit);
+void qedn_clr_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit);
+int qedn_initialize_endpoint(struct qedn_endpoint *ep, u8 *local_mac_addr,
+			     struct nvme_tcp_ofld_ctrl *ctrl);
+int qedn_wait_for_conn_est(struct qedn_conn_ctx *conn_ctx);
+int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
+		       enum qedn_conn_state new_state);
+void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx);
+void qedn_cleanp_fw(struct qedn_conn_ctx *conn_ctx);
+
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
new file mode 100644
index 000000000000..04c45a6fa14b
--- /dev/null
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -0,0 +1,564 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+ /* Kernel includes */
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <net/tcp.h>
+
+/* Driver includes */
+#include "qedn.h"
+
+extern const struct qed_nvmetcp_ops *qed_ops;
+
+static const char * const qedn_conn_state_str[] = {
+	"CONN_IDLE",
+	"CREATE_CONNECTION",
+	"WAIT_FOR_CONNECT_DONE",
+	"OFFLOAD_COMPLETE",
+	"WAIT_FOR_UPDATE_EQE",
+	"WAIT_FOR_IC_COMP",
+	"NVMETCP_CONN_ESTABLISHED",
+	"DESTROY_CONNECTION",
+	"WAIT_FOR_DESTROY_DONE",
+	"DESTROY_COMPLETE",
+	NULL
+};
+
+int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
+		       enum qedn_conn_state new_state)
+{
+	spin_lock_bh(&conn_ctx->conn_state_lock);
+	conn_ctx->state = new_state;
+	spin_unlock_bh(&conn_ctx->conn_state_lock);
+
+	return 0;
+}
+
+static void qedn_return_tcp_port(struct qedn_conn_ctx *conn_ctx)
+{
+	if (conn_ctx->sock && conn_ctx->sock->sk) {
+		qed_return_tcp_port(conn_ctx->sock);
+		conn_ctx->sock = NULL;
+	}
+
+	conn_ctx->ep.src_port = 0;
+}
+
+int qedn_wait_for_conn_est(struct qedn_conn_ctx *conn_ctx)
+{
+	u32 est_timeout = msecs_to_jiffies(QEDN_WAIT_CON_ESTABLSH_TMO);
+	atomic_t *conn_est_ind = &conn_ctx->est_conn_indicator;
+	int wrc, rc;
+
+	wrc = wait_event_interruptible_timeout(conn_ctx->conn_waitq,
+					       atomic_read(conn_est_ind) > 0,
+					       est_timeout);
+	atomic_set(conn_est_ind, 0);
+	if (!wrc ||
+	    conn_ctx->state != CONN_STATE_NVMETCP_CONN_ESTABLISHED) {
+		rc = -ETIMEDOUT;
+
+		/* If error was prior or during offload, conn_ctx was released.
+		 * If the error was after offload sync has completed, we need to
+		 * terminate the connection ourselves.
+		 */
+		if (conn_ctx &&
+		    conn_ctx->state >= CONN_STATE_WAIT_FOR_CONNECT_DONE &&
+		    conn_ctx->state <= CONN_STATE_NVMETCP_CONN_ESTABLISHED)
+			qedn_terminate_connection(conn_ctx);
+	} else {
+		rc = 0;
+	}
+
+	return rc;
+}
+
+int qedn_fill_ep_addr4(struct qedn_endpoint *ep,
+		       struct nvme_tcp_ofld_ctrl_con_params *conn_params)
+{
+	struct sockaddr_in *raddr, *laddr;
+
+	raddr = (struct sockaddr_in *)&conn_params->remote_ip_addr;
+	laddr = (struct sockaddr_in *)&conn_params->local_ip_addr;
+
+	ep->ip_type = TCP_IPV4;
+	ep->src_port = laddr->sin_port;
+	ep->dst_port = ntohs(raddr->sin_port);
+
+	ep->src_addr[0] = laddr->sin_addr.s_addr;
+	ep->dst_addr[0] = raddr->sin_addr.s_addr;
+
+	return 0;
+}
+
+int qedn_fill_ep_addr6(struct qedn_endpoint *ep,
+		       struct nvme_tcp_ofld_ctrl_con_params *conn_params)
+{
+	struct sockaddr_in6 *raddr6, *laddr6;
+	int i;
+
+	raddr6 = (struct sockaddr_in6 *)&conn_params->remote_ip_addr;
+	laddr6 = (struct sockaddr_in6 *)&conn_params->local_ip_addr;
+
+	ep->ip_type = TCP_IPV6;
+	ep->src_port = laddr6->sin6_port;
+	ep->dst_port = ntohs(raddr6->sin6_port);
+
+	for (i = 0; i < 4; i++) {
+		ep->src_addr[i] = laddr6->sin6_addr.in6_u.u6_addr32[i];
+		ep->dst_addr[i] = raddr6->sin6_addr.in6_u.u6_addr32[i];
+	}
+
+	return 0;
+}
+
+int qedn_initialize_endpoint(struct qedn_endpoint *ep, u8 *local_mac_addr,
+			     struct nvme_tcp_ofld_ctrl *ctrl)
+{
+	struct nvme_tcp_ofld_ctrl_con_params *conn_params = &ctrl->conn_params;
+	struct qedn_ctrl *qctrl;
+
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return -ENODEV;
+
+	ether_addr_copy(ep->dst_mac, qctrl->remote_mac_addr.sa_data);
+	ether_addr_copy(ep->src_mac, local_mac_addr);
+	ep->vlan_id = qctrl->vlan_id;
+	if (conn_params->remote_ip_addr.ss_family == AF_INET)
+		qedn_fill_ep_addr4(ep, conn_params);
+	else
+		qedn_fill_ep_addr6(ep, conn_params);
+
+	return -1;
+}
+
+static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int rc = 0;
+
+	if (test_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state)) {
+		qed_ops->common->chain_free(qedn->cdev,
+					    &conn_ctx->ep.fw_sq_chain);
+		clear_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state);
+	}
+
+	if (test_bit(QEDN_CONN_RESRC_DB_ADD, &conn_ctx->resrc_state)) {
+		rc = qed_ops->common->db_recovery_del(qedn->cdev,
+						      conn_ctx->ep.p_doorbell,
+						      &conn_ctx->ep.db_data);
+		if (rc)
+			pr_warn("Doorbell recovery del returned error %u\n",
+				rc);
+
+		clear_bit(QEDN_CONN_RESRC_DB_ADD, &conn_ctx->resrc_state);
+	}
+
+	if (test_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state)) {
+		hash_del(&conn_ctx->hash_node);
+		rc = qed_ops->release_conn(qedn->cdev, conn_ctx->conn_handle);
+		if (rc)
+			pr_warn("Release_conn returned with an error %u\n",
+				rc);
+
+		clear_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state);
+	}
+
+	if (test_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state)) {
+		dma_free_coherent(&qedn->pdev->dev,
+				  conn_ctx->sq_depth *
+				  sizeof(struct nvmetcp_host_cccid_itid_entry),
+				  conn_ctx->host_cccid_itid,
+				  conn_ctx->host_cccid_itid_phy_addr);
+		clear_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP,
+			  &conn_ctx->resrc_state);
+	}
+
+	if (test_bit(QEDN_CONN_RESRC_TCP_PORT, &conn_ctx->resrc_state)) {
+		qedn_return_tcp_port(conn_ctx);
+		clear_bit(QEDN_CONN_RESRC_TCP_PORT,
+			  &conn_ctx->resrc_state);
+	}
+
+	if (conn_ctx->resrc_state)
+		pr_err("Conn resources state isn't 0 as expected 0x%lx\n",
+		       conn_ctx->resrc_state);
+
+	atomic_inc(&conn_ctx->destroy_conn_indicator);
+	qedn_set_con_state(conn_ctx, CONN_STATE_DESTROY_COMPLETE);
+	wake_up_interruptible(&conn_ctx->conn_waitq);
+}
+
+static int qedn_alloc_fw_sq(struct qedn_ctx *qedn,
+			    struct qedn_endpoint *ep)
+{
+	struct qed_chain_init_params params = {
+		.mode           = QED_CHAIN_MODE_PBL,
+		.intended_use   = QED_CHAIN_USE_TO_PRODUCE,
+		.cnt_type       = QED_CHAIN_CNT_TYPE_U16,
+		.num_elems      = QEDN_SQ_SIZE,
+		.elem_size      = sizeof(struct nvmetcp_wqe),
+	};
+	int rc;
+
+	rc = qed_ops->common->chain_alloc(qedn->cdev,
+					   &ep->fw_sq_chain,
+					   &params);
+	if (rc) {
+		pr_err("Failed to allocate SQ chain\n");
+
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
+static int qedn_nvmetcp_offload_conn(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qed_nvmetcp_params_offload offld_prms = { 0 };
+	struct qedn_endpoint *qedn_ep = &conn_ctx->ep;
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	u8 ts_hdr_size = 0;
+	u32 hdr_size;
+	int rc, i;
+
+	ether_addr_copy(offld_prms.src.mac, qedn_ep->src_mac);
+	ether_addr_copy(offld_prms.dst.mac, qedn_ep->dst_mac);
+	offld_prms.vlan_id = qedn_ep->vlan_id;
+	offld_prms.ecn_en = QEDN_TCP_ECN_EN;
+	offld_prms.timestamp_en =  QEDN_TCP_TS_EN;
+	offld_prms.delayed_ack_en = QEDN_TCP_DA_EN;
+	offld_prms.tcp_keep_alive_en = QEDN_TCP_KA_EN;
+	offld_prms.ip_version = qedn_ep->ip_type;
+
+	offld_prms.src.ip[0] = ntohl(qedn_ep->src_addr[0]);
+	offld_prms.dst.ip[0] = ntohl(qedn_ep->dst_addr[0]);
+	if (qedn_ep->ip_type == TCP_IPV6) {
+		for (i = 1; i < 4; i++) {
+			offld_prms.src.ip[i] = ntohl(qedn_ep->src_addr[i]);
+			offld_prms.dst.ip[i] = ntohl(qedn_ep->dst_addr[i]);
+		}
+	}
+
+	offld_prms.ttl = QEDN_TCP_TTL;
+	offld_prms.tos_or_tc = QEDN_TCP_TOS;
+	offld_prms.dst.port = qedn_ep->dst_port;
+	offld_prms.src.port = qedn_ep->src_port;
+	offld_prms.nvmetcp_cccid_itid_table_addr =
+		conn_ctx->host_cccid_itid_phy_addr;
+	offld_prms.nvmetcp_cccid_max_range = conn_ctx->sq_depth;
+
+	/* Calculate MSS */
+	if (offld_prms.timestamp_en)
+		ts_hdr_size = QEDN_TCP_TS_OPTION_LEN;
+
+	hdr_size = qedn_ep->ip_type == TCP_IPV4 ?
+		   sizeof(struct iphdr) : sizeof(struct ipv6hdr);
+	hdr_size += sizeof(struct tcphdr) + ts_hdr_size;
+
+	offld_prms.mss = qedn->mtu - hdr_size;
+	offld_prms.rcv_wnd_scale = QEDN_TCP_RCV_WND_SCALE;
+	offld_prms.cwnd = QEDN_TCP_MAX_CWND * offld_prms.mss;
+	offld_prms.ka_max_probe_cnt = QEDN_TCP_KA_MAX_PROBE_COUNT;
+	offld_prms.ka_timeout = QEDN_TCP_KA_TIMEOUT;
+	offld_prms.ka_interval = QEDN_TCP_KA_INTERVAL;
+	offld_prms.max_rt_time = QEDN_TCP_MAX_RT_TIME;
+	offld_prms.sq_pbl_addr =
+		(u64)qed_chain_get_pbl_phys(&qedn_ep->fw_sq_chain);
+
+	rc = qed_ops->offload_conn(qedn->cdev,
+				   conn_ctx->conn_handle,
+				   &offld_prms);
+	if (rc)
+		pr_err("offload_conn returned with an error\n");
+
+	return rc;
+}
+
+static int qedn_fetch_tcp_port(struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	struct qedn_ctrl *qctrl;
+	int rc = 0;
+
+	ctrl = conn_ctx->ctrl;
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return -ENODEV;
+
+	rc = qed_fetch_tcp_port(ctrl->conn_params.local_ip_addr,
+				&conn_ctx->sock, &conn_ctx->ep.src_port);
+
+	return rc;
+}
+
+static void qedn_decouple_conn(struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_tcp_ofld_queue *queue;
+
+	queue = conn_ctx->queue;
+	queue->private_data = NULL;
+}
+
+void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctrl *qctrl;
+
+	if (!conn_ctx)
+		return;
+
+	qctrl = (struct qedn_ctrl *)conn_ctx->ctrl->private_data;
+	if (!qctrl)
+		return;
+
+	if (test_and_set_bit(DESTROY_CONNECTION, &conn_ctx->agg_work_action))
+		return;
+
+	qedn_set_con_state(conn_ctx, CONN_STATE_DESTROY_CONNECTION);
+	queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
+}
+
+/* Slowpath EQ Callback */
+int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
+{
+	struct nvmetcp_connect_done_results *eqe_connect_done;
+	struct nvmetcp_eqe_data *eqe_data;
+	struct nvme_tcp_ofld_ctrl *ctrl;
+	struct qedn_conn_ctx *conn_ctx;
+	struct qedn_ctrl *qctrl;
+	struct qedn_ctx *qedn;
+	u16 icid;
+	int rc;
+
+	if (!context || !event_ring_data) {
+		pr_err("Recv event with ctx NULL\n");
+
+		return -EINVAL;
+	}
+
+	qedn = (struct qedn_ctx *)context;
+
+	if (fw_event_code != NVMETCP_EVENT_TYPE_ASYN_CONNECT_COMPLETE) {
+		eqe_data = (struct nvmetcp_eqe_data *)event_ring_data;
+		icid = le16_to_cpu(eqe_data->icid);
+		pr_err("EQE Type=0x%x icid=0x%x, conn_id=0x%x err-code=0x%x\n",
+		       fw_event_code, eqe_data->icid, eqe_data->conn_id,
+		       eqe_data->error_code);
+	} else {
+		eqe_connect_done =
+			(struct nvmetcp_connect_done_results *)event_ring_data;
+		icid = le16_to_cpu(eqe_connect_done->icid);
+	}
+
+	conn_ctx = qedn_get_conn_hash(qedn, icid);
+	if (!conn_ctx) {
+		pr_err("Connection with icid=0x%x doesn't exist in conn list\n",
+		       icid);
+
+		return -EINVAL;
+	}
+
+	ctrl = conn_ctx->ctrl;
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return -ENODEV;
+
+	switch (fw_event_code) {
+	case NVMETCP_EVENT_TYPE_ASYN_CONNECT_COMPLETE:
+		if (conn_ctx->state != CONN_STATE_WAIT_FOR_CONNECT_DONE) {
+			pr_err("CID=0x%x:ASYN_CONNECT_COMPL:Wrong state %u\n",
+			       conn_ctx->fw_cid, conn_ctx->state);
+		} else {
+			rc = qedn_set_con_state(conn_ctx,
+						CONN_STATE_OFFLOAD_COMPLETE);
+
+			if (rc)
+				return rc;
+
+			/* Placeholder - for ICReq flow */
+		}
+
+		break;
+	case NVMETCP_EVENT_TYPE_ASYN_TERMINATE_DONE:
+		if (conn_ctx->state != CONN_STATE_WAIT_FOR_DESTROY_DONE)
+			pr_err("CID=0x%x:ASYN_TERMINATE_DONE:Wrong state %u\n",
+			       conn_ctx->fw_cid, conn_ctx->state);
+		else
+			queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
+
+		break;
+	default:
+		pr_err("CID=0x%x - Recv Unknown Event %u\n",
+		       conn_ctx->fw_cid, fw_event_code);
+		break;
+	}
+
+	return 0;
+}
+
+void qedn_prep_db_data(struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvmetcp_db_data *db_data = &conn_ctx->ep.db_data;
+	u8 params = 0;
+
+	params |= DB_DEST_XCM << NVMETCP_DB_DATA_DEST_SHIFT;
+	params |= DB_AGG_CMD_SET << NVMETCP_DB_DATA_AGG_CMD_SHIFT;
+	params |= DQ_XCM_ISCSI_SQ_PROD_CMD << NVMETCP_DB_DATA_AGG_VAL_SEL_SHIFT;
+	params |= 1 << NVMETCP_DB_DATA_BYPASS_EN_SHIFT;
+
+	db_data->params = params;
+	db_data->agg_flags = 0;
+}
+
+static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	size_t dma_size;
+	int rc;
+
+	rc = qedn_alloc_fw_sq(qedn, &conn_ctx->ep);
+	if (rc) {
+		pr_err("Failed to allocate FW SQ\n");
+		goto rel_conn;
+	}
+
+	set_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state);
+	rc = qed_ops->acquire_conn(qedn->cdev,
+				   &conn_ctx->conn_handle,
+				   &conn_ctx->fw_cid,
+				   &conn_ctx->ep.p_doorbell);
+	if (rc) {
+		pr_err("Couldn't acquire connection\n");
+		goto rel_conn;
+	}
+
+	hash_add(qedn->conn_ctx_hash, &conn_ctx->hash_node,
+		 conn_ctx->conn_handle);
+	set_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state);
+
+	/* Placeholder - Allocate task resources and initialize fields */
+
+	rc = qedn_fetch_tcp_port(conn_ctx);
+	if (rc)
+		goto rel_conn;
+
+	set_bit(QEDN_CONN_RESRC_TCP_PORT, &conn_ctx->resrc_state);
+	dma_size = conn_ctx->sq_depth *
+			   sizeof(struct nvmetcp_host_cccid_itid_entry);
+	conn_ctx->host_cccid_itid =
+			dma_alloc_coherent(&qedn->pdev->dev,
+					   dma_size,
+					   &conn_ctx->host_cccid_itid_phy_addr,
+					   GFP_ATOMIC);
+	if (!conn_ctx->host_cccid_itid) {
+		pr_err("CCCID-iTID Map allocation failed\n");
+		goto rel_conn;
+	}
+
+	memset(conn_ctx->host_cccid_itid, 0xFF, dma_size);
+	set_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state);
+	rc = qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_CONNECT_DONE);
+	if (rc)
+		goto rel_conn;
+
+	qedn_prep_db_data(conn_ctx);
+	rc = qed_ops->common->db_recovery_add(qedn->cdev,
+					      conn_ctx->ep.p_doorbell,
+					      &conn_ctx->ep.db_data,
+					      DB_REC_WIDTH_32B, DB_REC_KERNEL);
+	if (rc)
+		goto rel_conn;
+	set_bit(QEDN_CONN_RESRC_DB_ADD, &conn_ctx->resrc_state);
+
+	rc = qedn_nvmetcp_offload_conn(conn_ctx);
+	if (rc) {
+		pr_err("Offload error: CID=0x%x\n", conn_ctx->fw_cid);
+		goto rel_conn;
+	}
+
+	return 0;
+
+rel_conn:
+	pr_err("qedn create queue ended with ERROR\n");
+	qedn_release_conn_ctx(conn_ctx);
+
+	return -EINVAL;
+}
+
+void qedn_cleanp_fw(struct qedn_conn_ctx *conn_ctx)
+{
+	/* Placeholder - task cleanup */
+}
+
+void qedn_destroy_connection(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int rc;
+
+	qedn_decouple_conn(conn_ctx);
+
+	if (qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_DESTROY_DONE))
+		return;
+
+	rc = qed_ops->destroy_conn(qedn->cdev, conn_ctx->conn_handle,
+				   conn_ctx->abrt_flag);
+	if (rc)
+		pr_warn("destroy_conn failed - rc %u\n", rc);
+}
+
+void qedn_sp_wq_handler(struct work_struct *work)
+{
+	struct qedn_conn_ctx *conn_ctx;
+	struct qedn_ctx *qedn;
+	int rc;
+
+	conn_ctx = container_of(work, struct qedn_conn_ctx, sp_wq_entry);
+	qedn = conn_ctx->qedn;
+
+	if (conn_ctx->state == CONN_STATE_DESTROY_COMPLETE) {
+		pr_err("Connection already released!\n");
+
+		return;
+	}
+
+	if (conn_ctx->state == CONN_STATE_WAIT_FOR_DESTROY_DONE) {
+		qedn_release_conn_ctx(conn_ctx);
+
+		return;
+	}
+
+	qedn = conn_ctx->qedn;
+	if (test_bit(DESTROY_CONNECTION, &conn_ctx->agg_work_action)) {
+		qedn_destroy_connection(conn_ctx);
+
+		return;
+	}
+
+	if (test_bit(CREATE_CONNECTION, &conn_ctx->agg_work_action)) {
+		qedn_clr_sp_wa(conn_ctx, CREATE_CONNECTION);
+		rc = qedn_prep_and_offload_queue(conn_ctx);
+		if (rc) {
+			pr_err("Error in queue prepare & firmware offload\n");
+
+			return;
+		}
+	}
+}
+
+/* Clear connection aggregative slowpath work action */
+void qedn_clr_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit)
+{
+	clear_bit(bit, &conn_ctx->agg_work_action);
+}
+
+/* Set connection aggregative slowpath work action */
+void qedn_set_sp_wa(struct qedn_conn_ctx *conn_ctx, u32 bit)
+{
+	set_bit(bit, &conn_ctx->agg_work_action);
+}
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 6c0f36f7d9d1..bd5618f65c70 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -22,6 +22,15 @@ static struct pci_device_id qedn_pci_tbl[] = {
 	{0, 0},
 };
 
+static bool qedn_matches_qede(struct qedn_ctx *qedn, struct pci_dev *qede_pdev)
+{
+	struct pci_dev *qedn_pdev = qedn->pdev;
+
+	return (qede_pdev->bus->number == qedn_pdev->bus->number &&
+		PCI_SLOT(qede_pdev->devfn) == PCI_SLOT(qedn_pdev->devfn) &&
+		PCI_FUNC(qede_pdev->devfn) == qedn->dev_info.port_id);
+}
+
 static int
 qedn_find_dev(struct nvme_tcp_ofld_dev *dev,
 	      struct nvme_tcp_ofld_ctrl *ctrl)
@@ -29,7 +38,9 @@ qedn_find_dev(struct nvme_tcp_ofld_dev *dev,
 	struct nvme_tcp_ofld_ctrl_con_params *conn_params;
 	struct pci_dev *qede_pdev = NULL;
 	struct sockaddr remote_mac_addr;
+	struct qedn_ctrl *qctrl = NULL;
 	struct net_device *ndev = NULL;
+	struct qedn_ctx *qedn = NULL;
 	u16 vlan_id = 0;
 	int rc = 0;
 
@@ -65,11 +76,24 @@ qedn_find_dev(struct nvme_tcp_ofld_dev *dev,
 
 	qed_vlan_get_ndev(&ctrl->ndev, &vlan_id);
 
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (qctrl) {
+		qctrl->remote_mac_addr = remote_mac_addr;
+		qctrl->vlan_id = vlan_id;
+	}
+
 	/* route found through ndev - validate this is qede*/
 	qede_pdev = qed_validate_ndev(ctrl->ndev);
 	if (!qede_pdev)
 		return false;
 
+	qedn = container_of(dev, struct qedn_ctx, qedn_ofld_dev);
+	if (!qedn)
+		return false;
+
+	if (!qedn_matches_qede(qedn, qede_pdev))
+		return false;
+
 	return true;
 }
 
@@ -82,14 +106,73 @@ qedn_claim_dev(struct nvme_tcp_ofld_dev *dev,
 
 static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 {
-	/* Placeholder - qedn_setup_ctrl */
+	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+	struct qedn_ctrl *qctrl = NULL;
+	struct qedn_ctx *qedn = NULL;
+	bool new = true;
+	int rc = 0;
+
+	if (ctrl->private_data) {
+		qctrl = (struct qedn_ctrl *)ctrl->private_data;
+		new = false;
+	}
+
+	if (new) {
+		qctrl = kzalloc(sizeof(*qctrl), GFP_KERNEL);
+		if (!qctrl)
+			return -ENOMEM;
+
+		ctrl->private_data = (void *)qctrl;
+		set_bit(QEDN_CTRL_SET_TO_OFLD_CTRL, &qctrl->agg_state);
+
+		qctrl->sp_wq = alloc_workqueue(QEDN_SP_WORKQUEUE,
+					       WQ_MEM_RECLAIM,
+					       QEDN_SP_WORKQUEUE_MAX_ACTIVE);
+		if (!qctrl->sp_wq) {
+			rc = -ENODEV;
+			pr_err("Unable to create slowpath work queue!\n");
+			kfree(qctrl);
+
+			return rc;
+		}
+
+		set_bit(QEDN_STATE_SP_WORK_THREAD_SET, &qctrl->agg_state);
+	}
+
+	if (!qedn_find_dev(dev, ctrl)) {
+		rc = -ENODEV;
+		goto err_out;
+	}
+
+	qedn = container_of(dev, struct qedn_ctx, qedn_ofld_dev);
+	qctrl->qedn = qedn;
+
+	/* Placeholder - setup LLH filter */
 
 	return 0;
+err_out:
+	flush_workqueue(qctrl->sp_wq);
+	kfree(qctrl);
+
+	return rc;
 }
 
 static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 {
-	/* Placeholder - qedn_release_ctrl */
+	struct qedn_ctrl *qctrl;
+
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return -ENODEV;
+
+	if (test_and_clear_bit(QEDN_STATE_SP_WORK_THREAD_SET,
+			       &qctrl->agg_state))
+		flush_workqueue(qctrl->sp_wq);
+
+	if (test_and_clear_bit(QEDN_CTRL_SET_TO_OFLD_CTRL, &qctrl->agg_state)) {
+		kfree(qctrl);
+		ctrl->private_data = NULL;
+	}
 
 	return 0;
 }
@@ -97,19 +180,117 @@ static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 			     size_t queue_size)
 {
-	/* Placeholder - qedn_create_queue */
+	struct nvme_tcp_ofld_ctrl *ctrl = queue->ctrl;
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
+	struct qedn_conn_ctx *conn_ctx;
+	struct qedn_ctrl *qctrl;
+	struct qedn_ctx *qedn;
+	int rc;
+
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return -ENODEV;
+
+	qedn = qctrl->qedn;
+
+	/* Allocate qedn connection context */
+	conn_ctx = kzalloc(sizeof(*conn_ctx), GFP_KERNEL);
+	if (!conn_ctx)
+		return -ENOMEM;
+
+	queue->private_data = conn_ctx;
+	queue->hdr_digest = nctrl->opts->hdr_digest;
+	queue->data_digest = nctrl->opts->data_digest;
+	queue->tos = nctrl->opts->tos;
+
+	conn_ctx->qedn = qedn;
+	conn_ctx->queue = queue;
+	conn_ctx->ctrl = ctrl;
+	conn_ctx->sq_depth = queue_size;
+
+	init_waitqueue_head(&conn_ctx->conn_waitq);
+	atomic_set(&conn_ctx->est_conn_indicator, 0);
+	atomic_set(&conn_ctx->destroy_conn_indicator, 0);
+
+	spin_lock_init(&conn_ctx->conn_state_lock);
+
+	qedn_initialize_endpoint(&conn_ctx->ep, qedn->local_mac_addr, ctrl);
+
+	atomic_inc(&qctrl->host_num_active_conns);
+
+	qedn_set_sp_wa(conn_ctx, CREATE_CONNECTION);
+	qedn_set_con_state(conn_ctx, CONN_STATE_CREATE_CONNECTION);
+	INIT_WORK(&conn_ctx->sp_wq_entry, qedn_sp_wq_handler);
+	queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
+
+	/* Wait for the connection establishment to complete - this includes the
+	 * FW TCP connection establishment and the NVMeTCP ICReq & ICResp
+	 */
+	rc = qedn_wait_for_conn_est(conn_ctx);
+	if (rc)
+		return -ENXIO;
 
 	return 0;
 }
 
 static void qedn_drain_queue(struct nvme_tcp_ofld_queue *queue)
 {
-	/* Placeholder - qedn_drain_queue */
+	struct qedn_conn_ctx *conn_ctx;
+
+	if (!queue) {
+		pr_err("ctrl has no queues\n");
+
+		return;
+	}
+
+	conn_ctx = (struct qedn_conn_ctx *)queue->private_data;
+	if (!conn_ctx)
+		return;
+
+	qedn_cleanp_fw(conn_ctx);
+}
+
+static inline void
+qedn_queue_wait_for_terminate_complete(struct qedn_conn_ctx *conn_ctx)
+{
+	/* Returns valid non-0 */
+	atomic_t *conn_dest_ind = &conn_ctx->destroy_conn_indicator;
+	u32 dest_timeout = msecs_to_jiffies(QEDN_RLS_CONS_TMO);
+	int wrc, state;
+
+	wrc = wait_event_interruptible_timeout(conn_ctx->conn_waitq,
+					       atomic_read(conn_dest_ind) > 0,
+					       dest_timeout);
+
+	atomic_set(conn_dest_ind, 0);
+
+	spin_lock_bh(&conn_ctx->conn_state_lock);
+	state = conn_ctx->state;
+	spin_unlock_bh(&conn_ctx->conn_state_lock);
+
+	if (!wrc  || state != CONN_STATE_DESTROY_COMPLETE)
+		pr_warn("Timed out waiting for clear-SQ on FW conns");
 }
 
 static void qedn_destroy_queue(struct nvme_tcp_ofld_queue *queue)
 {
-	/* Placeholder - qedn_destroy_queue */
+	struct qedn_conn_ctx *conn_ctx;
+
+	if (!queue) {
+		pr_err("ctrl has no queues\n");
+
+		return;
+	}
+
+	conn_ctx = (struct qedn_conn_ctx *)queue->private_data;
+	if (!conn_ctx)
+		return;
+
+	qedn_terminate_connection(conn_ctx);
+
+	qedn_queue_wait_for_terminate_complete(conn_ctx);
+
+	kfree(conn_ctx);
 }
 
 static int qedn_poll_queue(struct nvme_tcp_ofld_queue *queue)
@@ -150,6 +331,21 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
 	.send_req = qedn_send_req,
 };
 
+struct qedn_conn_ctx *qedn_get_conn_hash(struct qedn_ctx *qedn, u16 icid)
+{
+	struct qedn_conn_ctx *conn = NULL;
+
+	hash_for_each_possible(qedn->conn_ctx_hash, conn, hash_node, icid) {
+		if (conn->conn_handle == icid)
+			break;
+	}
+
+	if (!conn || conn->conn_handle != icid)
+		return NULL;
+
+	return conn;
+}
+
 /* Fastpath IRQ handler */
 static irqreturn_t qedn_irq_handler(int irq, void *dev_id)
 {
@@ -250,7 +446,7 @@ static int qedn_setup_irq(struct qedn_ctx *qedn)
 
 static inline void qedn_init_pf_struct(struct qedn_ctx *qedn)
 {
-	/* Placeholder - Initialize qedn fields */
+	hash_init(qedn->conn_ctx_hash);
 }
 
 static inline void
@@ -591,7 +787,7 @@ static int __qedn_probe(struct pci_dev *pdev)
 	rc = qed_ops->start(qedn->cdev,
 			    NULL /* Placeholder for FW IO-path resources */,
 			    qedn,
-			    NULL /* Placeholder for FW Event callback */);
+			    qedn_event_cb);
 	if (rc) {
 		rc = -ENODEV;
 		pr_err("Cannot start NVMeTCP Function\n");
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:37 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:37 +0300
Subject: [PATCH v4 14/20] qedn: Add support of configuring HW filter block
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-15-pkushwaha@marvell.com>

HW filter can be configured to filter TCP packets based on either
source or target TCP port. QEDN leverage this feature to route
NVMeTCP traffic.

This patch configures HW filter block based on source port for all
receiving packets to deliver correct QEDN PF.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |  15 ++++
 drivers/nvme/hw/qedn/qedn_main.c | 113 ++++++++++++++++++++++++++++++-
 2 files changed, 127 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 38ab4ff88999..9e16297fa323 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -29,6 +29,11 @@
 #define QEDN_IRQ_NAME_LEN 24
 #define QEDN_IRQ_NO_FLAGS 0
 
+/* HW defines */
+
+/* QEDN_MAX_LLH_PORTS will be extended in future */
+#define QEDN_MAX_LLH_PORTS 16
+
 /* Destroy connection defines */
 #define QEDN_NON_ABORTIVE_TERMINATION 0
 #define QEDN_ABORTIVE_TERMINATION 1
@@ -68,6 +73,7 @@
 enum qedn_state {
 	QEDN_STATE_CORE_PROBED = 0,
 	QEDN_STATE_CORE_OPEN,
+	QEDN_STATE_LLH_PORT_FILTER_SET,
 	QEDN_STATE_MFW_STATE,
 	QEDN_STATE_NVMETCP_OPEN,
 	QEDN_STATE_IRQ_SET,
@@ -99,6 +105,8 @@ struct qedn_ctx {
 	/* Accessed with atomic bit ops, used with enum qedn_state */
 	unsigned long state;
 
+	u8 num_llh_filters;
+	struct list_head llh_filter_list;
 	u8 local_mac_addr[ETH_ALEN];
 	u16 mtu;
 
@@ -165,6 +173,12 @@ enum qedn_conn_state {
 	CONN_STATE_DESTROY_COMPLETE
 };
 
+struct qedn_llh_filter {
+	struct list_head entry;
+	u16 port;
+	u16 ref_cnt;
+};
+
 struct qedn_ctrl {
 	struct list_head glb_entry;
 	struct list_head pf_entry;
@@ -249,5 +263,6 @@ int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
 		       enum qedn_conn_state new_state);
 void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx);
 void qedn_cleanp_fw(struct qedn_conn_ctx *conn_ctx);
+__be16 qedn_get_in_port(struct sockaddr_storage *sa);
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index bd5618f65c70..3cadec6d6f5d 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -22,6 +22,86 @@ static struct pci_device_id qedn_pci_tbl[] = {
 	{0, 0},
 };
 
+__be16 qedn_get_in_port(struct sockaddr_storage *sa)
+{
+	return sa->ss_family == AF_INET
+		? ((struct sockaddr_in *)sa)->sin_port
+		: ((struct sockaddr_in6 *)sa)->sin6_port;
+}
+
+struct qedn_llh_filter *qedn_add_llh_filter(struct qedn_ctx *qedn, u16 tcp_port)
+{
+	struct qedn_llh_filter *llh_filter = NULL;
+	struct qedn_llh_filter *llh_tmp = NULL;
+	bool new_filter = 1;
+	int rc = 0;
+
+	/* Check if LLH filter already defined */
+	list_for_each_entry_safe(llh_filter, llh_tmp, &qedn->llh_filter_list,
+				 entry) {
+		if (llh_filter->port == tcp_port) {
+			new_filter = 0;
+			llh_filter->ref_cnt++;
+			break;
+		}
+	}
+
+	if (new_filter) {
+		if (qedn->num_llh_filters >= QEDN_MAX_LLH_PORTS) {
+			pr_err("PF reached the max target ports limit %u. %u\n",
+			       qedn->dev_info.common.abs_pf_id,
+			       qedn->num_llh_filters);
+
+			return NULL;
+		}
+
+		rc = qed_ops->add_src_tcp_port_filter(qedn->cdev, tcp_port);
+		if (rc) {
+			pr_err("LLH port config failed. port:%u; rc:%u\n",
+			       tcp_port, rc);
+
+			return NULL;
+		}
+
+		llh_filter = kzalloc(sizeof(*llh_filter), GFP_KERNEL);
+		if (!llh_filter) {
+			qed_ops->remove_src_tcp_port_filter(qedn->cdev,
+							    tcp_port);
+
+			return NULL;
+		}
+
+		llh_filter->port = tcp_port;
+		llh_filter->ref_cnt = 1;
+		++qedn->num_llh_filters;
+		list_add_tail(&llh_filter->entry, &qedn->llh_filter_list);
+		set_bit(QEDN_STATE_LLH_PORT_FILTER_SET, &qedn->state);
+	}
+
+	return llh_filter;
+}
+
+void qedn_dec_llh_filter(struct qedn_ctx *qedn,
+			 struct qedn_llh_filter *llh_filter)
+{
+	if (!llh_filter)
+		return;
+
+	llh_filter->ref_cnt--;
+	if (!llh_filter->ref_cnt) {
+		list_del(&llh_filter->entry);
+
+		/* Remove LLH protocol port filter */
+		qed_ops->remove_src_tcp_port_filter(qedn->cdev,
+						    llh_filter->port);
+
+		--qedn->num_llh_filters;
+		kfree(llh_filter);
+		if (!qedn->num_llh_filters)
+			clear_bit(QEDN_STATE_LLH_PORT_FILTER_SET, &qedn->state);
+	}
+}
+
 static bool qedn_matches_qede(struct qedn_ctx *qedn, struct pci_dev *qede_pdev)
 {
 	struct pci_dev *qedn_pdev = qedn->pdev;
@@ -107,8 +187,10 @@ qedn_claim_dev(struct nvme_tcp_ofld_dev *dev,
 static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 {
 	struct nvme_tcp_ofld_dev *dev = ctrl->dev;
+	struct qedn_llh_filter *llh_filter = NULL;
 	struct qedn_ctrl *qctrl = NULL;
 	struct qedn_ctx *qedn = NULL;
+	__be16 remote_port;
 	bool new = true;
 	int rc = 0;
 
@@ -147,7 +229,22 @@ static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 	qedn = container_of(dev, struct qedn_ctx, qedn_ofld_dev);
 	qctrl->qedn = qedn;
 
-	/* Placeholder - setup LLH filter */
+	if (qedn->num_llh_filters == 0) {
+		qedn->mtu = ctrl->ndev->mtu;
+		memcpy(qedn->local_mac_addr, ctrl->ndev->dev_addr, ETH_ALEN);
+	}
+
+	remote_port = qedn_get_in_port(&ctrl->conn_params.remote_ip_addr);
+	if (new) {
+		llh_filter = qedn_add_llh_filter(qedn, ntohs(remote_port));
+		if (!llh_filter) {
+			rc = -EFAULT;
+			goto err_out;
+		}
+
+		qctrl->llh_filter = llh_filter;
+		set_bit(LLH_FILTER, &qctrl->agg_state);
+	}
 
 	return 0;
 err_out:
@@ -165,6 +262,12 @@ static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 	if (!qctrl)
 		return -ENODEV;
 
+	if (test_and_clear_bit(LLH_FILTER, &qctrl->agg_state) &&
+	    qctrl->llh_filter) {
+		qedn_dec_llh_filter(qctrl->qedn, qctrl->llh_filter);
+		qctrl->llh_filter = NULL;
+	}
+
 	if (test_and_clear_bit(QEDN_STATE_SP_WORK_THREAD_SET,
 			       &qctrl->agg_state))
 		flush_workqueue(qctrl->sp_wq);
@@ -446,6 +549,8 @@ static int qedn_setup_irq(struct qedn_ctx *qedn)
 
 static inline void qedn_init_pf_struct(struct qedn_ctx *qedn)
 {
+	INIT_LIST_HEAD(&qedn->llh_filter_list);
+	qedn->num_llh_filters = 0;
 	hash_init(qedn->conn_ctx_hash);
 }
 
@@ -688,6 +793,12 @@ static void __qedn_remove(struct pci_dev *pdev)
 		return;
 	}
 
+	if (test_and_clear_bit(QEDN_STATE_LLH_PORT_FILTER_SET, &qedn->state)) {
+		pr_err("LLH port configuration removal. %d filters still set\n",
+		       qedn->num_llh_filters);
+		qed_ops->clear_all_filters(qedn->cdev);
+	}
+
 	if (test_and_clear_bit(QEDN_STATE_REGISTERED_OFFLOAD_DEV, &qedn->state))
 		nvme_tcp_ofld_unregister_dev(&qedn->qedn_ofld_dev);
 
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:38 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:38 +0300
Subject: [PATCH v4 15/20] qedn: Add IO level qedn_send_req and fw_cq workqueue
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-16-pkushwaha@marvell.com>

From: Shai Malin <smalin at marvell.com>

This patch will present the IO level skeleton flows:

- qedn_send_req(): process new requests, similar to nvme_tcp_queue_rq().

- qedn_fw_cq_fp_wq():   process new FW completions, the flow starts from
			the IRQ handler and for a single interrupt it will
			process all the pending NVMeoF Completions under
			polling mode.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
---
 drivers/nvme/hw/qedn/Makefile    |  2 +-
 drivers/nvme/hw/qedn/qedn.h      | 14 ++++++
 drivers/nvme/hw/qedn/qedn_conn.c |  2 +
 drivers/nvme/hw/qedn/qedn_main.c | 84 +++++++++++++++++++++++++++++---
 drivers/nvme/hw/qedn/qedn_task.c | 79 ++++++++++++++++++++++++++++++
 5 files changed, 172 insertions(+), 9 deletions(-)
 create mode 100644 drivers/nvme/hw/qedn/qedn_task.c

diff --git a/drivers/nvme/hw/qedn/Makefile b/drivers/nvme/hw/qedn/Makefile
index ece84772d317..888d466fa5ed 100644
--- a/drivers/nvme/hw/qedn/Makefile
+++ b/drivers/nvme/hw/qedn/Makefile
@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0
 
 obj-$(CONFIG_NVME_QEDN) += qedn.o
-qedn-y := qedn_main.o qedn_conn.o
+qedn-y := qedn_main.o qedn_conn.o qedn_task.o
\ No newline at end of file
diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 9e16297fa323..13d63f420a23 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -38,6 +38,8 @@
 #define QEDN_NON_ABORTIVE_TERMINATION 0
 #define QEDN_ABORTIVE_TERMINATION 1
 
+#define QEDN_FW_CQ_FP_WQ_WORKQUEUE "qedn_fw_cq_fp_wq"
+
 /*
  * TCP offload stack default configurations and defines.
  * Future enhancements will allow controlling the configurable
@@ -90,6 +92,7 @@ struct qedn_fp_queue {
 	struct qedn_ctx	*qedn;
 	struct qed_sb_info *sb_info;
 	unsigned int cpu;
+	struct work_struct fw_cq_fp_wq_entry;
 	u16 sb_id;
 	char irqname[QEDN_IRQ_NAME_LEN];
 };
@@ -118,6 +121,7 @@ struct qedn_ctx {
 	struct qedn_fp_queue *fp_q_arr;
 	struct nvmetcp_glbl_queue_entry *fw_cq_array_virt;
 	dma_addr_t fw_cq_array_phy; /* Physical address of fw_cq_array_virt */
+	struct workqueue_struct *fw_cq_fp_wq;
 };
 
 struct qedn_endpoint {
@@ -204,6 +208,12 @@ struct qedn_ctrl {
 
 /* Connection level struct */
 struct qedn_conn_ctx {
+	/* IO path */
+	struct qedn_fp_queue *fp_q;
+	/* mutex for queueing request */
+	struct mutex send_mutex;
+	int qid;
+
 	struct qedn_ctx *qedn;
 	struct nvme_tcp_ofld_queue *queue;
 	struct nvme_tcp_ofld_ctrl *ctrl;
@@ -264,5 +274,9 @@ int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
 void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx);
 void qedn_cleanp_fw(struct qedn_conn_ctx *conn_ctx);
 __be16 qedn_get_in_port(struct sockaddr_storage *sa);
+int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
+		       struct nvme_tcp_ofld_req *req);
+void qedn_nvme_req_fp_wq_handler(struct work_struct *work);
+void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe);
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
index 04c45a6fa14b..962c88a4f345 100644
--- a/drivers/nvme/hw/qedn/qedn_conn.c
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -190,6 +190,7 @@ static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx)
 		pr_err("Conn resources state isn't 0 as expected 0x%lx\n",
 		       conn_ctx->resrc_state);
 
+	mutex_destroy(&conn_ctx->send_mutex);
 	atomic_inc(&conn_ctx->destroy_conn_indicator);
 	qedn_set_con_state(conn_ctx, CONN_STATE_DESTROY_COMPLETE);
 	wake_up_interruptible(&conn_ctx->conn_waitq);
@@ -429,6 +430,7 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 	}
 
 	set_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state);
+
 	rc = qed_ops->acquire_conn(qedn->cdev,
 				   &conn_ctx->conn_handle,
 				   &conn_ctx->fw_cid,
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 3cadec6d6f5d..975949ce6fb0 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -310,6 +310,7 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 	conn_ctx->queue = queue;
 	conn_ctx->ctrl = ctrl;
 	conn_ctx->sq_depth = queue_size;
+	mutex_init(&conn_ctx->send_mutex);
 
 	init_waitqueue_head(&conn_ctx->conn_waitq);
 	atomic_set(&conn_ctx->est_conn_indicator, 0);
@@ -317,6 +318,8 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 
 	spin_lock_init(&conn_ctx->conn_state_lock);
 
+	conn_ctx->qid = qid;
+
 	qedn_initialize_endpoint(&conn_ctx->ep, qedn->local_mac_addr, ctrl);
 
 	atomic_inc(&qctrl->host_num_active_conns);
@@ -408,9 +411,18 @@ static int qedn_poll_queue(struct nvme_tcp_ofld_queue *queue)
 
 static int qedn_send_req(struct nvme_tcp_ofld_req *req)
 {
-	/* Placeholder - qedn_send_req */
+	struct qedn_conn_ctx *qedn_conn;
+	int rc = 0;
 
-	return 0;
+	qedn_conn = (struct qedn_conn_ctx *)req->queue->private_data;
+	if (unlikely(!qedn_conn))
+		return -ENXIO;
+
+	mutex_lock(&qedn_conn->send_mutex);
+	rc = qedn_queue_request(qedn_conn, req);
+	mutex_unlock(&qedn_conn->send_mutex);
+
+	return rc;
 }
 
 static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
@@ -450,9 +462,58 @@ struct qedn_conn_ctx *qedn_get_conn_hash(struct qedn_ctx *qedn, u16 icid)
 }
 
 /* Fastpath IRQ handler */
+void qedn_fw_cq_fp_handler(struct qedn_fp_queue *fp_q)
+{
+	u16 sb_id, cq_prod_idx, cq_cons_idx;
+	struct qedn_ctx *qedn = fp_q->qedn;
+	struct nvmetcp_fw_cqe *cqe = NULL;
+
+	sb_id = fp_q->sb_id;
+	qed_sb_update_sb_idx(fp_q->sb_info);
+
+	/* rmb - to prevent missing new cqes */
+	rmb();
+
+	/* Read the latest cq_prod from the SB */
+	cq_prod_idx = *fp_q->cq_prod;
+	cq_cons_idx = qed_chain_get_cons_idx(&fp_q->cq_chain);
+
+	while (cq_cons_idx != cq_prod_idx) {
+		cqe = qed_chain_consume(&fp_q->cq_chain);
+		if (likely(cqe))
+			qedn_io_work_cq(qedn, cqe);
+		else
+			pr_err("Failed consuming cqe\n");
+
+		cq_cons_idx = qed_chain_get_cons_idx(&fp_q->cq_chain);
+
+		/* Check if new completions were posted */
+		if (unlikely(cq_prod_idx == cq_cons_idx)) {
+			/* rmb - to prevent missing new cqes */
+			rmb();
+
+			/* Update the latest cq_prod from the SB */
+			cq_prod_idx = *fp_q->cq_prod;
+		}
+	}
+}
+
+static void qedn_fw_cq_fq_wq_handler(struct work_struct *work)
+{
+	struct qedn_fp_queue *fp_q = container_of(work, struct qedn_fp_queue,
+						  fw_cq_fp_wq_entry);
+
+	qedn_fw_cq_fp_handler(fp_q);
+	qed_sb_ack(fp_q->sb_info, IGU_INT_ENABLE, 1);
+}
+
 static irqreturn_t qedn_irq_handler(int irq, void *dev_id)
 {
-	/* Placeholder */
+	struct qedn_fp_queue *fp_q = dev_id;
+	struct qedn_ctx *qedn = fp_q->qedn;
+
+	qed_sb_ack(fp_q->sb_info, IGU_INT_DISABLE, 0);
+	queue_work_on(fp_q->cpu, qedn->fw_cq_fp_wq, &fp_q->fw_cq_fp_wq_entry);
 
 	return IRQ_HANDLED;
 }
@@ -586,6 +647,8 @@ static void qedn_free_function_queues(struct qedn_ctx *qedn)
 	int i;
 
 	/* Free workqueues */
+	destroy_workqueue(qedn->fw_cq_fp_wq);
+	qedn->fw_cq_fp_wq = NULL;
 
 	/* Free the fast path queues*/
 	for (i = 0; i < qedn->num_fw_cqs; i++) {
@@ -653,7 +716,14 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
 	u64 cq_phy_addr;
 	int i;
 
-	/* Place holder - IO-path workqueues */
+	qedn->fw_cq_fp_wq = alloc_workqueue(QEDN_FW_CQ_FP_WQ_WORKQUEUE,
+					    WQ_HIGHPRI | WQ_MEM_RECLAIM, 0);
+	if (!qedn->fw_cq_fp_wq) {
+		rc = -ENODEV;
+		pr_err("Unable to create fastpath FW CQ workqueue!\n");
+
+		return rc;
+	}
 
 	qedn->fp_q_arr = kcalloc(qedn->num_fw_cqs,
 				 sizeof(struct qedn_fp_queue), GFP_KERNEL);
@@ -681,8 +751,7 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
 		chain_params.mode = QED_CHAIN_MODE_PBL,
 		chain_params.cnt_type = QED_CHAIN_CNT_TYPE_U16,
 		chain_params.num_elems = QEDN_FW_CQ_SIZE;
-		/* Placeholder - sizeof(struct nvmetcp_fw_cqe)*/
-		chain_params.elem_size = 64;
+		chain_params.elem_size = sizeof(struct nvmetcp_fw_cqe);
 
 		rc = qed_ops->common->chain_alloc(qedn->cdev,
 						  &fp_q->cq_chain,
@@ -711,8 +780,7 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
 		sb = fp_q->sb_info->sb_virt;
 		fp_q->cq_prod = (u16 *)&sb->pi_array[QEDN_PROTO_CQ_PROD_IDX];
 		fp_q->qedn = qedn;
-
-		/* Placeholder - Init IO-path workqueue */
+		INIT_WORK(&fp_q->fw_cq_fp_wq_entry, qedn_fw_cq_fq_wq_handler);
 
 		/* Placeholder - Init IO-path resources */
 	}
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
new file mode 100644
index 000000000000..a3af55fba95c
--- /dev/null
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -0,0 +1,79 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021 Marvell. All rights reserved.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+ /* Kernel includes */
+#include <linux/kernel.h>
+
+/* Driver includes */
+#include "qedn.h"
+
+int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
+		       struct nvme_tcp_ofld_req *req)
+{
+	/* Process the request */
+
+	return 0;
+}
+
+struct qedn_task_ctx *qedn_cqe_get_active_task(struct nvmetcp_fw_cqe *cqe)
+{
+	struct regpair *p = &cqe->task_opaque;
+
+	return (struct qedn_task_ctx *)((((u64)(le32_to_cpu(p->hi)) << 32)
+					+ le32_to_cpu(p->lo)));
+}
+
+void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
+{
+	struct qedn_task_ctx *qedn_task = NULL;
+	struct qedn_conn_ctx *conn_ctx = NULL;
+	u16 itid;
+	u32 cid;
+
+	conn_ctx = qedn_get_conn_hash(qedn, le16_to_cpu(cqe->conn_id));
+	if (unlikely(!conn_ctx)) {
+		pr_err("CID 0x%x: Failed to fetch conn_ctx from hash\n",
+		       le16_to_cpu(cqe->conn_id));
+
+		return;
+	}
+
+	cid = conn_ctx->fw_cid;
+	itid = le16_to_cpu(cqe->itid);
+	qedn_task = qedn_cqe_get_active_task(cqe);
+	if (unlikely(!qedn_task))
+		return;
+
+	if (likely(cqe->cqe_type == NVMETCP_FW_CQE_TYPE_NORMAL)) {
+		/* Placeholder - verify the connection was established */
+
+		switch (cqe->task_type) {
+		case NVMETCP_TASK_TYPE_HOST_WRITE:
+		case NVMETCP_TASK_TYPE_HOST_READ:
+
+			/* Placeholder - IO flow */
+
+			break;
+
+		case NVMETCP_TASK_TYPE_HOST_READ_NO_CQE:
+
+			/* Placeholder - IO flow */
+
+			break;
+
+		case NVMETCP_TASK_TYPE_INIT_CONN_REQUEST:
+
+			/* Placeholder - ICReq flow */
+
+			break;
+		default:
+			pr_info("Could not identify task type\n");
+		}
+	} else {
+		/* Placeholder - Recovery flows */
+	}
+}
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:39 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:39 +0300
Subject: [PATCH v4 16/20] qedn: Add support of Task and SGL
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-17-pkushwaha@marvell.com>

This patch will add support of Task and SGL which is used
for slowpath and fast path IO. here Task is IO granule used
by firmware to perform tasks

The internal implementation:
- Create task/sgl resources used by all connection
- Provide APIs to allocate and free task.
- Add task support during connection establishment i.e. slowpath

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
---
 drivers/nvme/hw/qedn/qedn.h      |  61 ++++++
 drivers/nvme/hw/qedn/qedn_conn.c |  46 ++++-
 drivers/nvme/hw/qedn/qedn_main.c |  34 +++-
 drivers/nvme/hw/qedn/qedn_task.c | 326 +++++++++++++++++++++++++++++++
 4 files changed, 463 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 13d63f420a23..78efa8c02810 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -40,6 +40,16 @@
 
 #define QEDN_FW_CQ_FP_WQ_WORKQUEUE "qedn_fw_cq_fp_wq"
 
+/* Protocol defines */
+#define QEDN_MAX_IO_SIZE QED_NVMETCP_MAX_IO_SIZE
+
+#define QEDN_FW_SLOW_IO_MIN_SGE_LIMIT (9700 / 6)
+
+#define QEDN_MAX_HW_SECTORS (QEDN_MAX_IO_SIZE / 512)
+#define QEDN_MAX_SEGMENTS 2048
+
+#define QEDN_INVALID_ITID 0xFFFF
+
 /*
  * TCP offload stack default configurations and defines.
  * Future enhancements will allow controlling the configurable
@@ -84,6 +94,15 @@ enum qedn_state {
 	QEDN_STATE_MODULE_REMOVE_ONGOING,
 };
 
+struct qedn_io_resources {
+	/* Lock for IO resources */
+	spinlock_t resources_lock;
+	struct list_head task_free_list;
+	u32 num_alloc_tasks;
+	u32 num_free_tasks;
+	u32 no_avail_resrc_cnt;
+};
+
 /* Per CPU core params */
 struct qedn_fp_queue {
 	struct qed_chain cq_chain;
@@ -93,6 +112,10 @@ struct qedn_fp_queue {
 	struct qed_sb_info *sb_info;
 	unsigned int cpu;
 	struct work_struct fw_cq_fp_wq_entry;
+
+	/* IO related resources for host */
+	struct qedn_io_resources host_resrc;
+
 	u16 sb_id;
 	char irqname[QEDN_IRQ_NAME_LEN];
 };
@@ -116,12 +139,35 @@ struct qedn_ctx {
 	/* Connections */
 	DECLARE_HASHTABLE(conn_ctx_hash, 16);
 
+	u32 num_tasks_per_pool;
+
 	/* Fast path queues */
 	u8 num_fw_cqs;
 	struct qedn_fp_queue *fp_q_arr;
 	struct nvmetcp_glbl_queue_entry *fw_cq_array_virt;
 	dma_addr_t fw_cq_array_phy; /* Physical address of fw_cq_array_virt */
 	struct workqueue_struct *fw_cq_fp_wq;
+
+	/* Fast Path Tasks */
+	struct qed_nvmetcp_tid	tasks;
+};
+
+struct qedn_task_ctx {
+	struct qedn_conn_ctx *qedn_conn;
+	struct qedn_ctx *qedn;
+	void *fw_task_ctx;
+	struct qedn_fp_queue *fp_q;
+	struct scatterlist *nvme_sg;
+	struct nvme_tcp_ofld_req *req; /* currently proccessed request */
+	struct list_head entry;
+	spinlock_t lock; /* To protect task resources */
+	bool valid;
+	unsigned long flags; /* Used by qedn_task_flags */
+	u32 task_size;
+	u16 itid;
+	u16 cccid;
+	int req_direction;
+	struct storage_sgl_task_params sgl_task_params;
 };
 
 struct qedn_endpoint {
@@ -219,6 +265,7 @@ struct qedn_conn_ctx {
 	struct nvme_tcp_ofld_ctrl *ctrl;
 	u32 conn_handle;
 	u32 fw_cid;
+	u8 default_cq;
 
 	atomic_t est_conn_indicator;
 	atomic_t destroy_conn_indicator;
@@ -236,6 +283,11 @@ struct qedn_conn_ctx {
 	dma_addr_t host_cccid_itid_phy_addr;
 	struct qedn_endpoint ep;
 	int abrt_flag;
+	/* Spinlock for accessing active_task_list */
+	spinlock_t task_list_lock;
+	struct list_head active_task_list;
+	atomic_t num_active_tasks;
+	atomic_t num_active_fw_tasks;
 
 	/* Connection resources - turned on to indicate what resource was
 	 * allocated, to that it can later be released.
@@ -255,6 +307,7 @@ struct qedn_conn_ctx {
 enum qedn_conn_resources_state {
 	QEDN_CONN_RESRC_FW_SQ,
 	QEDN_CONN_RESRC_ACQUIRE_CONN,
+	QEDN_CONN_RESRC_TASKS,
 	QEDN_CONN_RESRC_CCCID_ITID_MAP,
 	QEDN_CONN_RESRC_TCP_PORT,
 	QEDN_CONN_RESRC_DB_ADD,
@@ -278,5 +331,13 @@ int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 		       struct nvme_tcp_ofld_req *req);
 void qedn_nvme_req_fp_wq_handler(struct work_struct *work);
 void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe);
+int qedn_alloc_tasks(struct qedn_conn_ctx *conn_ctx);
+inline int qedn_qid(struct nvme_tcp_ofld_queue *queue);
+void qedn_common_clear_fw_sgl(struct storage_sgl_task_params *sgl_task_params);
+void qedn_return_active_tasks(struct qedn_conn_ctx *conn_ctx);
+struct qedn_task_ctx *
+qedn_get_free_task_from_pool(struct qedn_conn_ctx *conn_ctx, u16 cccid);
+void qedn_destroy_free_tasks(struct qedn_fp_queue *fp_q,
+			     struct qedn_io_resources *io_resrc);
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
index 962c88a4f345..5c5b365df522 100644
--- a/drivers/nvme/hw/qedn/qedn_conn.c
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -29,6 +29,11 @@ static const char * const qedn_conn_state_str[] = {
 	NULL
 };
 
+inline int qedn_qid(struct nvme_tcp_ofld_queue *queue)
+{
+	return queue - queue->ctrl->queues;
+}
+
 int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
 		       enum qedn_conn_state new_state)
 {
@@ -170,6 +175,11 @@ static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx)
 		clear_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state);
 	}
 
+	if (test_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state)) {
+		clear_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state);
+			qedn_return_active_tasks(conn_ctx);
+	}
+
 	if (test_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state)) {
 		dma_free_coherent(&qedn->pdev->dev,
 				  conn_ctx->sq_depth *
@@ -272,6 +282,7 @@ static int qedn_nvmetcp_offload_conn(struct qedn_conn_ctx *conn_ctx)
 	offld_prms.max_rt_time = QEDN_TCP_MAX_RT_TIME;
 	offld_prms.sq_pbl_addr =
 		(u64)qed_chain_get_pbl_phys(&qedn_ep->fw_sq_chain);
+	offld_prms.default_cq = conn_ctx->default_cq;
 
 	rc = qed_ops->offload_conn(qedn->cdev,
 				   conn_ctx->conn_handle,
@@ -420,7 +431,10 @@ void qedn_prep_db_data(struct qedn_conn_ctx *conn_ctx)
 static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 {
 	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct qedn_io_resources *io_resrc;
+	struct qedn_fp_queue *fp_q;
 	size_t dma_size;
+	u8 qid;
 	int rc;
 
 	rc = qedn_alloc_fw_sq(qedn, &conn_ctx->ep);
@@ -431,6 +445,9 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 
 	set_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state);
 
+	atomic_set(&conn_ctx->num_active_tasks, 0);
+	atomic_set(&conn_ctx->num_active_fw_tasks, 0);
+
 	rc = qed_ops->acquire_conn(qedn->cdev,
 				   &conn_ctx->conn_handle,
 				   &conn_ctx->fw_cid,
@@ -444,7 +461,34 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 		 conn_ctx->conn_handle);
 	set_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state);
 
-	/* Placeholder - Allocate task resources and initialize fields */
+	qid = qedn_qid(conn_ctx->queue);
+
+	/* default_cq is mapped 1:1 with qid (with cpu core) which are
+	 * assigned for the driver
+	 */
+	conn_ctx->default_cq = qid ? qid - 1 : 0;
+	fp_q = &qedn->fp_q_arr[conn_ctx->default_cq];
+	conn_ctx->fp_q = fp_q;
+	io_resrc = &fp_q->host_resrc;
+
+	/* The first connection on each fp_q will fill task
+	 * resources
+	 */
+	spin_lock(&io_resrc->resources_lock);
+	if (io_resrc->num_alloc_tasks == 0) {
+		rc = qedn_alloc_tasks(conn_ctx);
+		if (rc) {
+			pr_err("Failed allocating tasks: CID=0x%x\n",
+			       conn_ctx->fw_cid);
+			spin_unlock(&io_resrc->resources_lock);
+			goto rel_conn;
+		}
+	}
+	spin_unlock(&io_resrc->resources_lock);
+
+	spin_lock_init(&conn_ctx->task_list_lock);
+	INIT_LIST_HEAD(&conn_ctx->active_task_list);
+	set_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state);
 
 	rc = qedn_fetch_tcp_port(conn_ctx);
 	if (rc)
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 975949ce6fb0..42c8ad6ac2d6 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -29,6 +29,12 @@ __be16 qedn_get_in_port(struct sockaddr_storage *sa)
 		: ((struct sockaddr_in6 *)sa)->sin6_port;
 }
 
+static void qedn_init_io_resc(struct qedn_io_resources *io_resrc)
+{
+	spin_lock_init(&io_resrc->resources_lock);
+	INIT_LIST_HEAD(&io_resrc->task_free_list);
+}
+
 struct qedn_llh_filter *qedn_add_llh_filter(struct qedn_ctx *qedn, u16 tcp_port)
 {
 	struct qedn_llh_filter *llh_filter = NULL;
@@ -436,6 +442,8 @@ static struct nvme_tcp_ofld_ops qedn_ofld_ops = {
 		 *	NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST |
 		 *	NVMF_OPT_NR_POLL_QUEUES | NVMF_OPT_TOS
 		 */
+	.max_hw_sectors = QEDN_MAX_HW_SECTORS,
+	.max_segments = QEDN_MAX_SEGMENTS,
 	.claim_dev = qedn_claim_dev,
 	.setup_ctrl = qedn_setup_ctrl,
 	.release_ctrl = qedn_release_ctrl,
@@ -640,8 +648,24 @@ static inline int qedn_core_probe(struct qedn_ctx *qedn)
 	return rc;
 }
 
+static void qedn_call_destroy_free_tasks(struct qedn_fp_queue *fp_q,
+					 struct qedn_io_resources *io_resrc)
+{
+	if (list_empty(&io_resrc->task_free_list))
+		return;
+
+	if (io_resrc->num_alloc_tasks != io_resrc->num_free_tasks)
+		pr_err("Task Pool:Not all returned allocated=0x%x, free=0x%x\n",
+		       io_resrc->num_alloc_tasks, io_resrc->num_free_tasks);
+
+	qedn_destroy_free_tasks(fp_q, io_resrc);
+	if (io_resrc->num_free_tasks)
+		pr_err("Expected num_free_tasks to be 0\n");
+}
+
 static void qedn_free_function_queues(struct qedn_ctx *qedn)
 {
+	struct qedn_io_resources *host_resrc;
 	struct qed_sb_info *sb_info = NULL;
 	struct qedn_fp_queue *fp_q;
 	int i;
@@ -653,6 +677,9 @@ static void qedn_free_function_queues(struct qedn_ctx *qedn)
 	/* Free the fast path queues*/
 	for (i = 0; i < qedn->num_fw_cqs; i++) {
 		fp_q = &qedn->fp_q_arr[i];
+		host_resrc = &fp_q->host_resrc;
+
+		qedn_call_destroy_free_tasks(fp_q, host_resrc);
 
 		/* Free SB */
 		sb_info = fp_q->sb_info;
@@ -740,7 +767,8 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
 		goto mem_alloc_failure;
 	}
 
-	/* placeholder - create task pools */
+	qedn->num_tasks_per_pool =
+		qedn->pf_params.nvmetcp_pf_params.num_tasks / qedn->num_fw_cqs;
 
 	for (i = 0; i < qedn->num_fw_cqs; i++) {
 		fp_q = &qedn->fp_q_arr[i];
@@ -782,7 +810,7 @@ static int qedn_alloc_function_queues(struct qedn_ctx *qedn)
 		fp_q->qedn = qedn;
 		INIT_WORK(&fp_q->fw_cq_fp_wq_entry, qedn_fw_cq_fq_wq_handler);
 
-		/* Placeholder - Init IO-path resources */
+		qedn_init_io_resc(&fp_q->host_resrc);
 	}
 
 	return 0;
@@ -964,7 +992,7 @@ static int __qedn_probe(struct pci_dev *pdev)
 
 	/* NVMeTCP start HW PF */
 	rc = qed_ops->start(qedn->cdev,
-			    NULL /* Placeholder for FW IO-path resources */,
+			    &qedn->tasks,
 			    qedn,
 			    qedn_event_cb);
 	if (rc) {
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
index a3af55fba95c..7b228b5c5169 100644
--- a/drivers/nvme/hw/qedn/qedn_task.c
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -11,6 +11,332 @@
 /* Driver includes */
 #include "qedn.h"
 
+static void qedn_free_nvme_sg(struct qedn_task_ctx *qedn_task)
+{
+	kfree(qedn_task->nvme_sg);
+	qedn_task->nvme_sg = NULL;
+}
+
+static void qedn_free_fw_sgl(struct qedn_task_ctx *qedn_task)
+{
+	struct qedn_ctx *qedn = qedn_task->qedn;
+	dma_addr_t sgl_pa;
+
+	sgl_pa = HILO_DMA_REGPAIR(qedn_task->sgl_task_params.sgl_phys_addr);
+	dma_free_coherent(&qedn->pdev->dev,
+			  QEDN_MAX_SEGMENTS * sizeof(struct nvmetcp_sge),
+			  qedn_task->sgl_task_params.sgl,
+			  sgl_pa);
+	qedn_task->sgl_task_params.sgl = NULL;
+}
+
+static void qedn_destroy_single_task(struct qedn_task_ctx *qedn_task)
+{
+	u16 itid;
+
+	itid = qedn_task->itid;
+	list_del(&qedn_task->entry);
+	qedn_free_nvme_sg(qedn_task);
+	qedn_free_fw_sgl(qedn_task);
+	kfree(qedn_task);
+	qedn_task = NULL;
+}
+
+void qedn_destroy_free_tasks(struct qedn_fp_queue *fp_q,
+			     struct qedn_io_resources *io_resrc)
+{
+	struct qedn_task_ctx *qedn_task, *task_tmp;
+
+	/* Destroy tasks from the free task list */
+	list_for_each_entry_safe(qedn_task, task_tmp,
+				 &io_resrc->task_free_list, entry) {
+		qedn_destroy_single_task(qedn_task);
+		io_resrc->num_free_tasks -= 1;
+	}
+}
+
+static int qedn_alloc_nvme_sg(struct qedn_task_ctx *qedn_task)
+{
+	int rc;
+
+	qedn_task->nvme_sg = kcalloc(QEDN_MAX_SEGMENTS,
+				     sizeof(*qedn_task->nvme_sg), GFP_KERNEL);
+	if (!qedn_task->nvme_sg) {
+		rc = -ENOMEM;
+
+		return rc;
+	}
+
+	return 0;
+}
+
+static int qedn_alloc_fw_sgl(struct qedn_task_ctx *qedn_task)
+{
+	struct nvmetcp_sge **sgl = &qedn_task->sgl_task_params.sgl;
+	struct qedn_ctx *qedn = qedn_task->qedn_conn->qedn;
+	dma_addr_t sgl_phys;
+	u32 sz;
+
+	sz = QEDN_MAX_SEGMENTS * sizeof(struct nvmetcp_sge);
+	*sgl = dma_alloc_coherent(&qedn->pdev->dev, sz, &sgl_phys, GFP_KERNEL);
+	if (!*sgl) {
+		pr_err("Couldn't allocate FW sgl\n");
+
+		return -ENOMEM;
+	}
+
+	DMA_REGPAIR_LE(qedn_task->sgl_task_params.sgl_phys_addr, sgl_phys);
+
+	return 0;
+}
+
+static inline void *qedn_get_fw_task(struct qed_nvmetcp_tid *info, u16 itid)
+{
+	return (void *)(info->blocks[itid / info->num_tids_per_block] +
+			(itid % info->num_tids_per_block) * info->size);
+}
+
+static struct qedn_task_ctx *qedn_alloc_task(struct qedn_conn_ctx *conn_ctx,
+					     u16 itid)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct qedn_task_ctx *qedn_task;
+	void *fw_task_ctx;
+	int rc = 0;
+
+	qedn_task = kzalloc(sizeof(*qedn_task), GFP_KERNEL);
+	if (!qedn_task)
+		return NULL;
+
+	spin_lock_init(&qedn_task->lock);
+	fw_task_ctx = qedn_get_fw_task(&qedn->tasks, itid);
+	if (!fw_task_ctx) {
+		pr_err("iTID: 0x%x; Failed getting fw_task_ctx memory\n", itid);
+		goto release_task;
+	}
+
+	/* No need to memset fw_task_ctx - its done in the HSI func */
+	qedn_task->qedn_conn = conn_ctx;
+	qedn_task->qedn = qedn;
+	qedn_task->fw_task_ctx = fw_task_ctx;
+	qedn_task->valid = 0;
+	qedn_task->flags = 0;
+	qedn_task->itid = itid;
+	rc = qedn_alloc_fw_sgl(qedn_task);
+	if (rc) {
+		pr_err("iTID: 0x%x; Failed allocating FW sgl\n", itid);
+		goto release_task;
+	}
+
+	rc = qedn_alloc_nvme_sg(qedn_task);
+	if (rc) {
+		pr_err("iTID: 0x%x; Failed allocating FW sgl\n", itid);
+		goto release_fw_sgl;
+	}
+
+	return qedn_task;
+
+release_fw_sgl:
+	qedn_free_fw_sgl(qedn_task);
+release_task:
+	kfree(qedn_task);
+
+	return NULL;
+}
+
+int qedn_alloc_tasks(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct qedn_task_ctx *qedn_task = NULL;
+	struct qedn_io_resources *io_resrc;
+	u16 itid, start_itid, offset;
+	struct qedn_fp_queue *fp_q;
+	int i, rc;
+
+	fp_q = conn_ctx->fp_q;
+
+	offset = fp_q->sb_id;
+	io_resrc = &fp_q->host_resrc;
+
+	start_itid = qedn->num_tasks_per_pool * offset;
+	for (i = 0; i < qedn->num_tasks_per_pool; ++i) {
+		itid = start_itid + i;
+		qedn_task = qedn_alloc_task(conn_ctx, itid);
+		if (!qedn_task) {
+			pr_err("Failed allocating task\n");
+			rc = -ENOMEM;
+			goto release_tasks;
+		}
+
+		qedn_task->fp_q = fp_q;
+		io_resrc->num_free_tasks += 1;
+		list_add_tail(&qedn_task->entry, &io_resrc->task_free_list);
+	}
+
+	io_resrc->num_alloc_tasks = io_resrc->num_free_tasks;
+
+	return 0;
+
+release_tasks:
+	qedn_destroy_free_tasks(fp_q, io_resrc);
+
+	return rc;
+}
+
+void qedn_common_clear_fw_sgl(struct storage_sgl_task_params *sgl_task_params)
+{
+	u16 sge_cnt = sgl_task_params->num_sges;
+
+	memset(&sgl_task_params->sgl[(sge_cnt - 1)], 0,
+	       sizeof(struct nvmetcp_sge));
+	sgl_task_params->total_buffer_size = 0;
+	sgl_task_params->small_mid_sge = false;
+	sgl_task_params->num_sges = 0;
+}
+
+inline void qedn_host_reset_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx,
+					     u16 cccid)
+{
+	conn_ctx->host_cccid_itid[cccid].itid = cpu_to_le16(QEDN_INVALID_ITID);
+}
+
+inline void qedn_host_set_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx,
+					   u16 cccid, u16 itid)
+{
+	conn_ctx->host_cccid_itid[cccid].itid = cpu_to_le16(itid);
+}
+
+static void qedn_clear_sgl(struct qedn_ctx *qedn,
+			   struct qedn_task_ctx *qedn_task)
+{
+	struct storage_sgl_task_params *sgl_task_params;
+	enum dma_data_direction dma_dir;
+	u32 sge_cnt;
+
+	sgl_task_params = &qedn_task->sgl_task_params;
+	sge_cnt = sgl_task_params->num_sges;
+
+	/* Nothing to do if no SGEs were used */
+	if (!qedn_task->task_size || !sge_cnt)
+		return;
+
+	dma_dir = (qedn_task->req_direction == WRITE ?
+			DMA_TO_DEVICE : DMA_FROM_DEVICE);
+	dma_unmap_sg(&qedn->pdev->dev, qedn_task->nvme_sg, sge_cnt, dma_dir);
+	memset(&qedn_task->nvme_sg[(sge_cnt - 1)], 0,
+	       sizeof(struct scatterlist));
+	qedn_common_clear_fw_sgl(sgl_task_params);
+	qedn_task->task_size = 0;
+}
+
+static void qedn_clear_task(struct qedn_conn_ctx *conn_ctx,
+			    struct qedn_task_ctx *qedn_task)
+{
+	/* Task lock isn't needed since it is no longer in use */
+	qedn_clear_sgl(conn_ctx->qedn, qedn_task);
+	qedn_task->valid = 0;
+	qedn_task->flags = 0;
+
+	atomic_dec(&conn_ctx->num_active_tasks);
+}
+
+void qedn_return_active_tasks(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_fp_queue *fp_q = conn_ctx->fp_q;
+	struct qedn_task_ctx *qedn_task, *task_tmp;
+	struct qedn_io_resources *io_resrc;
+	int num_returned_tasks = 0;
+	int num_active_tasks;
+
+	io_resrc = &fp_q->host_resrc;
+
+	/* Return tasks that aren't "Used by FW" to the pool */
+	list_for_each_entry_safe(qedn_task, task_tmp,
+				 &conn_ctx->active_task_list, entry) {
+		qedn_clear_task(conn_ctx, qedn_task);
+		num_returned_tasks++;
+	}
+
+	if (num_returned_tasks) {
+		spin_lock(&io_resrc->resources_lock);
+		/* Return tasks to FP_Q pool in one shot */
+
+		list_splice_tail_init(&conn_ctx->active_task_list,
+				      &io_resrc->task_free_list);
+		io_resrc->num_free_tasks += num_returned_tasks;
+		spin_unlock(&io_resrc->resources_lock);
+	}
+
+	num_active_tasks = atomic_read(&conn_ctx->num_active_tasks);
+	if (num_active_tasks)
+		pr_err("num_active_tasks is %u after cleanup.\n",
+		       num_active_tasks);
+}
+
+void qedn_return_task_to_pool(struct qedn_conn_ctx *conn_ctx,
+			      struct qedn_task_ctx *qedn_task)
+{
+	struct qedn_fp_queue *fp_q = conn_ctx->fp_q;
+	struct qedn_io_resources *io_resrc;
+	unsigned long lock_flags;
+
+	io_resrc = &fp_q->host_resrc;
+
+	spin_lock_irqsave(&qedn_task->lock, lock_flags);
+	qedn_task->valid = 0;
+	qedn_task->flags = 0;
+	qedn_clear_sgl(conn_ctx->qedn, qedn_task);
+	spin_unlock_irqrestore(&qedn_task->lock, lock_flags);
+
+	spin_lock(&conn_ctx->task_list_lock);
+	list_del(&qedn_task->entry);
+	qedn_host_reset_cccid_itid_entry(conn_ctx, qedn_task->cccid);
+	spin_unlock(&conn_ctx->task_list_lock);
+
+	atomic_dec(&conn_ctx->num_active_tasks);
+	atomic_dec(&conn_ctx->num_active_fw_tasks);
+
+	spin_lock(&io_resrc->resources_lock);
+	list_add_tail(&qedn_task->entry, &io_resrc->task_free_list);
+	io_resrc->num_free_tasks += 1;
+	spin_unlock(&io_resrc->resources_lock);
+}
+
+struct qedn_task_ctx *
+qedn_get_free_task_from_pool(struct qedn_conn_ctx *conn_ctx, u16 cccid)
+{
+	struct qedn_task_ctx *qedn_task = NULL;
+	struct qedn_io_resources *io_resrc;
+	struct qedn_fp_queue *fp_q;
+
+	fp_q = conn_ctx->fp_q;
+	io_resrc = &fp_q->host_resrc;
+
+	spin_lock(&io_resrc->resources_lock);
+	qedn_task = list_first_entry_or_null(&io_resrc->task_free_list,
+					     struct qedn_task_ctx, entry);
+	if (unlikely(!qedn_task)) {
+		spin_unlock(&io_resrc->resources_lock);
+
+		return NULL;
+	}
+	list_del(&qedn_task->entry);
+	io_resrc->num_free_tasks -= 1;
+	spin_unlock(&io_resrc->resources_lock);
+
+	spin_lock(&conn_ctx->task_list_lock);
+	list_add_tail(&qedn_task->entry, &conn_ctx->active_task_list);
+	qedn_host_set_cccid_itid_entry(conn_ctx, cccid, qedn_task->itid);
+	spin_unlock(&conn_ctx->task_list_lock);
+
+	atomic_inc(&conn_ctx->num_active_tasks);
+	qedn_task->cccid = cccid;
+	qedn_task->qedn_conn = conn_ctx;
+	qedn_task->valid = 1;
+
+	return qedn_task;
+}
+
 int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 		       struct nvme_tcp_ofld_req *req)
 {
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:40 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:40 +0300
Subject: [PATCH v4 17/20] qedn: Add support of NVME ICReq & ICResp
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-18-pkushwaha@marvell.com>

Once a TCP connection established, the host sends an Initialize
Connection Request (ICReq) PDU to the controller.
Further Initialize Connection Response (ICResp) PDU received from
controller is processed by host to establish a connection and
exchange connection configuration parameters.

This patch present support of generation of ICReq and processing of
ICResp. It also update host configuration based on exchanged parameters.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |  38 ++++
 drivers/nvme/hw/qedn/qedn_conn.c | 334 ++++++++++++++++++++++++++++++-
 drivers/nvme/hw/qedn/qedn_main.c |  14 ++
 drivers/nvme/hw/qedn/qedn_task.c |   8 +-
 4 files changed, 390 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 78efa8c02810..829d474b3ab1 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -16,6 +16,7 @@
 
 /* Driver includes */
 #include "../../host/tcp-offload.h"
+#include <linux/nvme-tcp.h>
 
 #define QEDN_MODULE_NAME "qedn"
 
@@ -42,6 +43,8 @@
 
 /* Protocol defines */
 #define QEDN_MAX_IO_SIZE QED_NVMETCP_MAX_IO_SIZE
+#define QEDN_MAX_PDU_SIZE 0x80000 /* 512KB */
+#define QEDN_MAX_OUTSTANDING_R2T_PDUS 0 /* 0 Based == 1 max R2T */
 
 #define QEDN_FW_SLOW_IO_MIN_SGE_LIMIT (9700 / 6)
 
@@ -50,6 +53,13 @@
 
 #define QEDN_INVALID_ITID 0xFFFF
 
+#define QEDN_ICREQ_FW_PAYLOAD (sizeof(struct nvme_tcp_icreq_pdu) -\
+			       QED_NVMETCP_NON_IO_HDR_SIZE)
+#define QEDN_ICREQ_FW_PAYLOAD_START 8
+
+/* The FW will handle the ICReq as CCCID 0 (FW internal design) */
+#define QEDN_ICREQ_CCCID 0
+
 /*
  * TCP offload stack default configurations and defines.
  * Future enhancements will allow controlling the configurable
@@ -120,6 +130,16 @@ struct qedn_fp_queue {
 	char irqname[QEDN_IRQ_NAME_LEN];
 };
 
+struct qedn_negotiation_params {
+	u32 maxh2cdata; /* Negotiation */
+	u32 maxr2t; /* Validation */
+	u16 pfv; /* Validation */
+	bool hdr_digest; /* Negotiation */
+	bool data_digest; /* Negotiation */
+	u8 cpda; /* Negotiation */
+	u8 hpda; /* Validation */
+};
+
 struct qedn_ctx {
 	struct pci_dev *pdev;
 	struct qed_dev *cdev;
@@ -176,6 +196,9 @@ struct qedn_endpoint {
 	struct nvmetcp_db_data db_data;
 	void __iomem *p_doorbell;
 
+	/* Spinlock for accessing FW queue */
+	spinlock_t doorbell_lock;
+
 	/* TCP Params */
 	__be32 dst_addr[4]; /* In network order */
 	__be32 src_addr[4]; /* In network order */
@@ -252,6 +275,12 @@ struct qedn_ctrl {
 	atomic_t host_num_active_conns;
 };
 
+struct qedn_icreq_padding {
+	u32 *buffer;
+	dma_addr_t pa;
+	struct nvmetcp_sge sge;
+};
+
 /* Connection level struct */
 struct qedn_conn_ctx {
 	/* IO path */
@@ -300,6 +329,11 @@ struct qedn_conn_ctx {
 
 	size_t sq_depth;
 
+	struct qedn_negotiation_params required_params;
+	struct qedn_negotiation_params pdu_params;
+	struct nvme_tcp_icresp_pdu icresp;
+	struct qedn_icreq_padding *icreq_pad;
+
 	/* "dummy" socket */
 	struct socket *sock;
 };
@@ -308,6 +342,7 @@ enum qedn_conn_resources_state {
 	QEDN_CONN_RESRC_FW_SQ,
 	QEDN_CONN_RESRC_ACQUIRE_CONN,
 	QEDN_CONN_RESRC_TASKS,
+	QEDN_CONN_RESRC_ICREQ_PAD,
 	QEDN_CONN_RESRC_CCCID_ITID_MAP,
 	QEDN_CONN_RESRC_TCP_PORT,
 	QEDN_CONN_RESRC_DB_ADD,
@@ -339,5 +374,8 @@ struct qedn_task_ctx *
 qedn_get_free_task_from_pool(struct qedn_conn_ctx *conn_ctx, u16 cccid);
 void qedn_destroy_free_tasks(struct qedn_fp_queue *fp_q,
 			     struct qedn_io_resources *io_resrc);
+void qedn_prep_icresp(struct qedn_conn_ctx *conn_ctx,
+		      struct nvmetcp_fw_cqe *cqe);
+void qedn_ring_doorbell(struct qedn_conn_ctx *conn_ctx);
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
index 5c5b365df522..b4c0a1a3e890 100644
--- a/drivers/nvme/hw/qedn/qedn_conn.c
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -34,6 +34,18 @@ inline int qedn_qid(struct nvme_tcp_ofld_queue *queue)
 	return queue - queue->ctrl->queues;
 }
 
+void qedn_ring_doorbell(struct qedn_conn_ctx *conn_ctx)
+{
+	u16 prod_idx;
+
+	prod_idx = qed_chain_get_prod_idx(&conn_ctx->ep.fw_sq_chain);
+	conn_ctx->ep.db_data.sq_prod = cpu_to_le16(prod_idx);
+
+	/* wmb - Make sure fw idx is coherent */
+	wmb();
+	writel(*(u32 *)&conn_ctx->ep.db_data, conn_ctx->ep.p_doorbell);
+}
+
 int qedn_set_con_state(struct qedn_conn_ctx *conn_ctx,
 		       enum qedn_conn_state new_state)
 {
@@ -143,6 +155,71 @@ int qedn_initialize_endpoint(struct qedn_endpoint *ep, u8 *local_mac_addr,
 	return -1;
 }
 
+static int qedn_alloc_icreq_pad(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct qedn_icreq_padding *icreq_pad;
+	u32 *buffer;
+	int rc = 0;
+
+	icreq_pad = kzalloc(sizeof(*icreq_pad), GFP_KERNEL);
+	if (!icreq_pad)
+		return -ENOMEM;
+
+	conn_ctx->icreq_pad = icreq_pad;
+	memset(&icreq_pad->sge, 0, sizeof(icreq_pad->sge));
+	buffer = dma_alloc_coherent(&qedn->pdev->dev,
+				    QEDN_ICREQ_FW_PAYLOAD,
+				    &icreq_pad->pa,
+				    GFP_KERNEL);
+	if (!buffer) {
+		pr_err("Could not allocate icreq_padding SGE buffer.\n");
+		rc =  -ENOMEM;
+		goto release_icreq_pad;
+	}
+
+	DMA_REGPAIR_LE(icreq_pad->sge.sge_addr, icreq_pad->pa);
+	icreq_pad->sge.sge_len = cpu_to_le32(QEDN_ICREQ_FW_PAYLOAD);
+	icreq_pad->buffer = buffer;
+	set_bit(QEDN_CONN_RESRC_ICREQ_PAD, &conn_ctx->resrc_state);
+
+	return 0;
+
+release_icreq_pad:
+	kfree(icreq_pad);
+	conn_ctx->icreq_pad = NULL;
+
+	return rc;
+}
+
+static void qedn_free_icreq_pad(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct qedn_icreq_padding *icreq_pad;
+	u32 *buffer;
+
+	icreq_pad = conn_ctx->icreq_pad;
+	if (unlikely(!icreq_pad)) {
+		pr_err("null ptr in icreq_pad in conn_ctx\n");
+		goto finally;
+	}
+
+	buffer = icreq_pad->buffer;
+	if (buffer) {
+		dma_free_coherent(&qedn->pdev->dev,
+				  QEDN_ICREQ_FW_PAYLOAD,
+				  (void *)buffer,
+				  icreq_pad->pa);
+		icreq_pad->buffer = NULL;
+	}
+
+	kfree(icreq_pad);
+	conn_ctx->icreq_pad = NULL;
+
+finally:
+	clear_bit(QEDN_CONN_RESRC_ICREQ_PAD, &conn_ctx->resrc_state);
+}
+
 static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx)
 {
 	struct qedn_ctx *qedn = conn_ctx->qedn;
@@ -175,6 +252,9 @@ static void qedn_release_conn_ctx(struct qedn_conn_ctx *conn_ctx)
 		clear_bit(QEDN_CONN_RESRC_ACQUIRE_CONN, &conn_ctx->resrc_state);
 	}
 
+	if (test_bit(QEDN_CONN_RESRC_ICREQ_PAD, &conn_ctx->resrc_state))
+		qedn_free_icreq_pad(conn_ctx);
+
 	if (test_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state)) {
 		clear_bit(QEDN_CONN_RESRC_TASKS, &conn_ctx->resrc_state);
 			qedn_return_active_tasks(conn_ctx);
@@ -336,6 +416,215 @@ void qedn_terminate_connection(struct qedn_conn_ctx *conn_ctx)
 	queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
 }
 
+static int qedn_nvmetcp_update_conn(struct qedn_ctx *qedn,
+				    struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_negotiation_params *pdu_params = &conn_ctx->pdu_params;
+	struct qed_nvmetcp_params_update *conn_info;
+	int rc;
+
+	conn_info = kzalloc(sizeof(*conn_info), GFP_KERNEL);
+	if (!conn_info)
+		return -ENOMEM;
+
+	conn_info->hdr_digest_en = pdu_params->hdr_digest;
+	conn_info->data_digest_en = pdu_params->data_digest;
+	conn_info->max_recv_pdu_length = QEDN_MAX_PDU_SIZE;
+	conn_info->max_io_size = QEDN_MAX_IO_SIZE;
+	conn_info->max_send_pdu_length = pdu_params->maxh2cdata;
+
+	rc = qed_ops->update_conn(qedn->cdev, conn_ctx->conn_handle, conn_info);
+	if (rc) {
+		pr_err("Could not update connection\n");
+		rc = -ENXIO;
+	}
+
+	kfree(conn_info);
+
+	return rc;
+}
+
+static int qedn_update_ramrod(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int rc = 0;
+
+	rc = qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_UPDATE_EQE);
+	if (rc)
+		return rc;
+
+	rc = qedn_nvmetcp_update_conn(qedn, conn_ctx);
+	if (rc)
+		return rc;
+
+	if (conn_ctx->state != CONN_STATE_WAIT_FOR_UPDATE_EQE) {
+		pr_err("cid 0x%x: Unexpected state 0x%x after update ramrod\n",
+		       conn_ctx->fw_cid, conn_ctx->state);
+
+		return -EINVAL;
+	}
+
+	return rc;
+}
+
+static int qedn_send_icreq(struct qedn_conn_ctx *conn_ctx)
+{
+	struct storage_sgl_task_params *sgl_task_params;
+	struct nvmetcp_task_params task_params;
+	struct qedn_task_ctx *qedn_task = NULL;
+	struct nvme_tcp_icreq_pdu icreq;
+	struct nvmetcp_wqe *chain_sqe;
+	struct nvmetcp_wqe local_sqe;
+
+	qedn_task = qedn_get_free_task_from_pool(conn_ctx, QEDN_ICREQ_CCCID);
+	if (!qedn_task)
+		return -EINVAL;
+
+	memset(&icreq, 0, sizeof(icreq));
+	memset(&local_sqe, 0, sizeof(local_sqe));
+
+	/* Initialize ICReq */
+	icreq.hdr.type = nvme_tcp_icreq;
+	icreq.hdr.hlen = sizeof(icreq);
+	icreq.hdr.pdo = 0;
+	icreq.hdr.plen = cpu_to_le32(icreq.hdr.hlen);
+	icreq.pfv = cpu_to_le16(conn_ctx->required_params.pfv);
+	icreq.maxr2t = cpu_to_le32(conn_ctx->required_params.maxr2t);
+	icreq.hpda = conn_ctx->required_params.hpda;
+	if (conn_ctx->required_params.hdr_digest)
+		icreq.digest |= NVME_TCP_HDR_DIGEST_ENABLE;
+	if (conn_ctx->required_params.data_digest)
+		icreq.digest |= NVME_TCP_DATA_DIGEST_ENABLE;
+
+	/* Initialize task params */
+	task_params.opq.lo = cpu_to_le32(((u64)(qedn_task)) & 0xffffffff);
+	task_params.opq.hi = cpu_to_le32(((u64)(qedn_task)) >> 32);
+	task_params.context = qedn_task->fw_task_ctx;
+	task_params.sqe = &local_sqe;
+	task_params.conn_icid = (u16)conn_ctx->conn_handle;
+	task_params.itid = qedn_task->itid;
+	task_params.cq_rss_number = conn_ctx->default_cq;
+	task_params.tx_io_size = QEDN_ICREQ_FW_PAYLOAD;
+	task_params.rx_io_size = 0; /* Rx doesn't use SGL for icresp */
+
+	/* Init SGE for ICReq padding */
+	sgl_task_params = &qedn_task->sgl_task_params;
+	sgl_task_params->total_buffer_size = task_params.tx_io_size;
+	sgl_task_params->small_mid_sge = false;
+	sgl_task_params->num_sges = 1;
+	memcpy(sgl_task_params->sgl, &conn_ctx->icreq_pad->sge,
+	       sizeof(conn_ctx->icreq_pad->sge));
+
+	/* ICReq is sent as two parts.
+	 * First part: (16 bytes + First 8 bytes of icreq.rsvd2[]) are sent
+	 *             via task context which is initialized above in icreq
+	 * Second part: Rest bytes are sent via SGE, happening here
+	 */
+	memcpy(conn_ctx->icreq_pad->buffer,
+	       &icreq.rsvd2[QEDN_ICREQ_FW_PAYLOAD_START],
+	       QEDN_ICREQ_FW_PAYLOAD);
+
+	qed_ops->init_icreq_exchange(&task_params, &icreq,
+				     sgl_task_params,  NULL);
+
+	qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_IC_COMP);
+	atomic_inc(&conn_ctx->num_active_fw_tasks);
+
+	/* spin_lock - doorbell is accessed  both Rx flow and response flow */
+	spin_lock(&conn_ctx->ep.doorbell_lock);
+	chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain);
+	memcpy(chain_sqe, &local_sqe, sizeof(local_sqe));
+	qedn_ring_doorbell(conn_ctx);
+	spin_unlock(&conn_ctx->ep.doorbell_lock);
+
+	return 0;
+}
+
+void qedn_prep_icresp(struct qedn_conn_ctx *conn_ctx,
+		      struct nvmetcp_fw_cqe *cqe)
+{
+	struct nvmetcp_icresp_mdata *icresp_from_cqe =
+		(struct nvmetcp_icresp_mdata *)&cqe->cqe_data.icresp_mdata;
+	struct nvme_tcp_icresp_pdu *icresp = &conn_ctx->icresp;
+	struct nvme_tcp_ofld_ctrl *ctrl = conn_ctx->ctrl;
+	struct qedn_ctrl *qctrl = NULL;
+
+	qctrl = (struct qedn_ctrl *)ctrl->private_data;
+	if (!qctrl)
+		return;
+
+	icresp->pfv = cpu_to_le16(icresp_from_cqe->pfv);
+	icresp->cpda = icresp_from_cqe->cpda;
+	icresp->digest = icresp_from_cqe->digest;
+	icresp->maxdata = cpu_to_le32(icresp_from_cqe->maxdata);
+
+	qedn_set_sp_wa(conn_ctx, HANDLE_ICRESP);
+	queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
+}
+
+static int qedn_handle_icresp(struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_tcp_icresp_pdu *icresp = &conn_ctx->icresp;
+	struct qedn_negotiation_params *pdu_params;
+	int rc = 0;
+	u16 pfv;
+
+	/* Swapping requirement will be removed in future FW versions */
+	pfv = __swab16(le16_to_cpu(icresp->pfv));
+
+	qedn_free_icreq_pad(conn_ctx);
+
+	/* Validate ICResp */
+	if (pfv != conn_ctx->required_params.pfv) {
+		pr_err("cid %u: unsupported pfv %u\n", conn_ctx->fw_cid, pfv);
+
+		return -EINVAL;
+	}
+
+	if (icresp->cpda > conn_ctx->required_params.cpda) {
+		pr_err("cid %u: unsupported cpda %u\n",
+		       conn_ctx->fw_cid, icresp->cpda);
+
+		return -EINVAL;
+	}
+
+	if ((NVME_TCP_HDR_DIGEST_ENABLE & icresp->digest) !=
+	    conn_ctx->required_params.hdr_digest) {
+		if ((NVME_TCP_HDR_DIGEST_ENABLE & icresp->digest) >
+		    conn_ctx->required_params.hdr_digest) {
+			pr_err("cid 0x%x: invalid header digest bit\n",
+			       conn_ctx->fw_cid);
+		}
+	}
+
+	if ((NVME_TCP_DATA_DIGEST_ENABLE & icresp->digest) !=
+	    conn_ctx->required_params.data_digest) {
+		if ((NVME_TCP_DATA_DIGEST_ENABLE & icresp->digest) >
+		    conn_ctx->required_params.data_digest) {
+			pr_err("cid 0x%x: invalid data digest bit\n",
+			       conn_ctx->fw_cid);
+	}
+	}
+
+	pdu_params = &conn_ctx->pdu_params;
+	memset(pdu_params, 0, sizeof(conn_ctx->pdu_params));
+	/* Swapping requirement will be removed in future FW versions */
+	pdu_params->maxh2cdata = __swab32(le32_to_cpu(icresp->maxdata));
+	pdu_params->maxh2cdata = QEDN_MAX_PDU_SIZE;
+	if (pdu_params->maxh2cdata > QEDN_MAX_PDU_SIZE)
+		pdu_params->maxh2cdata = QEDN_MAX_PDU_SIZE;
+
+	pdu_params->pfv = pfv;
+	pdu_params->cpda = icresp->cpda;
+	pdu_params->hpda = conn_ctx->required_params.hpda;
+	pdu_params->hdr_digest = NVME_TCP_HDR_DIGEST_ENABLE & icresp->digest;
+	pdu_params->data_digest = NVME_TCP_DATA_DIGEST_ENABLE & icresp->digest;
+	pdu_params->maxr2t = conn_ctx->required_params.maxr2t;
+	rc = qedn_update_ramrod(conn_ctx);
+
+	return rc;
+}
+
 /* Slowpath EQ Callback */
 int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
 {
@@ -393,7 +682,8 @@ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
 			if (rc)
 				return rc;
 
-			/* Placeholder - for ICReq flow */
+			qedn_set_sp_wa(conn_ctx, SEND_ICREQ);
+			queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
 		}
 
 		break;
@@ -445,6 +735,8 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 
 	set_bit(QEDN_CONN_RESRC_FW_SQ, &conn_ctx->resrc_state);
 
+	spin_lock_init(&conn_ctx->ep.doorbell_lock);
+
 	atomic_set(&conn_ctx->num_active_tasks, 0);
 	atomic_set(&conn_ctx->num_active_fw_tasks, 0);
 
@@ -509,6 +801,11 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 
 	memset(conn_ctx->host_cccid_itid, 0xFF, dma_size);
 	set_bit(QEDN_CONN_RESRC_CCCID_ITID_MAP, &conn_ctx->resrc_state);
+
+	rc = qedn_alloc_icreq_pad(conn_ctx);
+		if (rc)
+			goto rel_conn;
+
 	rc = qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_CONNECT_DONE);
 	if (rc)
 		goto rel_conn;
@@ -581,6 +878,9 @@ void qedn_sp_wq_handler(struct work_struct *work)
 
 	qedn = conn_ctx->qedn;
 	if (test_bit(DESTROY_CONNECTION, &conn_ctx->agg_work_action)) {
+		if (test_bit(HANDLE_ICRESP, &conn_ctx->agg_work_action))
+			qedn_clr_sp_wa(conn_ctx, HANDLE_ICRESP);
+
 		qedn_destroy_connection(conn_ctx);
 
 		return;
@@ -595,6 +895,38 @@ void qedn_sp_wq_handler(struct work_struct *work)
 			return;
 		}
 	}
+
+	if (test_bit(SEND_ICREQ, &conn_ctx->agg_work_action)) {
+		qedn_clr_sp_wa(conn_ctx, SEND_ICREQ);
+		rc = qedn_send_icreq(conn_ctx);
+		if (rc)
+			return;
+
+		return;
+	}
+
+	if (test_bit(HANDLE_ICRESP, &conn_ctx->agg_work_action)) {
+		rc = qedn_handle_icresp(conn_ctx);
+
+		qedn_clr_sp_wa(conn_ctx, HANDLE_ICRESP);
+		if (rc) {
+			pr_err("IC handling returned with 0x%x\n", rc);
+			if (test_and_set_bit(DESTROY_CONNECTION,
+					     &conn_ctx->agg_work_action))
+				return;
+
+			qedn_destroy_connection(conn_ctx);
+
+			return;
+		}
+
+		atomic_inc(&conn_ctx->est_conn_indicator);
+		qedn_set_con_state(conn_ctx,
+				   CONN_STATE_NVMETCP_CONN_ESTABLISHED);
+		wake_up_interruptible(&conn_ctx->conn_waitq);
+
+		return;
+	}
 }
 
 /* Clear connection aggregative slowpath work action */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 42c8ad6ac2d6..3cf913d527c0 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -286,6 +286,19 @@ static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 	return 0;
 }
 
+static void qedn_set_pdu_params(struct qedn_conn_ctx *conn_ctx)
+{
+	/* Enable digest once supported */
+	conn_ctx->required_params.hdr_digest = 0;
+	conn_ctx->required_params.data_digest = 0;
+
+	conn_ctx->required_params.maxr2t = QEDN_MAX_OUTSTANDING_R2T_PDUS;
+	conn_ctx->required_params.pfv = NVME_TCP_PFV_1_0;
+	conn_ctx->required_params.cpda = 0;
+	conn_ctx->required_params.hpda = 0;
+	conn_ctx->required_params.maxh2cdata = QEDN_MAX_PDU_SIZE;
+}
+
 static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 			     size_t queue_size)
 {
@@ -317,6 +330,7 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 	conn_ctx->ctrl = ctrl;
 	conn_ctx->sq_depth = queue_size;
 	mutex_init(&conn_ctx->send_mutex);
+	qedn_set_pdu_params(conn_ctx);
 
 	init_waitqueue_head(&conn_ctx->conn_waitq);
 	atomic_set(&conn_ctx->est_conn_indicator, 0);
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
index 7b228b5c5169..e52460f9650e 100644
--- a/drivers/nvme/hw/qedn/qedn_task.c
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -392,9 +392,11 @@ void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
 			break;
 
 		case NVMETCP_TASK_TYPE_INIT_CONN_REQUEST:
-
-			/* Placeholder - ICReq flow */
-
+			/* Clear ICReq-padding SGE from SGL */
+			qedn_common_clear_fw_sgl(&qedn_task->sgl_task_params);
+			/* Task is not required for icresp processing */
+			qedn_return_task_to_pool(conn_ctx, qedn_task);
+			qedn_prep_icresp(conn_ctx, cqe);
 			break;
 		default:
 			pr_info("Could not identify task type\n");
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:42 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:42 +0300
Subject: [PATCH v4 19/20] qedn: Add Connection and IO level recovery flows
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-20-pkushwaha@marvell.com>

From: Shai Malin <smalin at marvell.com>

This patch will present the connection level functionalities:
 - conn clear-sq: will release the FW restrictions in order to flush all
   the pending IOs.
 - drain: in case clear-sq is stuck, will release all the device FW
   restrictions in order to flush all the pending IOs.
 - task cleanup - will flush the IO level resources.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |   8 ++
 drivers/nvme/hw/qedn/qedn_conn.c | 135 ++++++++++++++++++++++++++++++-
 drivers/nvme/hw/qedn/qedn_main.c |   6 ++
 drivers/nvme/hw/qedn/qedn_task.c |  30 ++++++-
 4 files changed, 176 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index b36994be65cb..5539791fc49b 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -41,6 +41,8 @@
 
 #define QEDN_FW_CQ_FP_WQ_WORKQUEUE "qedn_fw_cq_fp_wq"
 
+#define QEDN_DRAIN_MAX_ATTEMPTS 3
+
 /* Protocol defines */
 #define QEDN_MAX_IO_SIZE QED_NVMETCP_MAX_IO_SIZE
 #define QEDN_MAX_PDU_SIZE 0x80000 /* 512KB */
@@ -91,6 +93,8 @@
 /* Timeouts and delay constants */
 #define QEDN_WAIT_CON_ESTABLSH_TMO 10000 /* 10 seconds */
 #define QEDN_RLS_CONS_TMO 5000 /* 5 sec */
+#define QEDN_TASK_CLEANUP_TMO 3000 /* 3 sec */
+#define QEDN_DRAIN_TMO 1000 /* 1 sec */
 
 enum qedn_state {
 	QEDN_STATE_CORE_PROBED = 0,
@@ -173,7 +177,9 @@ struct qedn_ctx {
 };
 
 enum qedn_task_flags {
+	QEDN_TASK_IS_ICREQ,
 	QEDN_TASK_USED_BY_FW,
+	QEDN_TASK_WAIT_FOR_CLEANUP,
 };
 
 struct qedn_task_ctx {
@@ -321,6 +327,8 @@ struct qedn_conn_ctx {
 	struct list_head active_task_list;
 	atomic_t num_active_tasks;
 	atomic_t num_active_fw_tasks;
+	atomic_t task_cleanups_cnt;
+	wait_queue_head_t cleanup_waitq;
 
 	/* Connection resources - turned on to indicate what resource was
 	 * allocated, to that it can later be released.
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
index ea072eff34a6..a61e77bf8bd0 100644
--- a/drivers/nvme/hw/qedn/qedn_conn.c
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -626,6 +626,11 @@ static int qedn_handle_icresp(struct qedn_conn_ctx *conn_ctx)
 	return rc;
 }
 
+void qedn_error_recovery(struct nvme_ctrl *nctrl)
+{
+	nvme_tcp_ofld_error_recovery(nctrl);
+}
+
 /* Slowpath EQ Callback */
 int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
 {
@@ -688,6 +693,7 @@ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
 		}
 
 		break;
+
 	case NVMETCP_EVENT_TYPE_ASYN_TERMINATE_DONE:
 		if (conn_ctx->state != CONN_STATE_WAIT_FOR_DESTROY_DONE)
 			pr_err("CID=0x%x:ASYN_TERMINATE_DONE:Wrong state %u\n",
@@ -696,6 +702,19 @@ int qedn_event_cb(void *context, u8 fw_event_code, void *event_ring_data)
 			queue_work(qctrl->sp_wq, &conn_ctx->sp_wq_entry);
 
 		break;
+
+	case NVMETCP_EVENT_TYPE_ASYN_CLOSE_RCVD:
+	case NVMETCP_EVENT_TYPE_ASYN_ABORT_RCVD:
+	case NVMETCP_EVENT_TYPE_ASYN_MAX_RT_TIME:
+	case NVMETCP_EVENT_TYPE_ASYN_MAX_RT_CNT:
+	case NVMETCP_EVENT_TYPE_ASYN_SYN_RCVD:
+	case NVMETCP_EVENT_TYPE_ASYN_MAX_KA_PROBES_CNT:
+	case NVMETCP_EVENT_TYPE_NVMETCP_CONN_ERROR:
+	case NVMETCP_EVENT_TYPE_TCP_CONN_ERROR:
+		qedn_error_recovery(&conn_ctx->ctrl->nctrl);
+
+		break;
+
 	default:
 		pr_err("CID=0x%x - Recv Unknown Event %u\n",
 		       conn_ctx->fw_cid, fw_event_code);
@@ -835,9 +854,123 @@ static int qedn_prep_and_offload_queue(struct qedn_conn_ctx *conn_ctx)
 	return -EINVAL;
 }
 
+static void qedn_cleanup_fw_task(struct qedn_ctx *qedn,
+				 struct qedn_task_ctx *qedn_task)
+{
+	struct qedn_conn_ctx *conn_ctx = qedn_task->qedn_conn;
+	struct nvmetcp_task_params task_params;
+	struct nvmetcp_wqe *chain_sqe;
+	struct nvmetcp_wqe local_sqe;
+	unsigned long lock_flags;
+
+	/* Take lock to prevent race with fastpath, we don't want to
+	 * invoke cleanup flows on tasks that already returned.
+	 */
+	spin_lock_irqsave(&qedn_task->lock, lock_flags);
+	if (!qedn_task->valid) {
+		spin_unlock_irqrestore(&qedn_task->lock, lock_flags);
+
+		return;
+	}
+	/* Skip tasks not used by FW */
+	if (!test_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags)) {
+		spin_unlock_irqrestore(&qedn_task->lock, lock_flags);
+
+		return;
+	}
+	/* Skip tasks that were already invoked for cleanup */
+	if (unlikely(test_bit(QEDN_TASK_WAIT_FOR_CLEANUP, &qedn_task->flags))) {
+		spin_unlock_irqrestore(&qedn_task->lock, lock_flags);
+
+		return;
+	}
+	set_bit(QEDN_TASK_WAIT_FOR_CLEANUP, &qedn_task->flags);
+	spin_unlock_irqrestore(&qedn_task->lock, lock_flags);
+
+	atomic_inc(&conn_ctx->task_cleanups_cnt);
+
+	task_params.sqe = &local_sqe;
+	task_params.itid = qedn_task->itid;
+	qed_ops->init_task_cleanup(&task_params);
+
+	/* spin_lock - doorbell is accessed  both Rx flow and response flow */
+	spin_lock(&conn_ctx->ep.doorbell_lock);
+	chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain);
+	memcpy(chain_sqe, &local_sqe, sizeof(local_sqe));
+	qedn_ring_doorbell(conn_ctx);
+	spin_unlock(&conn_ctx->ep.doorbell_lock);
+}
+
+inline int qedn_drain(struct qedn_conn_ctx *conn_ctx)
+{
+	u32 drain_timeout = msecs_to_jiffies(QEDN_DRAIN_TMO);
+	atomic_t *clnup_cnt = &conn_ctx->task_cleanups_cnt;
+	int drain_iter = QEDN_DRAIN_MAX_ATTEMPTS;
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int wrc;
+
+	while (drain_iter) {
+		qed_ops->common->drain(qedn->cdev);
+		msleep(100);
+
+		wrc = wait_event_interruptible_timeout(conn_ctx->cleanup_waitq,
+						       !atomic_read(clnup_cnt),
+						       drain_timeout);
+		if (!wrc) {
+			drain_iter--;
+			continue;
+		}
+
+		return 0;
+	}
+
+	pr_err("CID 0x%x: cleanup after drain failed - need hard reset.\n",
+	       conn_ctx->fw_cid);
+
+	return -EINVAL;
+}
+
+void qedn_cleanup_all_fw_tasks(struct qedn_conn_ctx *conn_ctx)
+{
+	u32 clnup_timeout = msecs_to_jiffies(QEDN_TASK_CLEANUP_TMO);
+	atomic_t *clnup_cnt = &conn_ctx->task_cleanups_cnt;
+	struct qedn_task_ctx *qedn_task, *task_tmp;
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int wrc;
+
+	list_for_each_entry_safe_reverse(qedn_task, task_tmp,
+					 &conn_ctx->active_task_list, entry) {
+		qedn_cleanup_fw_task(qedn, qedn_task);
+	}
+
+	wrc = wait_event_interruptible_timeout(conn_ctx->cleanup_waitq,
+					       atomic_read(clnup_cnt) == 0,
+					       clnup_timeout);
+	if (!wrc) {
+		if (qedn_drain(conn_ctx))
+			return;
+	}
+}
+
+static void qedn_clear_fw_sq(struct qedn_conn_ctx *conn_ctx)
+{
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	int rc;
+
+	rc = qed_ops->clear_sq(qedn->cdev, conn_ctx->conn_handle);
+	if (rc)
+		pr_warn("clear_sq failed - rc %u\n", rc);
+}
+
 void qedn_cleanp_fw(struct qedn_conn_ctx *conn_ctx)
 {
-	/* Placeholder - task cleanup */
+	if (atomic_read(&conn_ctx->num_active_fw_tasks)) {
+		conn_ctx->abrt_flag = QEDN_ABORTIVE_TERMINATION;
+		qedn_clear_fw_sq(conn_ctx);
+		qedn_cleanup_all_fw_tasks(conn_ctx);
+	} else {
+		conn_ctx->abrt_flag = QEDN_NON_ABORTIVE_TERMINATION;
+	}
 }
 
 void qedn_destroy_connection(struct qedn_conn_ctx *conn_ctx)
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index fb47e315ab03..d18cc3624764 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -262,12 +262,17 @@ static int qedn_setup_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 
 static int qedn_release_ctrl(struct nvme_tcp_ofld_ctrl *ctrl)
 {
+	struct nvme_ctrl *nctrl = &ctrl->nctrl;
 	struct qedn_ctrl *qctrl;
 
 	qctrl = (struct qedn_ctrl *)ctrl->private_data;
 	if (!qctrl)
 		return -ENODEV;
 
+	if (nctrl->state == NVME_CTRL_CONNECTING ||
+	    nctrl->state == NVME_CTRL_RESETTING)
+		return 0;
+
 	if (test_and_clear_bit(LLH_FILTER, &qctrl->agg_state) &&
 	    qctrl->llh_filter) {
 		qedn_dec_llh_filter(qctrl->qedn, qctrl->llh_filter);
@@ -333,6 +338,7 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 	qedn_set_pdu_params(conn_ctx);
 
 	init_waitqueue_head(&conn_ctx->conn_waitq);
+	init_waitqueue_head(&conn_ctx->cleanup_waitq);
 	atomic_set(&conn_ctx->est_conn_indicator, 0);
 	atomic_set(&conn_ctx->destroy_conn_indicator, 0);
 
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
index dd0b5f31c052..6664cc587f06 100644
--- a/drivers/nvme/hw/qedn/qedn_task.c
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -324,6 +324,17 @@ void qedn_return_active_tasks(struct qedn_conn_ctx *conn_ctx)
 	/* Return tasks that aren't "Used by FW" to the pool */
 	list_for_each_entry_safe(qedn_task, task_tmp,
 				 &conn_ctx->active_task_list, entry) {
+		/* If we got this far, cleanup was already done
+		 * in which case we want to return the task to the pool and
+		 * release it. So we make sure the cleanup indication is down
+		 */
+		clear_bit(QEDN_TASK_WAIT_FOR_CLEANUP, &qedn_task->flags);
+
+		/* Special handling in case of ICREQ task */
+		if (unlikely(conn_ctx->state ==	CONN_STATE_WAIT_FOR_IC_COMP &&
+			     test_bit(QEDN_TASK_IS_ICREQ, &(qedn_task)->flags)))
+			qedn_common_clear_fw_sgl(&qedn_task->sgl_task_params);
+
 		qedn_clear_task(conn_ctx, qedn_task);
 		num_returned_tasks++;
 	}
@@ -669,7 +680,9 @@ void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
 		return;
 
 	if (likely(cqe->cqe_type == NVMETCP_FW_CQE_TYPE_NORMAL)) {
-		/* Placeholder - verify the connection was established */
+		if (unlikely(test_bit(QEDN_TASK_WAIT_FOR_CLEANUP,
+				      &qedn_task->flags)))
+			return;
 
 		switch (cqe->task_type) {
 		case NVMETCP_TASK_TYPE_HOST_WRITE:
@@ -711,6 +724,19 @@ void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
 			pr_info("Could not identify task type\n");
 		}
 	} else {
-		/* Placeholder - Recovery flows */
+		if (cqe->cqe_type == NVMETCP_FW_CQE_TYPE_CLEANUP) {
+			clear_bit(QEDN_TASK_WAIT_FOR_CLEANUP,
+				  &qedn_task->flags);
+			qedn_return_task_to_pool(conn_ctx, qedn_task);
+			atomic_dec(&conn_ctx->task_cleanups_cnt);
+			wake_up_interruptible(&conn_ctx->cleanup_waitq);
+
+			return;
+		}
+
+		 /* The else is NVMETCP_FW_CQE_TYPE_DUMMY - in which don't
+		  * return the task. The task will return during
+		  * NVMETCP_FW_CQE_TYPE_CLEANUP.
+		  */
 	}
 }
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:41 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:41 +0300
Subject: [PATCH v4 18/20] qedn: Add IO level fastpath functionality
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-19-pkushwaha@marvell.com>

From: Shai Malin <smalin at marvell.com>

This patch will present the IO level functionality of qedn
nvme-tcp-offload host mode. The qedn_task_ctx structure is containing
various params and state of the current IO, and is mapped 1x1 to the
fw_task_ctx which is a HW and FW IO context.
A qedn_task is mapped directly to its parent connection.
For every new IO a qedn_task structure will be assigned and they will be
linked for the entire IO's life span.

The patch will include 2 flows:
  1. Send new command to the FW:
	 The flow is: nvme_tcp_ofld_queue_rq() which invokes qedn_send_req()
	 which invokes qedn_queue_request() which will:
         - Assign fw_task_ctx.
	 - Prepare the Read/Write SG buffer.
	 - Initialize the HW and FW context.
	 - Pass the IO to the FW.

  2. Process the IO completion:
     The flow is: qedn_irq_handler() which invokes qedn_fw_cq_fp_handler()
	 which invokes qedn_io_work_cq() which will:
	 - process the FW completion.
	 - Return the fw_task_ctx to the task pool.
	 - complete the nvme req.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |   5 +
 drivers/nvme/hw/qedn/qedn_conn.c |   1 +
 drivers/nvme/hw/qedn/qedn_main.c |   8 +
 drivers/nvme/hw/qedn/qedn_task.c | 317 ++++++++++++++++++++++++++++++-
 4 files changed, 327 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 829d474b3ab1..b36994be65cb 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -172,6 +172,10 @@ struct qedn_ctx {
 	struct qed_nvmetcp_tid	tasks;
 };
 
+enum qedn_task_flags {
+	QEDN_TASK_USED_BY_FW,
+};
+
 struct qedn_task_ctx {
 	struct qedn_conn_ctx *qedn_conn;
 	struct qedn_ctx *qedn;
@@ -376,6 +380,7 @@ void qedn_destroy_free_tasks(struct qedn_fp_queue *fp_q,
 			     struct qedn_io_resources *io_resrc);
 void qedn_prep_icresp(struct qedn_conn_ctx *conn_ctx,
 		      struct nvmetcp_fw_cqe *cqe);
+void qedn_swap_bytes(u32 *p, int size);
 void qedn_ring_doorbell(struct qedn_conn_ctx *conn_ctx);
 
 #endif /* _QEDN_H_ */
diff --git a/drivers/nvme/hw/qedn/qedn_conn.c b/drivers/nvme/hw/qedn/qedn_conn.c
index b4c0a1a3e890..ea072eff34a6 100644
--- a/drivers/nvme/hw/qedn/qedn_conn.c
+++ b/drivers/nvme/hw/qedn/qedn_conn.c
@@ -528,6 +528,7 @@ static int qedn_send_icreq(struct qedn_conn_ctx *conn_ctx)
 				     sgl_task_params,  NULL);
 
 	qedn_set_con_state(conn_ctx, CONN_STATE_WAIT_FOR_IC_COMP);
+	set_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags);
 	atomic_inc(&conn_ctx->num_active_fw_tasks);
 
 	/* spin_lock - doorbell is accessed  both Rx flow and response flow */
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index 3cf913d527c0..fb47e315ab03 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -1047,6 +1047,14 @@ static int qedn_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	return __qedn_probe(pdev);
 }
 
+void qedn_swap_bytes(u32 *p, int size)
+{
+	int i;
+
+	for (i = 0; i < size; ++i, ++p)
+		*p = __swab32(*p);
+}
+
 static struct pci_driver qedn_pci_driver = {
 	.name     = QEDN_MODULE_NAME,
 	.id_table = qedn_pci_tbl,
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
index e52460f9650e..dd0b5f31c052 100644
--- a/drivers/nvme/hw/qedn/qedn_task.c
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -11,6 +11,77 @@
 /* Driver includes */
 #include "qedn.h"
 
+extern const struct qed_nvmetcp_ops *qed_ops;
+
+static bool qedn_sgl_has_small_mid_sge(struct nvmetcp_sge *sgl, u16 sge_count)
+{
+	u16 sge_num;
+
+	if (sge_count > 8) {
+		for (sge_num = 0; sge_num < sge_count; sge_num++) {
+			if (le32_to_cpu(sgl[sge_num].sge_len) <
+			    QEDN_FW_SLOW_IO_MIN_SGE_LIMIT)
+				return true; /* small middle SGE found */
+		}
+	}
+
+	return false; /* no small middle SGEs */
+}
+
+static int qedn_init_sgl(struct qedn_ctx *qedn, struct qedn_task_ctx *qedn_task)
+{
+	struct storage_sgl_task_params *sgl_task_params;
+	enum dma_data_direction dma_dir;
+	struct scatterlist *sg;
+	struct request *rq;
+	u16 num_sges;
+	int index;
+	u32 len;
+	int rc;
+
+	sgl_task_params = &qedn_task->sgl_task_params;
+	rq = blk_mq_rq_from_pdu(qedn_task->req);
+	if (qedn_task->task_size == 0) {
+		sgl_task_params->num_sges = 0;
+
+		return 0;
+	}
+
+	/* Convert BIO to scatterlist */
+	num_sges = blk_rq_map_sg(rq->q, rq, qedn_task->nvme_sg);
+	if (qedn_task->req_direction == WRITE)
+		dma_dir = DMA_TO_DEVICE;
+	else
+		dma_dir = DMA_FROM_DEVICE;
+
+	/* DMA map the scatterlist */
+	if (dma_map_sg(&qedn->pdev->dev, qedn_task->nvme_sg,
+		       num_sges, dma_dir) != num_sges) {
+		pr_err("Couldn't map sgl\n");
+		rc = -EPERM;
+
+		return rc;
+	}
+
+	sgl_task_params->total_buffer_size = qedn_task->task_size;
+	sgl_task_params->num_sges = num_sges;
+
+	for_each_sg(qedn_task->nvme_sg, sg, num_sges, index) {
+		DMA_REGPAIR_LE(sgl_task_params->sgl[index].sge_addr,
+			       sg_dma_address(sg));
+		len = sg_dma_len(sg);
+		sgl_task_params->sgl[index].sge_len = cpu_to_le32(len);
+	}
+
+	/* Relevant for Host Write Only */
+	sgl_task_params->small_mid_sge = (qedn_task->req_direction == READ) ?
+		false :
+		qedn_sgl_has_small_mid_sge(sgl_task_params->sgl,
+					   sgl_task_params->num_sges);
+
+	return 0;
+}
+
 static void qedn_free_nvme_sg(struct qedn_task_ctx *qedn_task)
 {
 	kfree(qedn_task->nvme_sg);
@@ -337,12 +408,168 @@ qedn_get_free_task_from_pool(struct qedn_conn_ctx *conn_ctx, u16 cccid)
 	return qedn_task;
 }
 
+int qedn_send_read_cmd(struct qedn_task_ctx *qedn_task,
+		       struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_command *nvme_cmd = &qedn_task->req->nvme_cmd;
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct nvmetcp_task_params task_params;
+	struct nvme_tcp_cmd_pdu cmd_hdr;
+	struct nvmetcp_wqe *chain_sqe;
+	struct nvmetcp_wqe local_sqe;
+	int rc;
+
+	rc = qedn_init_sgl(qedn, qedn_task);
+	if (rc)
+		return rc;
+
+	task_params.opq.lo = cpu_to_le32(((u64)(qedn_task)) & 0xffffffff);
+	task_params.opq.hi = cpu_to_le32(((u64)(qedn_task)) >> 32);
+
+	/* Initialize task params */
+	task_params.context = qedn_task->fw_task_ctx;
+	task_params.sqe = &local_sqe;
+	task_params.tx_io_size = 0;
+	task_params.rx_io_size = qedn_task->task_size;
+	task_params.conn_icid = (u16)conn_ctx->conn_handle;
+	task_params.itid = qedn_task->itid;
+	task_params.cq_rss_number = conn_ctx->default_cq;
+	task_params.send_write_incapsule = 0;
+
+	cmd_hdr.hdr.type = nvme_tcp_cmd;
+	cmd_hdr.hdr.flags = 0;
+	cmd_hdr.hdr.hlen = sizeof(cmd_hdr);
+	cmd_hdr.hdr.pdo = 0x0;
+	cmd_hdr.hdr.plen = cpu_to_le32(cmd_hdr.hdr.hlen);
+
+	qed_ops->init_read_io(&task_params, &cmd_hdr, nvme_cmd,
+			      &qedn_task->sgl_task_params);
+
+	set_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags);
+	atomic_inc(&conn_ctx->num_active_fw_tasks);
+
+	spin_lock(&conn_ctx->ep.doorbell_lock);
+	chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain);
+	memcpy(chain_sqe, &local_sqe, sizeof(local_sqe));
+	qedn_ring_doorbell(conn_ctx);
+	spin_unlock(&conn_ctx->ep.doorbell_lock);
+
+	return 0;
+}
+
+int qedn_send_write_cmd(struct qedn_task_ctx *qedn_task,
+			struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_command *nvme_cmd = &qedn_task->req->nvme_cmd;
+	struct nvmetcp_task_params task_params;
+	struct qedn_ctx *qedn = conn_ctx->qedn;
+	struct nvme_tcp_cmd_pdu cmd_hdr;
+	u32 pdu_len = sizeof(cmd_hdr);
+	struct nvmetcp_wqe *chain_sqe;
+	struct nvmetcp_wqe local_sqe;
+	u8 send_write_incapsule;
+	int rc;
+
+	if (qedn_task->task_size <=
+	    nvme_tcp_ofld_inline_data_size(conn_ctx->queue) &&
+	    qedn_task->task_size) {
+		send_write_incapsule = 1;
+		pdu_len += qedn_task->task_size;
+
+		/* Add digest length once supported */
+		cmd_hdr.hdr.pdo = sizeof(cmd_hdr);
+	} else {
+		send_write_incapsule = 0;
+
+		cmd_hdr.hdr.pdo = 0x0;
+	}
+
+	rc = qedn_init_sgl(qedn, qedn_task);
+	if (rc)
+		return rc;
+
+	task_params.host_cccid = cpu_to_le16(qedn_task->cccid);
+	task_params.opq.lo = cpu_to_le32(((u64)(qedn_task)) & 0xffffffff);
+	task_params.opq.hi = cpu_to_le32(((u64)(qedn_task)) >> 32);
+
+	/* Initialize task params */
+	task_params.context = qedn_task->fw_task_ctx;
+	task_params.sqe = &local_sqe;
+	task_params.tx_io_size = qedn_task->task_size;
+	task_params.rx_io_size = 0;
+	task_params.conn_icid = (u16)conn_ctx->conn_handle;
+	task_params.itid = qedn_task->itid;
+	task_params.cq_rss_number = conn_ctx->default_cq;
+	task_params.send_write_incapsule = send_write_incapsule;
+
+	cmd_hdr.hdr.type = nvme_tcp_cmd;
+	cmd_hdr.hdr.flags = 0;
+	cmd_hdr.hdr.hlen = sizeof(cmd_hdr);
+	cmd_hdr.hdr.plen = cpu_to_le32(pdu_len);
+
+	qed_ops->init_write_io(&task_params, &cmd_hdr, nvme_cmd,
+			       &qedn_task->sgl_task_params);
+
+	set_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags);
+	atomic_inc(&conn_ctx->num_active_fw_tasks);
+
+	spin_lock(&conn_ctx->ep.doorbell_lock);
+	chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain);
+	memcpy(chain_sqe, &local_sqe, sizeof(local_sqe));
+	qedn_ring_doorbell(conn_ctx);
+	spin_unlock(&conn_ctx->ep.doorbell_lock);
+
+	return 0;
+}
+
 int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 		       struct nvme_tcp_ofld_req *req)
 {
-	/* Process the request */
+	struct qedn_task_ctx *qedn_task;
+	struct request *rq;
+	int rc = 0;
+	u16 cccid;
 
-	return 0;
+	rq = blk_mq_rq_from_pdu(req);
+
+	/* Placeholder - async */
+
+	cccid = rq->tag;
+	qedn_task = qedn_get_free_task_from_pool(qedn_conn, cccid);
+	if (unlikely(!qedn_task)) {
+		pr_err("Not able to allocate task context resource\n");
+
+		return BLK_STS_NOTSUPP;
+	}
+
+	req->private_data = qedn_task;
+	qedn_task->req = req;
+
+	/* Placeholder - handle (req->async) */
+
+	/* Check if there are physical segments in request to determine the
+	 * task size. The logic of nvme_tcp_set_sg_null() will be implemented
+	 * as a part of qedn_set_sg_host_data().
+	 */
+	qedn_task->task_size = blk_rq_nr_phys_segments(rq) ?
+				blk_rq_payload_bytes(rq) : 0;
+	qedn_task->req_direction = rq_data_dir(rq);
+	if (qedn_task->req_direction == WRITE)
+		rc = qedn_send_write_cmd(qedn_task, qedn_conn);
+	else
+		rc = qedn_send_read_cmd(qedn_task, qedn_conn);
+
+	if (unlikely(rc)) {
+		pr_err("Read/Write command failure\n");
+
+		return BLK_STS_TRANSPORT;
+	}
+
+	spin_lock(&qedn_conn->ep.doorbell_lock);
+	qedn_ring_doorbell(qedn_conn);
+	spin_unlock(&qedn_conn->ep.doorbell_lock);
+
+	return BLK_STS_OK;
 }
 
 struct qedn_task_ctx *qedn_cqe_get_active_task(struct nvmetcp_fw_cqe *cqe)
@@ -353,8 +580,75 @@ struct qedn_task_ctx *qedn_cqe_get_active_task(struct nvmetcp_fw_cqe *cqe)
 					+ le32_to_cpu(p->lo)));
 }
 
+static struct nvme_tcp_ofld_req *qedn_decouple_req_task(struct qedn_task_ctx
+							*qedn_task)
+{
+	struct nvme_tcp_ofld_req *ulp_req = qedn_task->req;
+
+	qedn_task->req = NULL;
+	if (ulp_req)
+		ulp_req->private_data = NULL;
+
+	return ulp_req;
+}
+
+static inline int qedn_comp_valid_task(struct qedn_task_ctx *qedn_task,
+				       union nvme_result *result, __le16 status)
+{
+	struct qedn_conn_ctx *conn_ctx = qedn_task->qedn_conn;
+	struct nvme_tcp_ofld_req *req;
+
+	req = qedn_decouple_req_task(qedn_task);
+	qedn_return_task_to_pool(conn_ctx, qedn_task);
+	if (!req) {
+		pr_err("req not found\n");
+
+		return -EINVAL;
+	}
+
+	/* Call request done to complete the request */
+	if (req->done)
+		req->done(req, result, status);
+	else
+		pr_err("request done not Set !!!\n");
+
+	return 0;
+}
+
+int qedn_process_nvme_cqe(struct qedn_task_ctx *qedn_task,
+			  struct nvme_completion *cqe)
+{
+	int rc = 0;
+
+	/* CQE arrives swapped
+	 * Swapping requirement will be removed in future FW versions
+	 */
+	qedn_swap_bytes((u32 *)cqe, (sizeof(*cqe) / sizeof(u32)));
+
+	/* Placeholder - async */
+
+	rc = qedn_comp_valid_task(qedn_task, &cqe->result, cqe->status);
+
+	return rc;
+}
+
+int qedn_complete_c2h(struct qedn_task_ctx *qedn_task)
+{
+	int rc = 0;
+
+	__le16 status = cpu_to_le16(NVME_SC_SUCCESS << 1);
+	union nvme_result result = {};
+
+	rc = qedn_comp_valid_task(qedn_task, &result, status);
+
+	return rc;
+}
+
 void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
 {
+	int rc = 0;
+
+	struct nvme_completion *nvme_cqe = NULL;
 	struct qedn_task_ctx *qedn_task = NULL;
 	struct qedn_conn_ctx *conn_ctx = NULL;
 	u16 itid;
@@ -381,13 +675,28 @@ void qedn_io_work_cq(struct qedn_ctx *qedn, struct nvmetcp_fw_cqe *cqe)
 		case NVMETCP_TASK_TYPE_HOST_WRITE:
 		case NVMETCP_TASK_TYPE_HOST_READ:
 
-			/* Placeholder - IO flow */
+			/* Verify data digest once supported */
 
+			nvme_cqe = (struct nvme_completion *)
+						&cqe->cqe_data.nvme_cqe;
+			rc = qedn_process_nvme_cqe(qedn_task, nvme_cqe);
+			if (rc) {
+				pr_err("Read/Write completion error\n");
+
+				return;
+			}
 			break;
 
 		case NVMETCP_TASK_TYPE_HOST_READ_NO_CQE:
 
-			/* Placeholder - IO flow */
+			/* Verify data digest once supported */
+
+			rc = qedn_complete_c2h(qedn_task);
+			if (rc) {
+				pr_err("Controller To Host Data Transfer error error\n");
+
+				return;
+			}
 
 			break;
 
-- 
2.24.1



From pkushwaha at marvell.com  Tue Jun 29 05:47:43 2021
From: pkushwaha at marvell.com (Prabhakar Kushwaha)
Date: Tue, 29 Jun 2021 15:47:43 +0300
Subject: [PATCH v4 20/20] qedn: Add support of ASYNC
In-Reply-To: <20210629124743.6898-1-pkushwaha@marvell.com>
References: <20210629124743.6898-1-pkushwaha@marvell.com>
Message-ID: <20210629124743.6898-21-pkushwaha@marvell.com>

This patch implement ASYNC request and response event notification
handling at qedn driver level.

NVME Ofld layer's ASYNC request is treated similar to read with
fake CCCID. This CCCID used to route ASYNC notification back to
the NVME ofld layer.

Acked-by: Igor Russkikh <irusskikh at marvell.com>
Signed-off-by: Prabhakar Kushwaha <pkushwaha at marvell.com>
Signed-off-by: Omkar Kulkarni <okulkarni at marvell.com>
Signed-off-by: Michal Kalderon <mkalderon at marvell.com>
Signed-off-by: Ariel Elior <aelior at marvell.com>
Signed-off-by: Shai Malin <smalin at marvell.com>
Reviewed-by: Hannes Reinecke <hare at suse.de>
---
 drivers/nvme/hw/qedn/qedn.h      |   8 ++
 drivers/nvme/hw/qedn/qedn_main.c |   1 +
 drivers/nvme/hw/qedn/qedn_task.c | 147 +++++++++++++++++++++++++++++--
 3 files changed, 148 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/hw/qedn/qedn.h b/drivers/nvme/hw/qedn/qedn.h
index 5539791fc49b..b275d6b9bc59 100644
--- a/drivers/nvme/hw/qedn/qedn.h
+++ b/drivers/nvme/hw/qedn/qedn.h
@@ -96,6 +96,9 @@
 #define QEDN_TASK_CLEANUP_TMO 3000 /* 3 sec */
 #define QEDN_DRAIN_TMO 1000 /* 1 sec */
 
+#define QEDN_MAX_OUTSTAND_ASYNC 32
+#define QEDN_INVALID_CCCID (-1)
+
 enum qedn_state {
 	QEDN_STATE_CORE_PROBED = 0,
 	QEDN_STATE_CORE_OPEN,
@@ -178,6 +181,7 @@ struct qedn_ctx {
 
 enum qedn_task_flags {
 	QEDN_TASK_IS_ICREQ,
+	QEDN_TASK_ASYNC,
 	QEDN_TASK_USED_BY_FW,
 	QEDN_TASK_WAIT_FOR_CLEANUP,
 };
@@ -346,6 +350,10 @@ struct qedn_conn_ctx {
 	struct nvme_tcp_icresp_pdu icresp;
 	struct qedn_icreq_padding *icreq_pad;
 
+	DECLARE_BITMAP(async_cccid_idx_map, QEDN_MAX_OUTSTAND_ASYNC);
+	/* Spinlock for fetching pseudo CCCID for async request */
+	spinlock_t async_cccid_bitmap_lock;
+
 	/* "dummy" socket */
 	struct socket *sock;
 };
diff --git a/drivers/nvme/hw/qedn/qedn_main.c b/drivers/nvme/hw/qedn/qedn_main.c
index d18cc3624764..2aaf1af8f9a7 100644
--- a/drivers/nvme/hw/qedn/qedn_main.c
+++ b/drivers/nvme/hw/qedn/qedn_main.c
@@ -343,6 +343,7 @@ static int qedn_create_queue(struct nvme_tcp_ofld_queue *queue, int qid,
 	atomic_set(&conn_ctx->destroy_conn_indicator, 0);
 
 	spin_lock_init(&conn_ctx->conn_state_lock);
+	spin_lock_init(&conn_ctx->async_cccid_bitmap_lock);
 
 	conn_ctx->qid = qid;
 
diff --git a/drivers/nvme/hw/qedn/qedn_task.c b/drivers/nvme/hw/qedn/qedn_task.c
index 6664cc587f06..351afbc750b2 100644
--- a/drivers/nvme/hw/qedn/qedn_task.c
+++ b/drivers/nvme/hw/qedn/qedn_task.c
@@ -266,9 +266,45 @@ void qedn_common_clear_fw_sgl(struct storage_sgl_task_params *sgl_task_params)
 }
 
 inline void qedn_host_reset_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx,
-					     u16 cccid)
+					     u16 cccid, bool async)
 {
 	conn_ctx->host_cccid_itid[cccid].itid = cpu_to_le16(QEDN_INVALID_ITID);
+	if (unlikely(async))
+		clear_bit(cccid - NVME_AQ_DEPTH,
+			  conn_ctx->async_cccid_idx_map);
+}
+
+static int qedn_get_free_idx(struct qedn_conn_ctx *conn_ctx, unsigned int size)
+{
+	int idx;
+
+	spin_lock(&conn_ctx->async_cccid_bitmap_lock);
+	idx = find_first_zero_bit(conn_ctx->async_cccid_idx_map, size);
+	if (unlikely(idx >= size)) {
+		idx = -1;
+		spin_unlock(&conn_ctx->async_cccid_bitmap_lock);
+		goto err_idx;
+	}
+	set_bit(idx, conn_ctx->async_cccid_idx_map);
+	spin_unlock(&conn_ctx->async_cccid_bitmap_lock);
+
+err_idx:
+
+	return idx;
+}
+
+int qedn_get_free_async_cccid(struct qedn_conn_ctx *conn_ctx)
+{
+	int async_cccid;
+
+	async_cccid =
+		qedn_get_free_idx(conn_ctx, QEDN_MAX_OUTSTAND_ASYNC);
+	if (unlikely(async_cccid == QEDN_INVALID_CCCID))
+		pr_err("No available CCCID for Async.\n");
+	else
+		async_cccid += NVME_AQ_DEPTH;
+
+	return async_cccid;
 }
 
 inline void qedn_host_set_cccid_itid_entry(struct qedn_conn_ctx *conn_ctx,
@@ -361,10 +397,12 @@ void qedn_return_task_to_pool(struct qedn_conn_ctx *conn_ctx,
 	struct qedn_fp_queue *fp_q = conn_ctx->fp_q;
 	struct qedn_io_resources *io_resrc;
 	unsigned long lock_flags;
+	bool async;
 
 	io_resrc = &fp_q->host_resrc;
 
 	spin_lock_irqsave(&qedn_task->lock, lock_flags);
+	async = test_bit(QEDN_TASK_ASYNC, &(qedn_task)->flags);
 	qedn_task->valid = 0;
 	qedn_task->flags = 0;
 	qedn_clear_sgl(conn_ctx->qedn, qedn_task);
@@ -372,7 +410,7 @@ void qedn_return_task_to_pool(struct qedn_conn_ctx *conn_ctx,
 
 	spin_lock(&conn_ctx->task_list_lock);
 	list_del(&qedn_task->entry);
-	qedn_host_reset_cccid_itid_entry(conn_ctx, qedn_task->cccid);
+	qedn_host_reset_cccid_itid_entry(conn_ctx, qedn_task->cccid, async);
 	spin_unlock(&conn_ctx->task_list_lock);
 
 	atomic_dec(&conn_ctx->num_active_tasks);
@@ -419,6 +457,60 @@ qedn_get_free_task_from_pool(struct qedn_conn_ctx *conn_ctx, u16 cccid)
 	return qedn_task;
 }
 
+void qedn_send_async_event_cmd(struct qedn_task_ctx *qedn_task,
+			       struct qedn_conn_ctx *conn_ctx)
+{
+	struct nvme_tcp_ofld_req *async_req = qedn_task->req;
+	struct nvme_command *nvme_cmd = &async_req->nvme_cmd;
+	struct storage_sgl_task_params *sgl_task_params;
+	struct nvmetcp_task_params task_params;
+	struct nvme_tcp_cmd_pdu cmd_hdr;
+	struct nvmetcp_wqe *chain_sqe;
+	struct nvmetcp_wqe local_sqe;
+
+	set_bit(QEDN_TASK_ASYNC, &qedn_task->flags);
+	nvme_cmd->common.command_id = qedn_task->cccid;
+	qedn_task->task_size = 0;
+
+	/* Initialize sgl params */
+	sgl_task_params = &qedn_task->sgl_task_params;
+	sgl_task_params->total_buffer_size = 0;
+	sgl_task_params->num_sges = 0;
+	sgl_task_params->small_mid_sge = false;
+
+	task_params.opq.lo = cpu_to_le32(((u64)(qedn_task)) & 0xffffffff);
+	task_params.opq.hi = cpu_to_le32(((u64)(qedn_task)) >> 32);
+
+	/* Initialize task params */
+	task_params.context = qedn_task->fw_task_ctx;
+	task_params.sqe = &local_sqe;
+	task_params.tx_io_size = 0;
+	task_params.rx_io_size = 0;
+	task_params.conn_icid = (u16)conn_ctx->conn_handle;
+	task_params.itid = qedn_task->itid;
+	task_params.cq_rss_number = conn_ctx->default_cq;
+	task_params.send_write_incapsule = 0;
+
+	/* Internal impl. - async is treated like zero len read */
+	cmd_hdr.hdr.type = nvme_tcp_cmd;
+	cmd_hdr.hdr.flags = 0;
+	cmd_hdr.hdr.hlen = sizeof(cmd_hdr);
+	cmd_hdr.hdr.pdo = 0x0;
+	cmd_hdr.hdr.plen = cpu_to_le32(cmd_hdr.hdr.hlen);
+
+	qed_ops->init_read_io(&task_params, &cmd_hdr, nvme_cmd,
+			      &qedn_task->sgl_task_params);
+
+	set_bit(QEDN_TASK_USED_BY_FW, &qedn_task->flags);
+	atomic_inc(&conn_ctx->num_active_fw_tasks);
+
+	spin_lock(&conn_ctx->ep.doorbell_lock);
+	chain_sqe = qed_chain_produce(&conn_ctx->ep.fw_sq_chain);
+	memcpy(chain_sqe, &local_sqe, sizeof(local_sqe));
+	qedn_ring_doorbell(conn_ctx);
+	spin_unlock(&conn_ctx->ep.doorbell_lock);
+}
+
 int qedn_send_read_cmd(struct qedn_task_ctx *qedn_task,
 		       struct qedn_conn_ctx *conn_ctx)
 {
@@ -533,6 +625,21 @@ int qedn_send_write_cmd(struct qedn_task_ctx *qedn_task,
 	return 0;
 }
 
+static void qedn_return_error_req(struct nvme_tcp_ofld_req *req)
+{
+	__le16 status = cpu_to_le16(NVME_SC_HOST_PATH_ERROR << 1);
+	union nvme_result res = {};
+
+	if (!req)
+		return;
+
+	/* Call request done to compelete the request */
+	if (req->done)
+		req->done(req, &res, status);
+	else
+		pr_err("request done not set !!!\n");
+}
+
 int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 		       struct nvme_tcp_ofld_req *req)
 {
@@ -543,9 +650,17 @@ int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 
 	rq = blk_mq_rq_from_pdu(req);
 
-	/* Placeholder - async */
+	if (unlikely(req->async)) {
+		cccid = qedn_get_free_async_cccid(qedn_conn);
+		if (cccid == QEDN_INVALID_CCCID) {
+			qedn_return_error_req(req);
+
+			return BLK_STS_NOTSUPP;
+		}
+	} else {
+		cccid = rq->tag;
+	}
 
-	cccid = rq->tag;
 	qedn_task = qedn_get_free_task_from_pool(qedn_conn, cccid);
 	if (unlikely(!qedn_task)) {
 		pr_err("Not able to allocate task context resource\n");
@@ -556,7 +671,11 @@ int qedn_queue_request(struct qedn_conn_ctx *qedn_conn,
 	req->private_data = qedn_task;
 	qedn_task->req = req;
 
-	/* Placeholder - handle (req->async) */
+	if (unlikely(req->async)) {
+		qedn_send_async_event_cmd(qedn_task, qedn_conn);
+
+		return BLK_STS_TRANSPORT;
+	}
 
 	/* Check if there are physical segments in request to determine the
 	 * task size. The logic of nvme_tcp_set_sg_null() will be implemented
@@ -629,16 +748,28 @@ static inline int qedn_comp_valid_task(struct qedn_task_ctx *qedn_task,
 int qedn_process_nvme_cqe(struct qedn_task_ctx *qedn_task,
 			  struct nvme_completion *cqe)
 {
+	struct qedn_conn_ctx *conn_ctx = qedn_task->qedn_conn;
+	struct nvme_tcp_ofld_req *req;
 	int rc = 0;
+	bool async;
+
+	async = test_bit(QEDN_TASK_ASYNC, &(qedn_task)->flags);
 
 	/* CQE arrives swapped
 	 * Swapping requirement will be removed in future FW versions
 	 */
 	qedn_swap_bytes((u32 *)cqe, (sizeof(*cqe) / sizeof(u32)));
 
-	/* Placeholder - async */
-
-	rc = qedn_comp_valid_task(qedn_task, &cqe->result, cqe->status);
+	if (unlikely(async)) {
+		qedn_return_task_to_pool(conn_ctx, qedn_task);
+		req = qedn_task->req;
+		if (req->done)
+			req->done(req, &cqe->result, cqe->status);
+		else
+			pr_err("request done not set for async request !!!\n");
+	} else {
+		rc = qedn_comp_valid_task(qedn_task, &cqe->result, cqe->status);
+	}
 
 	return rc;
 }
-- 
2.24.1



From ming.lei at redhat.com  Tue Jun 29 07:17:15 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Tue, 29 Jun 2021 22:17:15 +0800
Subject: [PATCH 1/2] blk-mq: not deactivate hctx if the device doesn't
 use managed irq
In-Reply-To: <1a14a397-6244-928e-5aaa-85c2ccbe0e40@suse.de>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <20210629074951.1981284-2-ming.lei@redhat.com>
 <1a14a397-6244-928e-5aaa-85c2ccbe0e40@suse.de>
Message-ID: <YNsrayg+pbVO+J7I@T590>

On Tue, Jun 29, 2021 at 02:39:14PM +0200, Hannes Reinecke wrote:
> On 6/29/21 9:49 AM, Ming Lei wrote:
> > hctx is deactivated when all CPU in hctx->cpumask become offline by
> > draining all requests originated from this hctx and moving new
> > allocation to active hctx. This way is for avoiding inflight IO when
> > the managed irq is shutdown.
> > 
> > Some drivers(nvme fc, rdma, tcp, loop) doesn't use managed irq, so
> > they needn't to deactivate hctx. Also, they are the only user of
> > blk_mq_alloc_request_hctx() which is used for connecting io queue.
> > And their requirement is that the connect request can be submitted
> > via one specified hctx on which all CPU in its hctx->cpumask may have
> > become offline.
> > 
> 
> How can you submit a connect request for a hctx on which all CPUs are
> offline? That hctx will be unusable as it'll never be able to receive
> interrupts ...

I believe BLK_MQ_F_NOT_USE_MANAGED_IRQ is self-explanatory. And the
interrupt(non-managed) of this hctx will be migrated to online CPUs,
see migrate_one_irq().

For managed irq, we have to prevent new allocation if all CPUs of this
hctx is offline because genirq will shutdown the interrupt.

> 
> > Address the requirement for nvme fc/rdma/loop, so the reported kernel
> > panic on the following line in blk_mq_alloc_request_hctx() can be fixed.
> > 
> > 	data.ctx = __blk_mq_get_ctx(q, cpu)
> > 
> > Cc: Sagi Grimberg <sagi at grimberg.me>
> > Cc: Daniel Wagner <dwagner at suse.de>
> > Cc: Wen Xiong <wenxiong at us.ibm.com>
> > Cc: John Garry <john.garry at huawei.com>
> > Signed-off-by: Ming Lei <ming.lei at redhat.com>
> > ---
> >   block/blk-mq.c         | 6 +++++-
> >   include/linux/blk-mq.h | 1 +
> >   2 files changed, 6 insertions(+), 1 deletion(-)
> > 
> > diff --git a/block/blk-mq.c b/block/blk-mq.c
> > index df5dc3b756f5..74632f50d969 100644
> > --- a/block/blk-mq.c
> > +++ b/block/blk-mq.c
> > @@ -494,7 +494,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
> >   	data.hctx = q->queue_hw_ctx[hctx_idx];
> >   	if (!blk_mq_hw_queue_mapped(data.hctx))
> >   		goto out_queue_exit;
> > -	cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
> > +	cpu = cpumask_first(data.hctx->cpumask);
> >   	data.ctx = __blk_mq_get_ctx(q, cpu);
> 
> I don't get it.
> Doesn't this allow us to allocate a request on a dead cpu, ie the very thing
> we try to prevent?

It is fine to allocate & dispatch one request to the hctx when all CPU on
its cpumask are offline if this hctx's interrupt isn't managed.


Thanks,
Ming



From john.garry at huawei.com  Tue Jun 29 08:49:56 2021
From: john.garry at huawei.com (John Garry)
Date: Tue, 29 Jun 2021 16:49:56 +0100
Subject: [PATCH 1/2] blk-mq: not deactivate hctx if the device doesn't use
 managed irq
In-Reply-To: <20210629074951.1981284-2-ming.lei@redhat.com>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <20210629074951.1981284-2-ming.lei@redhat.com>
Message-ID: <c31aa259-d3a8-8c70-efce-b7af02bfd609@huawei.com>

On 29/06/2021 08:49, Ming Lei wrote:
> hctx is deactivated when all CPU in hctx->cpumask become offline by
> draining all requests originated from this hctx and moving new
> allocation to active hctx. This way is for avoiding inflight IO when
> the managed irq is shutdown.
> 
> Some drivers(nvme fc, rdma, tcp, loop) doesn't use managed irq, so
> they needn't to deactivate hctx. Also, they are the only user of
> blk_mq_alloc_request_hctx() which is used for connecting io queue.
> And their requirement is that the connect request can be submitted
> via one specified hctx on which all CPU in its hctx->cpumask may have
> become offline.
> 
> Address the requirement for nvme fc/rdma/loop, so the reported kernel
> panic on the following line in blk_mq_alloc_request_hctx() can be fixed.
> 
> 	data.ctx = __blk_mq_get_ctx(q, cpu)
> 
> Cc: Sagi Grimberg <sagi at grimberg.me>
> Cc: Daniel Wagner <dwagner at suse.de>
> Cc: Wen Xiong <wenxiong at us.ibm.com>
> Cc: John Garry <john.garry at huawei.com>
> Signed-off-by: Ming Lei <ming.lei at redhat.com>
> ---
>   block/blk-mq.c         | 6 +++++-
>   include/linux/blk-mq.h | 1 +
>   2 files changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index df5dc3b756f5..74632f50d969 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -494,7 +494,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
>   	data.hctx = q->queue_hw_ctx[hctx_idx];
>   	if (!blk_mq_hw_queue_mapped(data.hctx))
>   		goto out_queue_exit;
> -	cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
> +	cpu = cpumask_first(data.hctx->cpumask);
>   	data.ctx = __blk_mq_get_ctx(q, cpu);
>   
>   	if (!q->elevator)
> @@ -2570,6 +2570,10 @@ static int blk_mq_hctx_notify_offline(unsigned int cpu, struct hlist_node *node)
>   	    !blk_mq_last_cpu_in_hctx(cpu, hctx))
>   		return 0;
>   
> +	/* Controller doesn't use managed IRQ, no need to deactivate hctx */
> +	if (hctx->flags & BLK_MQ_F_NOT_USE_MANAGED_IRQ)
> +		return 0;
Is there anything to be gained in registering the CPU hotplug handler 
for the hctx in this case at all?

> +
>   	/*
>   	 * Prevent new request from being allocated on the current hctx.
>   	 *
> diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
> index 21140132a30d..600c5dd1a069 100644
> --- a/include/linux/blk-mq.h
> +++ b/include/linux/blk-mq.h
> @@ -403,6 +403,7 @@ enum {
>   	 */
>   	BLK_MQ_F_STACKING	= 1 << 2,
>   	BLK_MQ_F_TAG_HCTX_SHARED = 1 << 3,
> +	BLK_MQ_F_NOT_USE_MANAGED_IRQ = 1 << 4,

Many block drivers don't use managed interrupts - to be proper, why not 
set this everywhere (which doesn't use managed interrupts)? I know why, 
but it's odd.

As an alternative, if the default queue mapping was used (in 
blk_mq_map_queues()), then that's the same thing as 
BLK_MQ_F_NOT_USE_MANAGED_IRQ in reality, right? If so, could we 
alternatively check for that somehow?

Thanks,
John

>   	BLK_MQ_F_BLOCKING	= 1 << 5,
>   	BLK_MQ_F_NO_SCHED	= 1 << 6,
>   	BLK_MQ_F_ALLOC_POLICY_START_BIT = 8,
> 



From Damien.LeMoal at wdc.com  Tue Jun 29 16:30:35 2021
From: Damien.LeMoal at wdc.com (Damien Le Moal)
Date: Tue, 29 Jun 2021 23:30:35 +0000
Subject: [PATCH 1/2] blk-mq: not deactivate hctx if the device doesn't use
 managed irq
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <20210629074951.1981284-2-ming.lei@redhat.com>
Message-ID: <DM6PR04MB70814885C27FF97CFC02A456E7029@DM6PR04MB7081.namprd04.prod.outlook.com>

On 2021/06/29 16:50, Ming Lei wrote:
> hctx is deactivated when all CPU in hctx->cpumask become offline by
> draining all requests originated from this hctx and moving new
> allocation to active hctx. This way is for avoiding inflight IO when
> the managed irq is shutdown.
> 
> Some drivers(nvme fc, rdma, tcp, loop) doesn't use managed irq, so
> they needn't to deactivate hctx. Also, they are the only user of
> blk_mq_alloc_request_hctx() which is used for connecting io queue.
> And their requirement is that the connect request can be submitted
> via one specified hctx on which all CPU in its hctx->cpumask may have
> become offline.
> 
> Address the requirement for nvme fc/rdma/loop, so the reported kernel
> panic on the following line in blk_mq_alloc_request_hctx() can be fixed.
> 
> 	data.ctx = __blk_mq_get_ctx(q, cpu)
> 
> Cc: Sagi Grimberg <sagi at grimberg.me>
> Cc: Daniel Wagner <dwagner at suse.de>
> Cc: Wen Xiong <wenxiong at us.ibm.com>
> Cc: John Garry <john.garry at huawei.com>
> Signed-off-by: Ming Lei <ming.lei at redhat.com>
> ---
>  block/blk-mq.c         | 6 +++++-
>  include/linux/blk-mq.h | 1 +
>  2 files changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index df5dc3b756f5..74632f50d969 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -494,7 +494,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
>  	data.hctx = q->queue_hw_ctx[hctx_idx];
>  	if (!blk_mq_hw_queue_mapped(data.hctx))
>  		goto out_queue_exit;
> -	cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
> +	cpu = cpumask_first(data.hctx->cpumask);
>  	data.ctx = __blk_mq_get_ctx(q, cpu);
>  
>  	if (!q->elevator)
> @@ -2570,6 +2570,10 @@ static int blk_mq_hctx_notify_offline(unsigned int cpu, struct hlist_node *node)
>  	    !blk_mq_last_cpu_in_hctx(cpu, hctx))
>  		return 0;
>  
> +	/* Controller doesn't use managed IRQ, no need to deactivate hctx */
> +	if (hctx->flags & BLK_MQ_F_NOT_USE_MANAGED_IRQ)
> +		return 0;
> +
>  	/*
>  	 * Prevent new request from being allocated on the current hctx.
>  	 *
> diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
> index 21140132a30d..600c5dd1a069 100644
> --- a/include/linux/blk-mq.h
> +++ b/include/linux/blk-mq.h
> @@ -403,6 +403,7 @@ enum {
>  	 */
>  	BLK_MQ_F_STACKING	= 1 << 2,
>  	BLK_MQ_F_TAG_HCTX_SHARED = 1 << 3,
> +	BLK_MQ_F_NOT_USE_MANAGED_IRQ = 1 << 4,

My 2 cents: BLK_MQ_F_NO_MANAGED_IRQ may be a better/shorter name.

>  	BLK_MQ_F_BLOCKING	= 1 << 5,
>  	BLK_MQ_F_NO_SCHED	= 1 << 6,
>  	BLK_MQ_F_ALLOC_POLICY_START_BIT = 8,
> 


-- 
Damien Le Moal
Western Digital Research


From ming.lei at redhat.com  Tue Jun 29 17:32:46 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Wed, 30 Jun 2021 08:32:46 +0800
Subject: [PATCH 1/2] blk-mq: not deactivate hctx if the device doesn't
 use managed irq
In-Reply-To: <c31aa259-d3a8-8c70-efce-b7af02bfd609@huawei.com>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <20210629074951.1981284-2-ming.lei@redhat.com>
 <c31aa259-d3a8-8c70-efce-b7af02bfd609@huawei.com>
Message-ID: <YNu7rs7j5/KtQjAi@T590>

On Tue, Jun 29, 2021 at 04:49:56PM +0100, John Garry wrote:
> On 29/06/2021 08:49, Ming Lei wrote:
> > hctx is deactivated when all CPU in hctx->cpumask become offline by
> > draining all requests originated from this hctx and moving new
> > allocation to active hctx. This way is for avoiding inflight IO when
> > the managed irq is shutdown.
> > 
> > Some drivers(nvme fc, rdma, tcp, loop) doesn't use managed irq, so
> > they needn't to deactivate hctx. Also, they are the only user of
> > blk_mq_alloc_request_hctx() which is used for connecting io queue.
> > And their requirement is that the connect request can be submitted
> > via one specified hctx on which all CPU in its hctx->cpumask may have
> > become offline.
> > 
> > Address the requirement for nvme fc/rdma/loop, so the reported kernel
> > panic on the following line in blk_mq_alloc_request_hctx() can be fixed.
> > 
> > 	data.ctx = __blk_mq_get_ctx(q, cpu)
> > 
> > Cc: Sagi Grimberg <sagi at grimberg.me>
> > Cc: Daniel Wagner <dwagner at suse.de>
> > Cc: Wen Xiong <wenxiong at us.ibm.com>
> > Cc: John Garry <john.garry at huawei.com>
> > Signed-off-by: Ming Lei <ming.lei at redhat.com>
> > ---
> >   block/blk-mq.c         | 6 +++++-
> >   include/linux/blk-mq.h | 1 +
> >   2 files changed, 6 insertions(+), 1 deletion(-)
> > 
> > diff --git a/block/blk-mq.c b/block/blk-mq.c
> > index df5dc3b756f5..74632f50d969 100644
> > --- a/block/blk-mq.c
> > +++ b/block/blk-mq.c
> > @@ -494,7 +494,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
> >   	data.hctx = q->queue_hw_ctx[hctx_idx];
> >   	if (!blk_mq_hw_queue_mapped(data.hctx))
> >   		goto out_queue_exit;
> > -	cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
> > +	cpu = cpumask_first(data.hctx->cpumask);
> >   	data.ctx = __blk_mq_get_ctx(q, cpu);
> >   	if (!q->elevator)
> > @@ -2570,6 +2570,10 @@ static int blk_mq_hctx_notify_offline(unsigned int cpu, struct hlist_node *node)
> >   	    !blk_mq_last_cpu_in_hctx(cpu, hctx))
> >   		return 0;
> > +	/* Controller doesn't use managed IRQ, no need to deactivate hctx */
> > +	if (hctx->flags & BLK_MQ_F_NOT_USE_MANAGED_IRQ)
> > +		return 0;
> Is there anything to be gained in registering the CPU hotplug handler for
> the hctx in this case at all?

Indeed, we may replace BLK_MQ_F_STACKING with BLK_MQ_F_NOT_USE_MANAGED_IRQ
(BLK_MQ_F_NO_MANAGED_IRQ) too, and the latter is one general solution.

> 
> > +
> >   	/*
> >   	 * Prevent new request from being allocated on the current hctx.
> >   	 *
> > diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
> > index 21140132a30d..600c5dd1a069 100644
> > --- a/include/linux/blk-mq.h
> > +++ b/include/linux/blk-mq.h
> > @@ -403,6 +403,7 @@ enum {
> >   	 */
> >   	BLK_MQ_F_STACKING	= 1 << 2,
> >   	BLK_MQ_F_TAG_HCTX_SHARED = 1 << 3,
> > +	BLK_MQ_F_NOT_USE_MANAGED_IRQ = 1 << 4,
> 
> Many block drivers don't use managed interrupts - to be proper, why not set
> this everywhere (which doesn't use managed interrupts)? I know why, but it's
> odd.

It is just one small optimization in slow path for other drivers, not sure these
drivers are interested in such change. It only serves as fix for callers of
blk_mq_alloc_request_hctx().

Anyway, we can document the situation.

> 
> As an alternative, if the default queue mapping was used (in
> blk_mq_map_queues()), then that's the same thing as
> BLK_MQ_F_NOT_USE_MANAGED_IRQ in reality, right? If so, could we
> alternatively check for that somehow?

This way can't work, for example of NVMe PCI, managed irq is used for
the default/write queues, but poll queues uses blk_mq_map_queues().

Also it can't cover all cases, such as MVMe RDMA.

Using managed irq or not is thing of driver's choice, and not sure if it
is good for block layer to provide such abstract. I'd suggest to fix the
issue still by passing one flag, given we needn't it everywhere so far.

 
Thanks,
Ming



From hare at suse.de  Wed Jun 30 01:15:46 2021
From: hare at suse.de (Hannes Reinecke)
Date: Wed, 30 Jun 2021 10:15:46 +0200
Subject: [PATCH 2/2] nvme: pass BLK_MQ_F_NOT_USE_MANAGED_IRQ for
 fc/rdma/tcp/loop
In-Reply-To: <20210629074951.1981284-3-ming.lei@redhat.com>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <20210629074951.1981284-3-ming.lei@redhat.com>
Message-ID: <46fdb02d-8d71-d465-072d-5e8d6ec3601f@suse.de>

On 6/29/21 9:49 AM, Ming Lei wrote:
> All the 4 host drivers don't use managed irq for completing request, so
> it is correct to pass the flag to blk-mq.
> 
> Secondly with this flag, blk-mq will help us dispatch connect request
> allocated via blk_mq_alloc_request_hctx() to driver even though all
> CPU in the specified hctx's cpumask are offline.
> 
How is this supposed to work?
To my understanding, only cpus in the hctx cpumask are eligible to send 
I/O. So if all of these CPUs are offline, who exactly is submitting I/O?
More to the point, which CPU will be submitting the connect request?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From hare at suse.de  Wed Jun 30 01:18:37 2021
From: hare at suse.de (Hannes Reinecke)
Date: Wed, 30 Jun 2021 10:18:37 +0200
Subject: [PATCH 0/2] blk-mq: fix blk_mq_alloc_request_hctx
In-Reply-To: <20210629074951.1981284-1-ming.lei@redhat.com>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
Message-ID: <5f304121-38ce-034b-2d17-93d136c77fe6@suse.de>

On 6/29/21 9:49 AM, Ming Lei wrote:
> Hi,
> 
> blk_mq_alloc_request_hctx() is used by NVMe fc/rdma/tcp/loop to connect
> io queue. Also the sw ctx is chosen as the 1st online cpu in hctx->cpumask.
> However, all cpus in hctx->cpumask may be offline.
> 
> This usage model isn't well supported by blk-mq which supposes allocator is
> always done on one online CPU in hctx->cpumask. This assumption is
> related with managed irq, which also requires blk-mq to drain inflight
> request in this hctx when the last cpu in hctx->cpumask is going to
> offline.
> 
> However, NVMe fc/rdma/tcp/loop don't use managed irq, so we should allow
> them to ask for request allocation when the specified hctx is inactive
> (all cpus in hctx->cpumask are offline).
> 
> Fix blk_mq_alloc_request_hctx() by adding/passing flag of
> BLK_MQ_F_NOT_USE_MANAGED_IRQ.
> 
> 
> Ming Lei (2):
>    blk-mq: not deactivate hctx if the device doesn't use managed irq
>    nvme: pass BLK_MQ_F_NOT_USE_MANAGED_IRQ for fc/rdma/tcp/loop
> 
>   block/blk-mq.c             | 6 +++++-
>   drivers/nvme/host/fc.c     | 3 ++-
>   drivers/nvme/host/rdma.c   | 3 ++-
>   drivers/nvme/host/tcp.c    | 3 ++-
>   drivers/nvme/target/loop.c | 3 ++-
>   include/linux/blk-mq.h     | 1 +
>   6 files changed, 14 insertions(+), 5 deletions(-)
> 
> Cc: Sagi Grimberg <sagi at grimberg.me>
> Cc: Daniel Wagner <dwagner at suse. thede>
> Cc: Wen Xiong <wenxiong at us.ibm.com>
> Cc: John Garry <john.garry at huawei.com>
> 
> 
I have my misgivings about this patchset.
To my understanding, only CPUs present in the hctx cpumask are eligible 
to submit I/O to that hctx.
Consequently if all cpus in that mask are offline, where is the point of 
even transmitting a 'connect' request?
Shouldn't we rather modify the tagset to only refer to the current 
online CPUs _only_, thereby never submit a connect request for hctx with 
only offline CPUs?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From ming.lei at redhat.com  Wed Jun 30 01:42:43 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Wed, 30 Jun 2021 16:42:43 +0800
Subject: [PATCH 0/2] blk-mq: fix blk_mq_alloc_request_hctx
In-Reply-To: <5f304121-38ce-034b-2d17-93d136c77fe6@suse.de>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <5f304121-38ce-034b-2d17-93d136c77fe6@suse.de>
Message-ID: <YNwug8n7qGL5uXfo@T590>

On Wed, Jun 30, 2021 at 10:18:37AM +0200, Hannes Reinecke wrote:
> On 6/29/21 9:49 AM, Ming Lei wrote:
> > Hi,
> > 
> > blk_mq_alloc_request_hctx() is used by NVMe fc/rdma/tcp/loop to connect
> > io queue. Also the sw ctx is chosen as the 1st online cpu in hctx->cpumask.
> > However, all cpus in hctx->cpumask may be offline.
> > 
> > This usage model isn't well supported by blk-mq which supposes allocator is
> > always done on one online CPU in hctx->cpumask. This assumption is
> > related with managed irq, which also requires blk-mq to drain inflight
> > request in this hctx when the last cpu in hctx->cpumask is going to
> > offline.
> > 
> > However, NVMe fc/rdma/tcp/loop don't use managed irq, so we should allow
> > them to ask for request allocation when the specified hctx is inactive
> > (all cpus in hctx->cpumask are offline).
> > 
> > Fix blk_mq_alloc_request_hctx() by adding/passing flag of
> > BLK_MQ_F_NOT_USE_MANAGED_IRQ.
> > 
> > 
> > Ming Lei (2):
> >    blk-mq: not deactivate hctx if the device doesn't use managed irq
> >    nvme: pass BLK_MQ_F_NOT_USE_MANAGED_IRQ for fc/rdma/tcp/loop
> > 
> >   block/blk-mq.c             | 6 +++++-
> >   drivers/nvme/host/fc.c     | 3 ++-
> >   drivers/nvme/host/rdma.c   | 3 ++-
> >   drivers/nvme/host/tcp.c    | 3 ++-
> >   drivers/nvme/target/loop.c | 3 ++-
> >   include/linux/blk-mq.h     | 1 +
> >   6 files changed, 14 insertions(+), 5 deletions(-)
> > 
> > Cc: Sagi Grimberg <sagi at grimberg.me>
> > Cc: Daniel Wagner <dwagner at suse. thede>
> > Cc: Wen Xiong <wenxiong at us.ibm.com>
> > Cc: John Garry <john.garry at huawei.com>
> > 
> > 
> I have my misgivings about this patchset.
> To my understanding, only CPUs present in the hctx cpumask are eligible to
> submit I/O to that hctx.

It is just true for managed irq, and should be CPUs online.

However, no such constraint for non managed irq, since irq may migrate to
other online CPUs if all CPUs in irq's current affinity become offline.

> Consequently if all cpus in that mask are offline, where is the point of
> even transmitting a 'connect' request?

nvmef requires to submit the connect request via one specified hctx
which index has to be same with the io queue's index.

Almost all nvmef drivers fail to setup controller in case of
connect io queue error.

Also CPU can become offline & online, especially it is done in
lots of sanity test.

So we should allow to allocate the connect request successful, and
submit it to drivers given it is allowed in this way for non-managed
irq.

> Shouldn't we rather modify the tagset to only refer to the current online
> CPUs _only_, thereby never submit a connect request for hctx with only
> offline CPUs?

Then you may setup very less io queues, and performance may suffer even
though lots of CPUs become online later.


Thanks,
Ming



From ming.lei at redhat.com  Wed Jun 30 01:47:44 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Wed, 30 Jun 2021 16:47:44 +0800
Subject: [PATCH 2/2] nvme: pass BLK_MQ_F_NOT_USE_MANAGED_IRQ for
 fc/rdma/tcp/loop
In-Reply-To: <46fdb02d-8d71-d465-072d-5e8d6ec3601f@suse.de>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <20210629074951.1981284-3-ming.lei@redhat.com>
 <46fdb02d-8d71-d465-072d-5e8d6ec3601f@suse.de>
Message-ID: <YNwvsNcTBu6yGW1o@T590>

On Wed, Jun 30, 2021 at 10:15:46AM +0200, Hannes Reinecke wrote:
> On 6/29/21 9:49 AM, Ming Lei wrote:
> > All the 4 host drivers don't use managed irq for completing request, so
> > it is correct to pass the flag to blk-mq.
> > 
> > Secondly with this flag, blk-mq will help us dispatch connect request
> > allocated via blk_mq_alloc_request_hctx() to driver even though all
> > CPU in the specified hctx's cpumask are offline.
> > 
> How is this supposed to work?
> To my understanding, only cpus in the hctx cpumask are eligible to send I/O.
> So if all of these CPUs are offline, who exactly is submitting I/O?
> More to the point, which CPU will be submitting the connect request?

Please see __blk_mq_delay_run_hw_queue(), which will pass WORK_CPU_UNBOUND 
to kblockd_mod_delayed_work_on() for this situation.


Thanks,
Ming



From john.garry at huawei.com  Wed Jun 30 02:25:41 2021
From: john.garry at huawei.com (John Garry)
Date: Wed, 30 Jun 2021 10:25:41 +0100
Subject: [PATCH 1/2] blk-mq: not deactivate hctx if the device doesn't use
 managed irq
In-Reply-To: <YNu7rs7j5/KtQjAi@T590>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <20210629074951.1981284-2-ming.lei@redhat.com>
 <c31aa259-d3a8-8c70-efce-b7af02bfd609@huawei.com> <YNu7rs7j5/KtQjAi@T590>
Message-ID: <adf34d6d-b209-78d3-8df9-4e3398e34612@huawei.com>

On 30/06/2021 01:32, Ming Lei wrote:
>> Many block drivers don't use managed interrupts - to be proper, why not set
>> this everywhere (which doesn't use managed interrupts)? I know why, but it's
>> odd.
> It is just one small optimization in slow path for other drivers, not sure these
> drivers are interested in such change. It only serves as fix for callers of
> blk_mq_alloc_request_hctx().
> 
> Anyway, we can document the situation.
> 
>> As an alternative, if the default queue mapping was used (in
>> blk_mq_map_queues()), then that's the same thing as
>> BLK_MQ_F_NOT_USE_MANAGED_IRQ in reality, right? If so, could we
>> alternatively check for that somehow?
> This way can't work, for example of NVMe PCI, managed irq is used for
> the default/write queues, but poll queues uses blk_mq_map_queues().
> 
> Also it can't cover all cases, such as MVMe RDMA.
> 
> Using managed irq or not is thing of driver's choice, and not sure if it
> is good for block layer to provide such abstract. I'd suggest to fix the
> issue still by passing one flag, given we needn't it everywhere so far.

Sure, but I am just trying to suggest a more automatic way of passing 
this info. Like, for example, if we use blk_mq_pci_map_queues() on the 
qmap, then you prob want blk-mq managed interrupt support (for that 
qmap), i.e. CPU hotplug handlers.

Thanks,
John


From hare at suse.de  Wed Jun 30 02:43:41 2021
From: hare at suse.de (Hannes Reinecke)
Date: Wed, 30 Jun 2021 11:43:41 +0200
Subject: [PATCH 0/2] blk-mq: fix blk_mq_alloc_request_hctx
In-Reply-To: <YNwug8n7qGL5uXfo@T590>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <5f304121-38ce-034b-2d17-93d136c77fe6@suse.de> <YNwug8n7qGL5uXfo@T590>
Message-ID: <c1de513a-5477-9d1d-0ddc-24e9166cc717@suse.de>

On 6/30/21 10:42 AM, Ming Lei wrote:
> On Wed, Jun 30, 2021 at 10:18:37AM +0200, Hannes Reinecke wrote:
>> On 6/29/21 9:49 AM, Ming Lei wrote:
>>> Hi,
>>>
>>> blk_mq_alloc_request_hctx() is used by NVMe fc/rdma/tcp/loop to connect
>>> io queue. Also the sw ctx is chosen as the 1st online cpu in hctx->cpumask.
>>> However, all cpus in hctx->cpumask may be offline.
>>>
>>> This usage model isn't well supported by blk-mq which supposes allocator is
>>> always done on one online CPU in hctx->cpumask. This assumption is
>>> related with managed irq, which also requires blk-mq to drain inflight
>>> request in this hctx when the last cpu in hctx->cpumask is going to
>>> offline.
>>>
>>> However, NVMe fc/rdma/tcp/loop don't use managed irq, so we should allow
>>> them to ask for request allocation when the specified hctx is inactive
>>> (all cpus in hctx->cpumask are offline).
>>>
>>> Fix blk_mq_alloc_request_hctx() by adding/passing flag of
>>> BLK_MQ_F_NOT_USE_MANAGED_IRQ.
>>>
>>>
>>> Ming Lei (2):
>>>     blk-mq: not deactivate hctx if the device doesn't use managed irq
>>>     nvme: pass BLK_MQ_F_NOT_USE_MANAGED_IRQ for fc/rdma/tcp/loop
>>>
>>>    block/blk-mq.c             | 6 +++++-
>>>    drivers/nvme/host/fc.c     | 3 ++-
>>>    drivers/nvme/host/rdma.c   | 3 ++-
>>>    drivers/nvme/host/tcp.c    | 3 ++-
>>>    drivers/nvme/target/loop.c | 3 ++-
>>>    include/linux/blk-mq.h     | 1 +
>>>    6 files changed, 14 insertions(+), 5 deletions(-)
>>>
>>> Cc: Sagi Grimberg <sagi at grimberg.me>
>>> Cc: Daniel Wagner <dwagner at suse. thede>
>>> Cc: Wen Xiong <wenxiong at us.ibm.com>
>>> Cc: John Garry <john.garry at huawei.com>
>>>
>>>
>> I have my misgivings about this patchset.
>> To my understanding, only CPUs present in the hctx cpumask are eligible to
>> submit I/O to that hctx.
> 
> It is just true for managed irq, and should be CPUs online.
> 
> However, no such constraint for non managed irq, since irq may migrate to
> other online CPUs if all CPUs in irq's current affinity become offline.
> 

But there shouldn't be any I/O pending during CPU offline (cf 
blk_mq_hctx_notify_offline()), so no interrupts should be triggered, 
either, no?

>> Consequently if all cpus in that mask are offline, where is the point of
>> even transmitting a 'connect' request?
> 
> nvmef requires to submit the connect request via one specified hctx
> which index has to be same with the io queue's index.
> 
> Almost all nvmef drivers fail to setup controller in case of
> connect io queue error.
> 

And I would prefer to fix that, namely allowing blk-mq to run on a 
sparse set of io queues.
The remaining io queues can be connected once the first cpu in the hctx 
cpumask is onlined; we already have blk_mq_hctx_notify_online(), which 
could easily be expanded to connect to relevant I/O queue...

> Also CPU can become offline & online, especially it is done in
> lots of sanity test.
> 

True, but then again all I/O on the hctx should be quiesced during cpu 
offline.

> So we should allow to allocate the connect request successful, and
> submit it to drivers given it is allowed in this way for non-managed
> irq.
> 

I'd rather not do this, as the 'connect' command runs on the 'normal' 
I/O tagset, and hence runs into the risk of being issues against 
non-existing CPUs.

>> Shouldn't we rather modify the tagset to only refer to the current online
>> CPUs _only_, thereby never submit a connect request for hctx with only
>> offline CPUs?
> 
> Then you may setup very less io queues, and performance may suffer even
> though lots of CPUs become online later.
> ;
Only if we stay with the reduced number of I/O queues. Which is not what 
I'm proposing; I'd rather prefer to connect and disconnect queues from 
the cpu hotplug handler. For starters we could even trigger a reset once 
the first cpu within a hctx is onlined.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From ming.lei at redhat.com  Wed Jun 30 02:53:17 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Wed, 30 Jun 2021 17:53:17 +0800
Subject: [PATCH 0/2] blk-mq: fix blk_mq_alloc_request_hctx
In-Reply-To: <c1de513a-5477-9d1d-0ddc-24e9166cc717@suse.de>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <5f304121-38ce-034b-2d17-93d136c77fe6@suse.de>
 <YNwug8n7qGL5uXfo@T590>
 <c1de513a-5477-9d1d-0ddc-24e9166cc717@suse.de>
Message-ID: <YNw/DcxIIMeg/2VK@T590>

On Wed, Jun 30, 2021 at 11:43:41AM +0200, Hannes Reinecke wrote:
> On 6/30/21 10:42 AM, Ming Lei wrote:
> > On Wed, Jun 30, 2021 at 10:18:37AM +0200, Hannes Reinecke wrote:
> > > On 6/29/21 9:49 AM, Ming Lei wrote:
> > > > Hi,
> > > > 
> > > > blk_mq_alloc_request_hctx() is used by NVMe fc/rdma/tcp/loop to connect
> > > > io queue. Also the sw ctx is chosen as the 1st online cpu in hctx->cpumask.
> > > > However, all cpus in hctx->cpumask may be offline.
> > > > 
> > > > This usage model isn't well supported by blk-mq which supposes allocator is
> > > > always done on one online CPU in hctx->cpumask. This assumption is
> > > > related with managed irq, which also requires blk-mq to drain inflight
> > > > request in this hctx when the last cpu in hctx->cpumask is going to
> > > > offline.
> > > > 
> > > > However, NVMe fc/rdma/tcp/loop don't use managed irq, so we should allow
> > > > them to ask for request allocation when the specified hctx is inactive
> > > > (all cpus in hctx->cpumask are offline).
> > > > 
> > > > Fix blk_mq_alloc_request_hctx() by adding/passing flag of
> > > > BLK_MQ_F_NOT_USE_MANAGED_IRQ.
> > > > 
> > > > 
> > > > Ming Lei (2):
> > > >     blk-mq: not deactivate hctx if the device doesn't use managed irq
> > > >     nvme: pass BLK_MQ_F_NOT_USE_MANAGED_IRQ for fc/rdma/tcp/loop
> > > > 
> > > >    block/blk-mq.c             | 6 +++++-
> > > >    drivers/nvme/host/fc.c     | 3 ++-
> > > >    drivers/nvme/host/rdma.c   | 3 ++-
> > > >    drivers/nvme/host/tcp.c    | 3 ++-
> > > >    drivers/nvme/target/loop.c | 3 ++-
> > > >    include/linux/blk-mq.h     | 1 +
> > > >    6 files changed, 14 insertions(+), 5 deletions(-)
> > > > 
> > > > Cc: Sagi Grimberg <sagi at grimberg.me>
> > > > Cc: Daniel Wagner <dwagner at suse. thede>
> > > > Cc: Wen Xiong <wenxiong at us.ibm.com>
> > > > Cc: John Garry <john.garry at huawei.com>
> > > > 
> > > > 
> > > I have my misgivings about this patchset.
> > > To my understanding, only CPUs present in the hctx cpumask are eligible to
> > > submit I/O to that hctx.
> > 
> > It is just true for managed irq, and should be CPUs online.
> > 
> > However, no such constraint for non managed irq, since irq may migrate to
> > other online CPUs if all CPUs in irq's current affinity become offline.
> > 
> 
> But there shouldn't be any I/O pending during CPU offline (cf
> blk_mq_hctx_notify_offline()), so no interrupts should be triggered, either,
> no?
> 
> > > Consequently if all cpus in that mask are offline, where is the point of
> > > even transmitting a 'connect' request?
> > 
> > nvmef requires to submit the connect request via one specified hctx
> > which index has to be same with the io queue's index.
> > 
> > Almost all nvmef drivers fail to setup controller in case of
> > connect io queue error.
> > 
> 
> And I would prefer to fix that, namely allowing blk-mq to run on a sparse
> set of io queues.
> The remaining io queues can be connected once the first cpu in the hctx
> cpumask is onlined; we already have blk_mq_hctx_notify_online(), which could
> easily be expanded to connect to relevant I/O queue...

Then you need a big patches for doing that.

> 
> > Also CPU can become offline & online, especially it is done in
> > lots of sanity test.
> > 
> 
> True, but then again all I/O on the hctx should be quiesced during cpu
> offline.

Again that is only necessary for managed irq.

> 
> > So we should allow to allocate the connect request successful, and
> > submit it to drivers given it is allowed in this way for non-managed
> > irq.
> > 
> 
> I'd rather not do this, as the 'connect' command runs on the 'normal' I/O
> tagset, and hence runs into the risk of being issues against non-existing
> CPUs.

Can you explain what the risk is?

> 
> > > Shouldn't we rather modify the tagset to only refer to the current online
> > > CPUs _only_, thereby never submit a connect request for hctx with only
> > > offline CPUs?
> > 
> > Then you may setup very less io queues, and performance may suffer even
> > though lots of CPUs become online later.
> > ;
> Only if we stay with the reduced number of I/O queues. Which is not what I'm
> proposing; I'd rather prefer to connect and disconnect queues from the cpu
> hotplug handler. For starters we could even trigger a reset once the first
> cpu within a hctx is onlined.

Yeah, that need one big/complicated patchset, but not see any advantages
over this simple approach.


Thanks,
Ming



From hare at suse.de  Wed Jun 30 03:03:42 2021
From: hare at suse.de (Hannes Reinecke)
Date: Wed, 30 Jun 2021 12:03:42 +0200
Subject: [PATCH] blk-mq: don't allocate requests when all cpus in a hctx are
 offline
Message-ID: <20210630100342.1100-1-hare@suse.de>

When all CPUs in a hctx are offline in blk_mq_alloc_request_hctx() we should
not try to allocate the request, as we'll fail later on in blk_mq_get_tag() anyway.

Signed-off-by: Hannes Reinecke <hare at suse.de>
---
 block/blk-mq.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index f11d4018ce2e..1622d0cabe3a 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -468,13 +468,16 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
 	data.hctx = q->queue_hw_ctx[hctx_idx];
 	if (!blk_mq_hw_queue_mapped(data.hctx))
 		goto out_queue_exit;
+	ret = -EWOULDBLOCK;
+	/* all cpus in the hctx are offline, don't try to allocate a tag */
 	cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
+	if (unlikely(cpu >= nr_cpu_ids))
+		goto out_queue_exit;
 	data.ctx = __blk_mq_get_ctx(q, cpu);
 
 	if (!q->elevator)
 		blk_mq_tag_busy(data.hctx);
 
-	ret = -EWOULDBLOCK;
 	tag = blk_mq_get_tag(&data);
 	if (tag == BLK_MQ_NO_TAG)
 		goto out_queue_exit;
-- 
2.29.2



From ming.lei at redhat.com  Wed Jun 30 03:13:49 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Wed, 30 Jun 2021 18:13:49 +0800
Subject: [PATCH] blk-mq: don't allocate requests when all cpus in a hctx
 are offline
In-Reply-To: <20210630100342.1100-1-hare@suse.de>
References: <20210630100342.1100-1-hare@suse.de>
Message-ID: <YNxD3ekRJrZK8FKm@T590>

On Wed, Jun 30, 2021 at 12:03:42PM +0200, Hannes Reinecke wrote:
> When all CPUs in a hctx are offline in blk_mq_alloc_request_hctx() we should
> not try to allocate the request, as we'll fail later on in blk_mq_get_tag() anyway.
> 
> Signed-off-by: Hannes Reinecke <hare at suse.de>
> ---

Please see my yesterday's reply why this way isn't good:

https://lore.kernel.org/lkml/YNrwnWfsxf8cJcoe at T590/
https://lore.kernel.org/lkml/YNrhXFgv%2FgEWbhbl at T590/


Thanks,
Ming



From Vinay.Rao at dell.com  Wed Jun 30 07:00:33 2021
From: Vinay.Rao at dell.com (Rao, Vinay)
Date: Wed, 30 Jun 2021 14:00:33 +0000
Subject: [PATCH 1/1] nvme-fabrics: Generate uevent on namespace AENs
In-Reply-To: <097e56b6-2b13-e00c-ef84-509fa0e887ca@suse.de>
References: <BL0PR13MB429142FD8B9FCC57C2FE451D9C409@BL0PR13MB4291.namprd13.prod.outlook.com>
 <20210429063056.GA2482@lst.de>
 <SJ0PR19MB454499BF2899EF772B1EA11CF25F9@SJ0PR19MB4544.namprd19.prod.outlook.com>
 <BYAPR04MB4965BD2AE65469C96CBF1354865F9@BYAPR04MB4965.namprd04.prod.outlook.com>
 <BYAPR19MB29351987D7ABBA3F28791D978F5E9@BYAPR19MB2935.namprd19.prod.outlook.com>
 <20210430162104.GA907317@dhcp-10-100-145-180.wdc.com>
 <097e56b6-2b13-e00c-ef84-509fa0e887ca@suse.de>
Message-ID: <BYAPR19MB2935A9169D186DD3C05979668F019@BYAPR19MB2935.namprd19.prod.outlook.com>



-----Original Message-----
From: Hannes Reinecke <hare at suse.de> 
Sent: Thursday, June 24, 2021 4:19 PM
To: Keith Busch; Rao, Vinay
Cc: Chaitanya Kulkarni; Belanger, Martin; Hayes, Stuart; Tarikere, Madhu; Christoph Hellwig; Martin Belanger; linux-nvme at lists.infradead.org; axboe at fb.com; sagi at grimberg.me
Subject: Re: [PATCH 1/1] nvme-fabrics: Generate uevent on namespace AENs


[EXTERNAL EMAIL] 

On 4/30/21 6:21 PM, Keith Busch wrote:
> On Fri, Apr 30, 2021 at 05:14:55AM +0000, Rao, Vinay wrote:
>> + Madhu
>>
>> Hi Chaitanya,
>>
>> Some of the AEN's like ANA state change and Name Space attribute change events are of interest outside the NVME driver. 
>>
>> It would be good to consider these events to be propagated up to user 
>> space. I am not yet convinced on how propagating this above to user space is problematic.
> 
> The namespace attribute change event pairs with the Changed Namespace 
> List log, and reading that log will change the result for a subsequent 
> reader. User space racing with the kernel on a log access when there 
> are read side effects creates non-deterministic behavior, and that is 
> problematic.
> 
It would, if we were relying on the Changed Namespace List log.
But in the end we're just reading it to clear the AEN flag; the contents are completely immaterial to us as we're doing a full namespace scan afterwards anyway.
So I don't think that that'll be a problem for us.

What will be problematic, though, is if we were to allow userspace to register an AEN; as soon as the controller allows more than one pending AERs there nothing preventing it from doing so.
If he does our internal states will be messed up as we're not receiving all events. I'd rather error out any attempt here, but then we can't do a full passthrough, and I guess things like SPDK might become unhappy.
> Wanted to check if there is a timeline by when we can expect the AEN's to be propagated to user space. 


Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare at suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 N?rnberg HRB 36809 (AG N?rnberg), GF: Felix Imend?rffer

From cachen at purestorage.com  Wed Jun 30 11:01:37 2021
From: cachen at purestorage.com (Casey Chen)
Date: Wed, 30 Jun 2021 11:01:37 -0700
Subject: [PATCH 0/2] rr: nvme-pci: fix races and UAF
In-Reply-To: <20210624173125.20376-1-cachen@purestorage.com>
References: <20210624173125.20376-1-cachen@purestorage.com>
Message-ID: <e6a90107-b5b8-ad53-1e79-7691c37fd011@purestorage.com>

This review request has been there nearly a week. Except one comment 
from Chaitanya that adjust the alignment of each line in commit message 
of the second patch, I don't see any other comments. Do you guys have 
more comments ? Let me know!

On 6/24/21 10:31 AM, Casey Chen wrote:
> Found two bugs while power-cycling PCIe NVMe drive for hours:
> - Races in nvme_setup_io_queues()
> - UAF introduced by nvme_dev_remove_admin(), which is found
> after fixing the races. Without fixing the races, the system
> just crashes and cannot survive to reproduce the UAF.
>
> The proposed fixes have been tested for several days for correctness.
>
> 0. Code baseline
>
> Tag nvme-5.14-2021-06-08 of repo http://git.infradead.org/nvme.git
>
> 1. Testing method
>
> while :; power off one drive;
>      sleep $((RANDOM%3)).$((RANDOM%10));
>      power on the same drive;
>      sleep $((RANDOM%3)).$((RANDOM%10));
> done
>
> 2. Sample crash trace due to races in nvme_setup_io_queues()
> (task ID shown after the timestamps)
>
> [11668.533431][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Card present
> ...
> [11668.681298][T251231] nvme nvme12: pci function 0000:8c:00.0
> [11668.681354][T26714] nvme 0000:8c:00.0: enabling device (0100 -> 0102)
> [11669.046119][   C31] pcieport 0000:87:08.0: pciehp: pending interrupts 0x0108 from Slot Status
> [11669.046142][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Link Down
> [11669.046146][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Card not present
> [11669.046149][  T716] pcieport 0000:87:08.0: pciehp: pciehp_unconfigure_device: domain:bus:dev = 0000:8c:00
> [11669.077428][  T716] ------------[ cut here ]------------
> [11669.077431][  T716] kernel BUG at drivers/pci/msi.c:348!
> [11669.077555][  T716] invalid opcode: 0000 [#1] SMP KASAN
> [11669.077658][  T716] CPU: 31 PID: 716 Comm: irq/127-pciehp Not tainted 5.13.0-rc3+
> [11669.078022][  T716] RIP: 0010:free_msi_irqs+0x28a/0x2d0
> ...
> [11669.093982][  T716] Call Trace:
> [11669.096850][  T716]  pci_free_irq_vectors+0xe/0x20
> [11669.099695][  T716]  nvme_dev_disable+0x140/0x760 [nvme]
> [11669.102503][  T716]  ? _raw_spin_lock_irqsave+0x9c/0x100
> [11669.105271][  T716]  ? trace_hardirqs_on+0x2c/0xe0
> [11669.107994][  T716]  nvme_remove+0x191/0x1e0 [nvme]
> [11669.110689][  T716]  pci_device_remove+0x6b/0x110
> [11669.113316][  T716]  device_release_driver_internal+0x14f/0x280
> [11669.115939][  T716]  pci_stop_bus_device+0xcb/0x100
> [11669.118515][  T716]  pci_stop_and_remove_bus_device+0xe/0x20
> [11669.121079][  T716]  pciehp_unconfigure_device+0xfa/0x200
> [11669.123597][  T716]  ? pciehp_configure_device+0x1c0/0x1c0
> [11669.126049][  T716]  ? trace_hardirqs_on+0x2c/0xe0
> [11669.128444][  T716]  pciehp_disable_slot+0xc4/0x1a0
> [11669.130771][  T716]  ? pciehp_runtime_suspend+0x40/0x40
> [11669.133054][  T716]  ? __mutex_lock_slowpath+0x10/0x10
> [11669.135289][  T716]  ? trace_hardirqs_on+0x2c/0xe0
> [11669.137462][  T716]  pciehp_handle_presence_or_link_change+0x15c/0x4f0
> [11669.139632][  T716]  ? down_read+0x11f/0x1a0
> [11669.141731][  T716]  ? pciehp_handle_disable_request+0x80/0x80
> [11669.143817][  T716]  ? rwsem_down_read_slowpath+0x600/0x600
> [11669.145851][  T716]  ? __radix_tree_lookup+0xb2/0x130
> [11669.147842][  T716]  pciehp_ist+0x19d/0x1a0
> [11669.149790][  T716]  ? pciehp_set_indicators+0xe0/0xe0
> [11669.151704][  T716]  ? irq_finalize_oneshot.part.46+0x1d0/0x1d0
> [11669.153588][  T716]  irq_thread_fn+0x3f/0xa0
> [11669.155407][  T716]  irq_thread+0x195/0x290
> [11669.157147][  T716]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
> [11669.158883][  T716]  ? _raw_read_lock_irq+0x50/0x50
> [11669.160611][  T716]  ? _raw_read_lock_irq+0x50/0x50
> [11669.162320][  T716]  ? irq_forced_thread_fn+0xf0/0xf0
> [11669.164032][  T716]  ? trace_hardirqs_on+0x2c/0xe0
> [11669.165731][  T716]  ? irq_thread_check_affinity.part.49+0xe0/0xe0
> [11669.167461][  T716]  kthread+0x1c8/0x1f0
> [11669.169173][  T716]  ? kthread_parkme+0x40/0x40
> [11669.170883][  T716]  ret_from_fork+0x22/0x30
>
> 3. KASAN report for the UAF introduced by nvme_dev_remove_admin()
> (task ID is shown after the timestamp)
>
> [18319.015748][T246989] nvme nvme13: pci function 0000:8c:00.0
> [18319.015795][T215541] nvme 0000:8c:00.0: enabling device (0100 -> 0102)
> [18319.369086][   C31] pcieport 0000:87:08.0: pciehp: pending interrupts 0x0108 from Slot Status
> [18319.369107][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Link Down
> [18319.369111][  T716] pcieport 0000:87:08.0: pciehp: Slot(402): Card not present
> [18319.369116][  T716] pcieport 0000:87:08.0: pciehp: pciehp_unconfigure_device: domain:bus:dev = 0000:8c:00
> [18320.452045][T215541] nvme nvme13: 88/0/0 default/read/poll queues
> [18320.469475][T215541] nvme nvme13: failed to mark controller live state
> [18320.469483][T215541] nvme nvme13: Removing after probe failure status: -19
> [18320.551295][T215541] ==================================================================
> [18320.551299][T215541] BUG: KASAN: use-after-free in __blk_mq_all_tag_iter+0x9c/0x3f0
> [18320.551311][T215541] Read of size 4 at addr ffff888897904d04 by task kworker/u178:2/215541
> [18320.551315][T215541]
> [18320.551318][T215541] CPU: 86 PID: 215541 Comm: kworker/u178:2 Not tainted 5.13.0-rc3+
> [18320.551327][T215541] Workqueue: nvme-reset-wq nvme_reset_work [nvme]
> [18320.551339][T215541] Call Trace:
> [18320.551343][T215541]  dump_stack+0xa4/0xdb
> [18320.551354][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
> [18320.551359][T215541]  print_address_description.constprop.10+0x3a/0x60
> [18320.551366][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
> [18320.551372][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
> [18320.551377][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
> [18320.551382][T215541]  kasan_report.cold.15+0x7c/0xd8
> [18320.551390][T215541]  ? __blk_mq_all_tag_iter+0x9c/0x3f0
> [18320.551395][T215541]  __blk_mq_all_tag_iter+0x9c/0x3f0
> [18320.551401][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
> [18320.551407][T215541]  ? bt_iter+0xf0/0xf0
> [18320.551412][T215541]  ? __blk_mq_all_tag_iter+0x2c9/0x3f0
> [18320.551417][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
> [18320.551422][T215541]  ? bt_iter+0xf0/0xf0
> [18320.551427][T215541]  ? dev_printk_emit+0x95/0xbb
> [18320.551436][T215541]  blk_mq_tagset_busy_iter+0x75/0xa0
> [18320.551441][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
> [18320.551446][T215541]  ? blk_mq_update_nr_requests+0x2a0/0x2a0
> [18320.551451][T215541]  blk_mq_tagset_wait_completed_request+0x86/0xc0
> [18320.551457][T215541]  ? blk_mq_tagset_busy_iter+0xa0/0xa0
> [18320.551463][T215541]  ? blk_mq_tagset_busy_iter+0x80/0xa0
> [18320.551469][T215541]  ? trace_event_raw_event_nvme_setup_cmd+0x2d0/0x2d0 [nvme_core]
> [18320.551493][T215541]  nvme_dev_disable+0x4f6/0x760 [nvme]
> [18320.551502][T215541]  ? trace_hardirqs_on+0x2c/0xe0
> [18320.551510][T215541]  nvme_reset_work+0x226/0x2060 [nvme]
> [18320.551520][T215541]  ? nvme_remove+0x1e0/0x1e0 [nvme]
> [18320.551528][T215541]  ? __update_load_avg_cfs_rq+0x1d8/0x550
> [18320.551537][T215541]  ? down_read+0x11f/0x1a0
> [18320.551545][T215541]  ? newidle_balance+0x444/0x690
> [18320.551552][T215541]  ? update_load_avg+0x626/0xbe0
> [18320.551557][T215541]  ? update_cfs_group+0x1e/0x150
> [18320.551562][T215541]  ? load_balance+0x11d0/0x11d0
> [18320.551567][T215541]  ? dequeue_entity+0x150/0x730
> [18320.551573][T215541]  ? nvme_irq_check+0x60/0x60 [nvme]
> [18320.551581][T215541]  ? finish_task_switch+0x101/0x3d0
> [18320.551588][T215541]  ? read_word_at_a_time+0xe/0x20
> [18320.551594][T215541]  ? strscpy+0xc1/0x1d0
> [18320.551598][T215541]  process_one_work+0x4b9/0x7b0
> [18320.551604][T215541]  worker_thread+0x72/0x710
> [18320.551610][T215541]  ? process_one_work+0x7b0/0x7b0
> [18320.551614][T215541]  kthread+0x1c8/0x1f0
> [18320.551618][T215541]  ? kthread_parkme+0x40/0x40
> [18320.551622][T215541]  ret_from_fork+0x22/0x30
> [18320.551630][T215541]
> [18320.551632][T215541] Allocated by task 215541:
> [18320.551635][T215541]  kasan_save_stack+0x19/0x40
> [18320.551639][T215541]  __kasan_kmalloc+0x7f/0xa0
> [18320.551642][T215541]  kmem_cache_alloc_node_trace+0x187/0x2b0
> [18320.551648][T215541]  blk_mq_init_tags+0x47/0x100
> [18320.551651][T215541]  blk_mq_alloc_rq_map+0x44/0xf0
> [18320.551656][T215541]  __blk_mq_alloc_map_and_request+0x7f/0x140
> [18320.551661][T215541]  blk_mq_alloc_tag_set+0x25e/0x510
> [18320.551666][T215541]  nvme_reset_work+0x14f9/0x2060 [nvme]
> [18320.551674][T215541]  process_one_work+0x4b9/0x7b0
> [18320.551678][T215541]  worker_thread+0x72/0x710
> [18320.551682][T215541]  kthread+0x1c8/0x1f0
> [18320.551685][T215541]  ret_from_fork+0x22/0x30
> [18320.551689][T215541]
> [18320.551690][T215541] Freed by task 716:
> [18320.551693][T215541]  kasan_save_stack+0x19/0x40
> [18320.551696][T215541]  kasan_set_track+0x1c/0x30
> [18320.551699][T215541]  kasan_set_free_info+0x20/0x30
> [18320.551704][T215541]  __kasan_slab_free+0xec/0x130
> [18320.551707][T215541]  kfree+0xa8/0x460
> [18320.551712][T215541]  blk_mq_free_map_and_requests+0x8d/0xc0
> [18320.551717][T215541]  blk_mq_free_tag_set+0x30/0xf0
> [18320.551721][T215541]  nvme_remove+0x199/0x1e0 [nvme]
> [18320.551729][T215541]  pci_device_remove+0x6b/0x110
> [18320.551735][T215541]  device_release_driver_internal+0x14f/0x280
> [18320.551744][T215541]  pci_stop_bus_device+0xcb/0x100
> [18320.551750][T215541]  pci_stop_and_remove_bus_device+0xe/0x20
> [18320.551754][T215541]  pciehp_unconfigure_device+0xfa/0x200
> [18320.551761][T215541]  pciehp_disable_slot+0xc4/0x1a0
> [18320.551765][T215541]  pciehp_handle_presence_or_link_change+0x15c/0x4f0
> [18320.551770][T215541]  pciehp_ist+0x19d/0x1a0
> [18320.551774][T215541]  irq_thread_fn+0x3f/0xa0
> [18320.551780][T215541]  irq_thread+0x195/0x290
> [18320.551783][T215541]  kthread+0x1c8/0x1f0
> [18320.551786][T215541]  ret_from_fork+0x22/0x30
> [18320.551791][T215541]
> [18320.551792][T215541] The buggy address belongs to the object at ffff888897904d00
> [18320.551792][T215541]  which belongs to the cache kmalloc-192 of size 192
> [18320.551795][T215541] The buggy address is located 4 bytes inside of
> [18320.551795][T215541]  192-byte region [ffff888897904d00, ffff888897904dc0)
> [18320.551800][T215541] The buggy address belongs to the page:
> [18320.551802][T215541] page:000000002f3df664 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x897904
> [18320.551807][T215541] head:000000002f3df664 order:1 compound_mapcount:0
> [18320.551810][T215541] flags: 0x200000000010200(slab|head|node=0|zone=2)
> [18320.551819][T215541] raw: 0200000000010200 dead000000000100 dead000000000122 ffff88810004ca00
> [18320.551824][T215541] raw: 0000000000000000 0000000000200020 00000001ffffffff 0000000000000000
> [18320.551826][T215541] page dumped because: kasan: bad access detected
> [18320.551828][T215541]
> [18320.551829][T215541] Memory state around the buggy address:
> [18320.551832][T215541]  ffff888897904c00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [18320.551835][T215541]  ffff888897904c80: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
> [18320.551838][T215541] >ffff888897904d00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [18320.551841][T215541]                    ^
> [18320.551843][T215541]  ffff888897904d80: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
> [18320.551846][T215541]  ffff888897904e00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [18320.551848][T215541] ==================================================================
> [18320.551850][T215541] Disabling lock debugging due to kernel taint
>
> Casey Chen (2):
>    nvme-pci: Fix multiple races in nvme_setup_io_queues()
>    nvme-pci: Fix UAF introduced by nvme_dev_remove_admin()
>
>   drivers/nvme/host/pci.c | 60 ++++++++++++++++++++++++++++++++++++-----
>   1 file changed, 53 insertions(+), 7 deletions(-)
>


From sagi at grimberg.me  Wed Jun 30 11:50:06 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 30 Jun 2021 11:50:06 -0700
Subject: [PATCH] nvmet: allow setting ctrl to ready without iosqes/iocqes
 set
In-Reply-To: <20210502153106.18970-1-ran.anner@dell.com>
References: <20210502153106.18970-1-ran.anner@dell.com>
Message-ID: <a1767b33-561d-3c9f-cdd0-b338f2967f6e@grimberg.me>

> From: Ran Anner <Ran.Anner at dell.com>
> 
> According to the spec we should fail creation of IO queues
> if iosqes or iocqes properties are not set

Would be good to add the specific section in the spec.

  but does not mention
> preventing from changing ctrl status to ready.
> We have seen host implementation which sets the property to 0x1
> and waits endlessly until ctrl status changes to ready.
> 
> Signed-off-by: Ran Anner <Ran.Anner at dell.com>
> ---
>   drivers/nvme/target/core.c        | 39 +------------------------------
>   drivers/nvme/target/fabrics-cmd.c |  9 +++++++
>   drivers/nvme/target/nvmet.h       | 34 +++++++++++++++++++++++++++
>   3 files changed, 44 insertions(+), 38 deletions(-)
> 
> diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
> index adbede9ab7f3..559dc7354adb 100644
> --- a/drivers/nvme/target/core.c
> +++ b/drivers/nvme/target/core.c
> @@ -1078,48 +1078,11 @@ void nvmet_req_free_sgls(struct nvmet_req *req)
>   }
>   EXPORT_SYMBOL_GPL(nvmet_req_free_sgls);
>   
> -static inline bool nvmet_cc_en(u32 cc)
> -{
> -	return (cc >> NVME_CC_EN_SHIFT) & 0x1;
> -}
> -
> -static inline u8 nvmet_cc_css(u32 cc)
> -{
> -	return (cc >> NVME_CC_CSS_SHIFT) & 0x7;
> -}
> -
> -static inline u8 nvmet_cc_mps(u32 cc)
> -{
> -	return (cc >> NVME_CC_MPS_SHIFT) & 0xf;
> -}
> -
> -static inline u8 nvmet_cc_ams(u32 cc)
> -{
> -	return (cc >> NVME_CC_AMS_SHIFT) & 0x7;
> -}
> -
> -static inline u8 nvmet_cc_shn(u32 cc)
> -{
> -	return (cc >> NVME_CC_SHN_SHIFT) & 0x3;
> -}
> -
> -static inline u8 nvmet_cc_iosqes(u32 cc)
> -{
> -	return (cc >> NVME_CC_IOSQES_SHIFT) & 0xf;
> -}
> -
> -static inline u8 nvmet_cc_iocqes(u32 cc)
> -{
> -	return (cc >> NVME_CC_IOCQES_SHIFT) & 0xf;
> -}
> -
>   static void nvmet_start_ctrl(struct nvmet_ctrl *ctrl)
>   {
>   	lockdep_assert_held(&ctrl->lock);
>   
> -	if (nvmet_cc_iosqes(ctrl->cc) != NVME_NVM_IOSQES ||
> -	    nvmet_cc_iocqes(ctrl->cc) != NVME_NVM_IOCQES ||
> -	    nvmet_cc_mps(ctrl->cc) != 0 ||
> +	if (nvmet_cc_mps(ctrl->cc) != 0 ||
>   	    nvmet_cc_ams(ctrl->cc) != 0 ||
>   	    nvmet_cc_css(ctrl->cc) != 0) {
>   		ctrl->csts = NVME_CSTS_CFS;
> diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c
> index 1420a8e3e0b1..421ab564ed5c 100644
> --- a/drivers/nvme/target/fabrics-cmd.c
> +++ b/drivers/nvme/target/fabrics-cmd.c
> @@ -259,6 +259,15 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
>   		goto out_ctrl_put;
>   	}
>   
> +	if (nvmet_cc_iosqes(ctrl->cc) != NVME_NVM_IOSQES ||
> +			nvmet_cc_iocqes(ctrl->cc) != NVME_NVM_IOCQES) {

Can you indent to the start of the parenthesis?


From sagi at grimberg.me  Wed Jun 30 11:58:47 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 30 Jun 2021 11:58:47 -0700
Subject: [PATCH 1/2] blk-mq: not deactivate hctx if the device doesn't use
 managed irq
In-Reply-To: <DM6PR04MB70814885C27FF97CFC02A456E7029@DM6PR04MB7081.namprd04.prod.outlook.com>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <20210629074951.1981284-2-ming.lei@redhat.com>
 <DM6PR04MB70814885C27FF97CFC02A456E7029@DM6PR04MB7081.namprd04.prod.outlook.com>
Message-ID: <0f8be11b-c6de-1c2b-323b-1e059c2ac4f8@grimberg.me>


>> diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
>> index 21140132a30d..600c5dd1a069 100644
>> --- a/include/linux/blk-mq.h
>> +++ b/include/linux/blk-mq.h
>> @@ -403,6 +403,7 @@ enum {
>>   	 */
>>   	BLK_MQ_F_STACKING	= 1 << 2,
>>   	BLK_MQ_F_TAG_HCTX_SHARED = 1 << 3,
>> +	BLK_MQ_F_NOT_USE_MANAGED_IRQ = 1 << 4,
> 
> My 2 cents: BLK_MQ_F_NO_MANAGED_IRQ may be a better/shorter name.

Maybe BLK_MQ_F_UNMANAGED_IRQ?


From sagi at grimberg.me  Wed Jun 30 11:59:33 2021
From: sagi at grimberg.me (Sagi Grimberg)
Date: Wed, 30 Jun 2021 11:59:33 -0700
Subject: [PATCH 0/2] blk-mq: fix blk_mq_alloc_request_hctx
In-Reply-To: <YNw/DcxIIMeg/2VK@T590>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <5f304121-38ce-034b-2d17-93d136c77fe6@suse.de> <YNwug8n7qGL5uXfo@T590>
 <c1de513a-5477-9d1d-0ddc-24e9166cc717@suse.de> <YNw/DcxIIMeg/2VK@T590>
Message-ID: <e106f9c4-35c3-b2da-cdd8-3c4dff8234d6@grimberg.me>


>>>> Shouldn't we rather modify the tagset to only refer to the current online
>>>> CPUs _only_, thereby never submit a connect request for hctx with only
>>>> offline CPUs?
>>>
>>> Then you may setup very less io queues, and performance may suffer even
>>> though lots of CPUs become online later.
>>> ;
>> Only if we stay with the reduced number of I/O queues. Which is not what I'm
>> proposing; I'd rather prefer to connect and disconnect queues from the cpu
>> hotplug handler. For starters we could even trigger a reset once the first
>> cpu within a hctx is onlined.
> 
> Yeah, that need one big/complicated patchset, but not see any advantages
> over this simple approach.

I tend to agree with Ming here.


From hare at suse.de  Wed Jun 30 12:46:35 2021
From: hare at suse.de (Hannes Reinecke)
Date: Wed, 30 Jun 2021 21:46:35 +0200
Subject: [PATCH 0/2] blk-mq: fix blk_mq_alloc_request_hctx
In-Reply-To: <e106f9c4-35c3-b2da-cdd8-3c4dff8234d6@grimberg.me>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <5f304121-38ce-034b-2d17-93d136c77fe6@suse.de> <YNwug8n7qGL5uXfo@T590>
 <c1de513a-5477-9d1d-0ddc-24e9166cc717@suse.de> <YNw/DcxIIMeg/2VK@T590>
 <e106f9c4-35c3-b2da-cdd8-3c4dff8234d6@grimberg.me>
Message-ID: <89081624-fedd-aa94-1ba2-9a137708a1f1@suse.de>

On 6/30/21 8:59 PM, Sagi Grimberg wrote:
> 
>>>>> Shouldn't we rather modify the tagset to only refer to the current 
>>>>> online
>>>>> CPUs _only_, thereby never submit a connect request for hctx with only
>>>>> offline CPUs?
>>>>
>>>> Then you may setup very less io queues, and performance may suffer even
>>>> though lots of CPUs become online later.
>>>> ;
>>> Only if we stay with the reduced number of I/O queues. Which is not 
>>> what I'm
>>> proposing; I'd rather prefer to connect and disconnect queues from 
>>> the cpu
>>> hotplug handler. For starters we could even trigger a reset once the 
>>> first
>>> cpu within a hctx is onlined.
>>
>> Yeah, that need one big/complicated patchset, but not see any advantages
>> over this simple approach.
> 
> I tend to agree with Ming here.

Actually, Daniel and me came to a slightly different idea: use cpu 
hotplug notifier.
Thing is, blk-mq already has cpu hotplug notifier, which should ensure 
that no I/O is pending during cpu hotplug.
If we now add a nvme cpu hotplug notifier which essentially kicks off a 
reset once all cpu in a hctx are offline the reset logic will rearrange 
the queues to match the current cpu layout.
And when the cpus are getting onlined we'll do another reset.

Daniel is currently preparing a patch; let's see how it goes.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare at suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 N?rnberg
HRB 36809 (AG N?rnberg), Gesch?ftsf?hrer: Felix Imend?rffer


From Damien.LeMoal at wdc.com  Wed Jun 30 14:57:05 2021
From: Damien.LeMoal at wdc.com (Damien Le Moal)
Date: Wed, 30 Jun 2021 21:57:05 +0000
Subject: [PATCH 1/2] blk-mq: not deactivate hctx if the device doesn't use
 managed irq
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <20210629074951.1981284-2-ming.lei@redhat.com>
 <DM6PR04MB70814885C27FF97CFC02A456E7029@DM6PR04MB7081.namprd04.prod.outlook.com>
 <0f8be11b-c6de-1c2b-323b-1e059c2ac4f8@grimberg.me>
Message-ID: <CH2PR04MB70781FC64AC0621F7E55B408E7019@CH2PR04MB7078.namprd04.prod.outlook.com>

On 2021/07/01 3:58, Sagi Grimberg wrote:
> 
>>> diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
>>> index 21140132a30d..600c5dd1a069 100644
>>> --- a/include/linux/blk-mq.h
>>> +++ b/include/linux/blk-mq.h
>>> @@ -403,6 +403,7 @@ enum {
>>>   	 */
>>>   	BLK_MQ_F_STACKING	= 1 << 2,
>>>   	BLK_MQ_F_TAG_HCTX_SHARED = 1 << 3,
>>> +	BLK_MQ_F_NOT_USE_MANAGED_IRQ = 1 << 4,
>>
>> My 2 cents: BLK_MQ_F_NO_MANAGED_IRQ may be a better/shorter name.
> 
> Maybe BLK_MQ_F_UNMANAGED_IRQ?

Even better :)


-- 
Damien Le Moal
Western Digital Research


From ming.lei at redhat.com  Wed Jun 30 16:59:26 2021
From: ming.lei at redhat.com (Ming Lei)
Date: Thu, 1 Jul 2021 07:59:26 +0800
Subject: [PATCH 0/2] blk-mq: fix blk_mq_alloc_request_hctx
In-Reply-To: <89081624-fedd-aa94-1ba2-9a137708a1f1@suse.de>
References: <20210629074951.1981284-1-ming.lei@redhat.com>
 <5f304121-38ce-034b-2d17-93d136c77fe6@suse.de>
 <YNwug8n7qGL5uXfo@T590>
 <c1de513a-5477-9d1d-0ddc-24e9166cc717@suse.de>
 <YNw/DcxIIMeg/2VK@T590>
 <e106f9c4-35c3-b2da-cdd8-3c4dff8234d6@grimberg.me>
 <89081624-fedd-aa94-1ba2-9a137708a1f1@suse.de>
Message-ID: <YN0FXrcwXfAwGU6w@T590>

On Wed, Jun 30, 2021 at 09:46:35PM +0200, Hannes Reinecke wrote:
> On 6/30/21 8:59 PM, Sagi Grimberg wrote:
> > 
> > > > > > Shouldn't we rather modify the tagset to only refer to
> > > > > > the current online
> > > > > > CPUs _only_, thereby never submit a connect request for hctx with only
> > > > > > offline CPUs?
> > > > > 
> > > > > Then you may setup very less io queues, and performance may suffer even
> > > > > though lots of CPUs become online later.
> > > > > ;
> > > > Only if we stay with the reduced number of I/O queues. Which is
> > > > not what I'm
> > > > proposing; I'd rather prefer to connect and disconnect queues
> > > > from the cpu
> > > > hotplug handler. For starters we could even trigger a reset once
> > > > the first
> > > > cpu within a hctx is onlined.
> > > 
> > > Yeah, that need one big/complicated patchset, but not see any advantages
> > > over this simple approach.
> > 
> > I tend to agree with Ming here.
> 
> Actually, Daniel and me came to a slightly different idea: use cpu hotplug
> notifier.
> Thing is, blk-mq already has cpu hotplug notifier, which should ensure that
> no I/O is pending during cpu hotplug.

Why should we ensure that for non-managed irq?

> If we now add a nvme cpu hotplug notifier which essentially kicks off a
> reset once all cpu in a hctx are offline the reset logic will rearrange the
> queues to match the current cpu layout.
> And when the cpus are getting onlined we'll do another reset.
> 
> Daniel is currently preparing a patch; let's see how it goes.

What is the advantage of that big change over this simple way?

Thanks, 
Ming



From dan.j.williams at intel.com  Sun Jun  6 21:43:18 2021
From: dan.j.williams at intel.com (Dan Williams)
Date: Mon, 07 Jun 2021 04:43:18 -0000
Subject: [PATCH 17/26] nvdimm-pmem: convert to
 blk_alloc_disk/blk_cleanup_disk
In-Reply-To: <20210521055116.1053587-18-hch@lst.de>
References: <20210521055116.1053587-1-hch@lst.de>
 <20210521055116.1053587-18-hch@lst.de>
Message-ID: <CAPcyv4gFEbZH4sXbkvQ32Xv1HiZ6JPL04efGpAWCqaJP_X9jaA@mail.gmail.com>

[ add Sachin who reported this commit in -next ]

On Thu, May 20, 2021 at 10:52 PM Christoph Hellwig <hch at lst.de> wrote:
>
> Convert the nvdimm-pmem driver to use the blk_alloc_disk and
> blk_cleanup_disk helpers to simplify gendisk and request_queue
> allocation.
>
> Signed-off-by: Christoph Hellwig <hch at lst.de>
> ---
>  drivers/nvdimm/pmem.c | 15 +++++----------
>  1 file changed, 5 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
> index 968b8483c763..9fcd05084564 100644
> --- a/drivers/nvdimm/pmem.c
> +++ b/drivers/nvdimm/pmem.c
> @@ -338,7 +338,7 @@ static void pmem_pagemap_cleanup(struct dev_pagemap *pgmap)
>         struct request_queue *q =
>                 container_of(pgmap->ref, struct request_queue, q_usage_counter);
>
> -       blk_cleanup_queue(q);
> +       blk_cleanup_disk(queue_to_disk(q));

This is broken. This comes after del_gendisk() which means the queue
device is no longer associated with its disk parent. Perhaps @pmem
could be stashed in pgmap->owner and then this can use pmem->disk? Not
see any other readily available ways to get back to the disk from here
after del_gendisk().


From hch at lst.de  Tue Jun  8 06:43:15 2021
From: hch at lst.de (Christoph Hellwig)
Date: Tue, 08 Jun 2021 13:43:15 -0000
Subject: please drop the nvme code from net-next
In-Reply-To: <CAMuHMdWgLf8GJfOaRUyx=AvOTnuOs8FS-2=C+OCk02OLDCyrDg@mail.gmail.com>
References: <39b1a3684880e1d85ef76e34403886e8f1d22508.1623149635.git.geert+renesas@glider.be>
 <20210608121925.GA24201@lst.de>
 <CAMuHMdWgLf8GJfOaRUyx=AvOTnuOs8FS-2=C+OCk02OLDCyrDg@mail.gmail.com>
Message-ID: <20210608134303.GA30977@lst.de>

Hi Dave,

please drop the nvme-offload code from net-next.  Code for drivers/nvme/
needs ACKs from us nvme maintainers and for something this significant
also needs to go through the NVMe tree.  And this code is not ready yet.


From smalin at marvell.com  Tue Jun  8 08:57:09 2021
From: smalin at marvell.com (Shai Malin)
Date: Tue, 08 Jun 2021 15:57:09 -0000
Subject: please drop the nvme code from net-next
Message-ID: <SJ0PR18MB38822C7D07B54625B26C39A6CC379@SJ0PR18MB3882.namprd18.prod.outlook.com>


On Tue, Jun 8, 2021 at 4:38 PM Christoph Hellwig <hch at lst.de> wrote:
> Hi Dave,
> 
> please drop the nvme-offload code from net-next.  Code for drivers/nvme/
> needs ACKs from us nvme maintainers and for something this significant
> also needs to go through the NVMe tree.  And this code is not ready yet.

Hi all,

Sorry for any confusion we may have caused. Our plan was for the qed series 
to be considered for net-next, and for the nvme-tcp-offload series to be 
considered for nvme mailing list.
We attempted to communicate this in the cover letter and the addresses 
in to: vs. those in cc:, but perhaps this was not clear enough.

The plan was detailed in the cover latter under the "upstream plan" section 
https://lore.kernel.org/netdev/20210531225222.16992-1-smalin at marvell.com/
The series is structured in a modular way so that part 1 (nvme-tcp-offload) 
and part 2 (qed) are independent and part 3 (qedn) depends on both parts 1+2.

We have sent the first 2 parts which are independent:

- QED NVMeTCP Offload - https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=eda1bc65b0dc1b03006e427430ba23746ec44714
  This part includes the qed infrastructure which was discussed over the RFC.
  Our intent for this part was for it to be accepted to net-next, which it was.
  Dave, from our perspective this piece can stay in net-next.

- NVMeTCP Offload ULP - https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=5ff5622ea1f16d535f1be4e478e712ef48fe183b
  This is the nvme-tcp-offload ULP which we intended the NVMe tree,
  and it shouldn't be merged to net-next.
  Dave, please revert this from net-next until nvme maintainers are completely 
  satisfied with it.

Christoph, we would be more than happy to incorporate any feedback you may 
provide for any part of the series.


From smalin at marvell.com  Tue Jun  8 12:51:40 2021
From: smalin at marvell.com (Shai Malin)
Date: Tue, 08 Jun 2021 19:51:40 -0000
Subject: please drop the nvme code from net-next
Message-ID: <SJ0PR18MB3882C20793EA35A3E8DAE300CC379@SJ0PR18MB3882.namprd18.prod.outlook.com>


On Tue, Jun 08, 2021 at 10:41:00PM -0700, Keith Busch wrote:
> On Tue, Jun 08, 2021 at 12:08:39PM -0700, David Miller wrote:
> > From: Christoph Hellwig <hch at lst.de>
> > Date: Tue, 8 Jun 2021 15:43:03 +0200
> >
> > > please drop the nvme-offload code from net-next.  Code for drivers/nvme/
> > > needs ACKs from us nvme maintainers and for something this significant
> > > also needs to go through the NVMe tree.  And this code is not ready yet.
> >
> > Please send me a revert, and I will apply it, thank you.
> >
> > It's tricky because at least one driver uses the new interfaces.
> 
> Shouldn't whoever merged un-ACK'ed patches from a different subsystem
> get to own the tricky revert?

Dave, we will provide a revert patch.


