Make NVME shutdown two-pass - Version 5

[Jeremy Allison]

This is version 5 of a patchset originally written by
Tanjore Suresh <tansuresh at google.com> to make shutdown
of nvme devices two-pass.

Changes from version 4:

1). Isolated 'enum shutdown_type' change into
drivers/nvme/host/pci.c only.

2). Added a new function: nvme_request_shutdown()
that requests shutdown but doesn't wait for completion,
called from within nvme_dev_disable() when shutdown_type
is NVME_PCI_DISABLE_SHUTDOWN_TWOPASS.

I didn't get feedback on my 'bool or enum' request,
so decided to go with the enum as it reduced the amount
of code changes and makes things cleaner (IMHO).

-------------------------------------------------------------
Currently the Linux nvme driver shutdown code steps
through each connected drive, sets the NVME_CC_SHN_NORMAL
(normal shutdown) flag and then polls the given drive
waiting for the response NVME_CSTS_SHST_CMPLT flag
(shutdown complete).

Each drive is taking around 13 seconds to respond to this.

The customer has 20+ drives on the box so this time adds
up on shutdown when the nvme driver is being shut down.

This patchset changes shutdown to proceed in parallel,
so the NVME_CC_SHN_NORMAL (normal shutdown) flag is
sent to all drives first, and then it polls waiting
for the NVME_CSTS_SHST_CMPLT flag (shutdown complete)
for all drives.

In the specific customer case it reduces the NVME
shutdown time from over 300 seconds to around 15
seconds.
-------------------------------------------------------------

Thanks for your consideration,

Jeremy Allison.
CIQ / Samba Team.