[PATCHv2 00/10] Second attempt at blk-mq + nvme hotplug fixes

[Keith Busch]

On Tue, 6 Jan 2015, Ming Lin wrote:
> Hi Keith,
>
> Tested with qemu-nvme. Hotplug seems work., but has some issues.
>
> On guest: run fio for a while
> root at block:~# fio --name=global --filename=/dev/nvme0n1 --direct=1 --bs=4k \
>     --rw=randrw --ioengine=libaio --iodepth=128 --name=foobar
> foobar: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> 2.0.8
> Starting 1 process
> Jobs: 1 (f=1): [m] [10.3% done] [3746K/3980K /s] [936 /995  iops] [eta
> 08m:52s]s]
>
> On guest: then remove nvme device
> root at block:~# time echo 1 > /sys/devices/pci0000:00/0000:00:04.0/remove
> real 1m0.040s
> user 0m0.000s
> sys 0m0.012
>
> It works, but took long time(1 minutes) to return.
> And during this time frame, the whole qemu system was not responsible.
> Ping from host to guest also failed.

Thanks for the feedback. Is there anything special about your emulated
device? I haven't been able to duplicate these results. When running
the same fio job, I get ~0.4s on qemu, and ~2.2s on real h/w.

A 1 mintute removal sounds like you're hitting the admin timeout on
queue deletion.

On the other hand, your namespace backing storage appears to be really
slow, so maybe it's stuck in blk_flush on shutdown. You say the guest
was unresponsive during removal, so that sounds plausible since qemu's
nvme shutdown is down synchronously with the CC register MMIO write.

What if you try this using a ramdisk or tmpfs file for the namespace
storage instead of what you're currently using?

Or add some prints in both the driver's and emualted target's shutdown
path and see where you're getting stuck?