<div dir="ltr"><br><div class="gmail_extra">Hi Takao,</div><div class="gmail_extra"><br></div><div class="gmail_extra" style>Because I want to config the kdump in our lab, I have some questions</div><div class="gmail_extra" style>
about the backgroud of your patch.</div><div class="gmail_extra"><br></div><div class="gmail_extra" style>1. Will you patch only work while IOMMU is enabled?</div><div class="gmail_extra" style><br></div><div class="gmail_extra" style>
2. Looks like kdump doesn't disable DMA and interrupts in the systems, </div><div class="gmail_extra" style>how can Linux make sure second kernel's memory is not corrupted by</div><div class="gmail_extra" style>devices initialized by first kernel?</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">2012/11/27 Takao Indoh <span dir="ltr"><<a href="mailto:indou.takao@jp.fujitsu.com" target="_blank">indou.takao@jp.fujitsu.com</a>></span><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
These patches reset PCIe devices at boot time to address DMA problem on<br>
kdump with iommu. When "reset_devices" is specified, a hot reset is<br>
triggered on each PCIe root port and downstream port to reset its<br>
downstream endpoint.<br>
<br>
Background:<br>
A kdump problem about DMA has been discussed for a long time. That is,<br>
when a kernel is switched to the kdump kernel, DMA derived from first<br>
kernel affects second kernel. Especially this problem surfaces when<br>
iommu is used for PCI passthrough on KVM guest. In the case of the<br>
machine I use, when intel_iommu=on is specified, DMAR error is detected<br>
in kdump kernel and PCI SERR is also detected. Finally kdump fails<br>
because some devices does not work correctly.<br>
<br>
The root cause is that ongoing DMA from first kernel causes DMAR fault<br>
because page table of DMAR is initialized while kdump kernel is booting<br>
up. Therefore to solve this problem DMA needs to be stopped before DMAR<br>
is initialized at kdump kernel boot time. By these patches, PCIe devices<br>
are reset by hot reset and its DMA is stopped when reset_devices is<br>
specified. One problem of this solution is that the monitor blacks out<br>
when VGA controller is reset. So this patch does not reset the port<br>
whose child endpoint is VGA device.<br>
<br>
What I tried:<br>
- Clearing bus master bit and INTx disable bit at boot time<br>
This did not solve this problem. I still got DMAR error on devices.<br>
- Resetting devices in fixup_final(v1 patch)<br>
DMAR error disappeared, but sometimes PCI SERR was detected. This<br>
is well explained here.<br>
<a href="https://lkml.org/lkml/2012/9/9/245" target="_blank">https://lkml.org/lkml/2012/9/9/245</a><br>
This PCI SERR seems to be related to interrupt remapping.<br>
- Clearing bus master in setup_arch() and resetting devices in<br>
fixup_final<br>
Neither DMAR error nor PCI SERR occurred. But on certain machine<br>
kdump kernel hung up when resetting devices. It seems to be a<br>
problem specific to the platform.<br>
- Resetting devices in setup_arch() (v2 and later patch)<br>
This solution solves all problems I found so far.<br>
<br>
Changelog:<br>
v7:<br>
Update Yinghai's dummy-pci patch with macros in linux/pci.h, and fix<br>
some bugs<br>
<br>
v6:<br>
Rewrite using Yinghai's dummy-pci patch<br>
<a href="https://lkml.org/lkml/2012/11/13/118" target="_blank">https://lkml.org/lkml/2012/11/13/118</a><br>
<br>
v5:<br>
Do bus reset after all devices are scanned and its config registers are<br>
saved. This fixes a bug that config register is accessed without delay<br>
after reset.<br>
<a href="https://lkml.org/lkml/2012/10/17/47" target="_blank">https://lkml.org/lkml/2012/10/17/47</a><br>
<br>
v4:<br>
Reduce waiting time after resetting devices. A previous patch does reset<br>
like this:<br>
for (each device) {<br>
save config registers<br>
reset<br>
wait for 500 ms<br>
restore config registers<br>
}<br>
<br>
If there are N devices to be reset, it takes N*500 ms. On the other<br>
hand, the v4 patch does:<br>
for (each device) {<br>
save config registers<br>
reset<br>
}<br>
wait 500 ms<br>
for (each device) {<br>
restore config registers<br>
}<br>
Though it needs more memory space to save config registers, the waiting<br>
time is always 500ms.<br>
<a href="https://lkml.org/lkml/2012/10/15/49" target="_blank">https://lkml.org/lkml/2012/10/15/49</a><br>
<br>
v3:<br>
Move alloc_bootmem and free_bootmem to early_reset_pcie_devices so that<br>
they are called only once.<br>
<a href="https://lkml.org/lkml/2012/10/10/57" target="_blank">https://lkml.org/lkml/2012/10/10/57</a><br>
<br>
v2:<br>
Reset devices in setup_arch() because reset need to be done before<br>
interrupt remapping is initialized.<br>
<a href="https://lkml.org/lkml/2012/10/2/54" target="_blank">https://lkml.org/lkml/2012/10/2/54</a><br>
<br>
v1:<br>
Add fixup_final quirk to reset PCIe devices<br>
<a href="https://lkml.org/lkml/2012/8/3/160" target="_blank">https://lkml.org/lkml/2012/8/3/160</a><br>
<br>
Takao Indoh (5):<br>
x86, pci: add dummy pci device for early stage<br>
PCI: Define the maximum number of PCI function<br>
Make reset_devices available at early stage<br>
x86, pci: Reset PCIe devices at boot time<br>
x86, pci: Enable PCI INTx when MSI is disabled<br>
<br>
arch/x86/include/asm/pci-direct.h | 3 +<br>
arch/x86/kernel/setup.c | 3 +<br>
arch/x86/pci/common.c | 4 +-<br>
arch/x86/pci/early.c | 315 +++++++++++++++++++++++++++++++++++++<br>
include/linux/pci.h | 2 +<br>
init/main.c | 4 +-<br>
6 files changed, 328 insertions(+), 3 deletions(-)<br>
<br>
<br>
--<br>
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in<br>
the body of a message to <a href="mailto:majordomo@vger.kernel.org">majordomo@vger.kernel.org</a><br>
More majordomo info at <a href="http://vger.kernel.org/majordomo-info.html" target="_blank">http://vger.kernel.org/majordomo-info.html</a><br>
Please read the FAQ at <a href="http://www.tux.org/lkml/" target="_blank">http://www.tux.org/lkml/</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>------------------<br>Oliver Yang
</div></div>