Trying to test my gart/iommu vmcore problem on RH

[Bob Montgomery]

On Tue, 2008-09-23 at 02:29 +0000, Eric W. Biederman wrote:
> Bob Montgomery <bob.montgomery at hp.com> writes:
> > And that leads to the Kdump IO Rule:
> >
> >         The primary kernel is responsible for setting up any necessary
> >         conditions to allow the kdump kernel to perform its required
> >         IO without detecting any iommu.
> 
> Reserving a range or addresses in the iommu I agree with.
> If that range of addresses allows for identity mapping I
> like it better.
> 
> I'm not certain about requiring it.
> 
> I don't like setting up the identity mapping before hand,
> it allows devices to trash the kdump kernel by accident.

The reason for having the primary kernel set up any mapping needed by a
kdump kernel *in advance* is that for a HW IOMMU, this setup actually
consists of modifying data structures (arrays, trees, lists) that are in
the primary kernel's memory, as well as setting registers in the HW.
When the kdump kernel comes up, none of those structures are in its
memory range.  They're just part of the artifacts left in
/dev/oldmem.  So yes, the kdump kernel could query any hardware that it
found, verify that the hardware had previously been in use, read HW
registers to get the root pointers, or list addresses or whatever, and
then modify arrays, trees, or lists in that non-owned memory to map its
DMA, but it's kind of an unprecedented step for the kdump kernel to
take.  (Blindly copying oldmem pages is one thing, manipulating live
data structures in oldmem seems like quite another thing.)

Regarding the danger of trashing the kdump kernel prior to its launch:
Currently, any driver or errant kernel code can trash the kdump area.
And any IO card on a non-IOMMU or swiotlb system can trash it.  So it
doesn't seem like much of an extension of a risk that already exists.

It does however negate one possibility to lower some of that risk.


> >         The kdump kernel must refrain from detecting and initializing
> >         any iommu.
> 
> Why?  I can fully understand avoiding addresses that are in flight.
> I can definitely see this being simpler in the kdump kernel.
> However this feels like it makes a less robust kdump kernel by
> not allowing it to touch the iommu.

As pointed out above, "touching the iommu" really includes touching its
data structures created by the primary kernel in what is now the oldmem
area.  In addition, I'm not sure the kdump kernel can determine which
addresses are in flight by querying either the HW or the oldmem
structures.   It could probably determine which ones were unused at the
time of the crash.


> > This has a these effects:
> >
> > A) Primary kernel: depending on what it is using for as an IOMMU,
> >         it may have to do some (or considerable) setup, to guarantee
> >         that the kdump kernel can have IO capability to its Crash
> >         kernel address range.
> >
> > B) Primary kernel: the Crash kernel range must be set up in an address
> >         range whose physical addresses are accessible to IO cards
> >         without address remapping.
> 
> Below <= 16MB?  That doesn't work in general.

I didn't think this was working now.  Aren't most crash kernels
allocated above 16 MB?  And I assumed lots of systems don't have IOMMU
capability.  Do you have an example where this would be an issue for an
IO card needed by the kdump kernel?  

> Especially not if we are running on an SGI box and someone had
> unplugged node 0 (with all of the memory below 4G).

How does an SGI box with an unplugged node 0 do kdump IO currently?


> > Possible? Comments? Corrections?
> 
> Possible.
> 
> I would very much like the option of doing the iommu setup, and possibly
> fiddling in the kdump kernel.   As long as we are not reusing the same
> addresses in the iommu I don't see a problem.

The problem I see is the oldmem area.  Now we could come up with a plan
to allow the primary kernel to do all of its iommu related allocations
in the Crash kernel area, effectively creating an area of memory that is
shared between the primary kernel and kdump.  (This would be complicated
in cases where the iommu state is in a dynamic tree vs. a fixed size
array.)  Then the kdump kernel would wake up and just take over
maintenance of the iommu.  But even the much smaller proposal to
preallocate entries in the iommu data structures to allow the kdump
kernel to do its IO is already violating one of the principals of kdump.
It is making kdump operation dependent on the integrity of a primary
kernel data structure.  Actually taking over a shared iommu data
structure from the primary kernel seems like an even bigger
philosophical violation.

> I like the theoretical option of disabling ongoing DMA's, with the
> more complete IOMMUs.  It isn't strictly necessary but I expect it
> would give a better result.

It seems like this implies 1) stopping the DMA at the IOMMU, 2)
surviving the resulting error condition when the IO card fails its next
access (hopefully not MCE on a modern IOMMU), 3) verifying that the IO
card won't try another access later after you've started using the IOMMU
in the kdump kernel, and then 4) reinitializing and using the IOMMU.  Is
it doable?

Thanks for considering,
Bob Montgomery