<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 129.75pt 1.0in 129.7pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoPlainText>Thierry,<o:p></o:p></p><p class=MsoPlainText>Are you doing below BARs programing for config space or extended config space? <b>pcie->cfg</b> <b>& comments</b> indicates config and <b>fpci_bar = 0xfe0... & size</b> indicates extended config.<o:p></o:p></p><p class=MsoPlainText>Also make sure you add up 0x10000000(according to your below impl.) to pcie->cs/pcie->extcs in pcie_read(write)_conf api so that type1 addresses will actually be mapped to 0xfe1… fpci address.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>> -----Original Message-----</p><p class=MsoPlainText>> From: Thierry Reding [mailto:thierry.reding@avionic-design.de]</p><p class=MsoPlainText>> Sent: Thursday, December 20, 2012 9:03 PM</p><p class=MsoPlainText>> To: Stephen Warren</p><p class=MsoPlainText>> Cc: Jason Gunthorpe; Jay Agarwal; Thomas Petazzoni; Andrew Lunn; Jason</p><p class=MsoPlainText>> Cooper; Gregory Clement; linux-arm-kernel@lists.infradead.org</p><p class=MsoPlainText>> Subject: Re: [RFC v1] PCIe support for the Armada 370 and Armada XP SoCs</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> * PGP Signed by an unknown key</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> On Tue, Dec 18, 2012 at 10:03:38AM -0700, Stephen Warren wrote:</p><p class=MsoPlainText>> > On 12/17/2012 07:51 PM, Jason Gunthorpe wrote:</p><p class=MsoPlainText>> > > [trimmed the cc list]</p><p class=MsoPlainText>> > ></p><p class=MsoPlainText>> > > On Mon, Dec 17, 2012 at 07:10:59PM -0700, Stephen Warren wrote:</p><p class=MsoPlainText>> > ></p><p class=MsoPlainText>> > >>> What I'll do next is add some caching of written values, so that</p><p class=MsoPlainText>> > >>> reading them back actually yields something correct. After that</p><p class=MsoPlainText>> > >>> I'll post what I have so that others can look at it or even reuse</p><p class=MsoPlainText>> > >>> it.</p><p class=MsoPlainText>> > >></p><p class=MsoPlainText>> > >> Some internal discussion implies this shouldn't be required; here's</p><p class=MsoPlainText>> > >> what they said:</p><p class=MsoPlainText>> > ></p><p class=MsoPlainText>> > > Looking at the driver, it says:</p><p class=MsoPlainText>> > ></p><p class=MsoPlainText>> > > + * 0x80000000 - 0x80003fff - PCI controller registers</p><p class=MsoPlainText>> > > + * 0x80004000 - 0x80103fff - PCI configuration space</p><p class=MsoPlainText>> > > + * 0x80104000 - 0x80203fff - PCI extended configuration space</p><p class=MsoPlainText>> > ></p><p class=MsoPlainText>> > > And the bridge config regions are at:</p><p class=MsoPlainText>> > ></p><p class=MsoPlainText>> > > +#define RP0_OFFSET 0x0000</p><p class=MsoPlainText>> > > +#define RP1_OFFSET 0x1000</p><p class=MsoPlainText>> > ></p><p class=MsoPlainText>> > > And from the first checkin, config space accesses have been split</p><p class=MsoPlainText>> > > into three designations:</p><p class=MsoPlainText>> > > bridge #0 - 0x80000000 + 0</p><p class=MsoPlainText>> > > bridge #1 - 0x80000000 + 0x1000</p><p class=MsoPlainText>> > > Every other bus - 0x80104000 + [equation]</p><p class=MsoPlainText>> > ></p><p class=MsoPlainText>> > > And based on Thierry's results it looks like once the bridge</p><p class=MsoPlainText>> > > subordinate bus numbers are setup (via their lower mappings) then</p><p class=MsoPlainText>> > > the 'every other bus' mapping starts working as expected.</p><p class=MsoPlainText>> > ></p><p class=MsoPlainText>> > > So.. is there a register to be set that lets the bridge</p><p class=MsoPlainText>> > > configuration be accessed through the [0x80004000 - 0x80103fff] region</p><p class=MsoPlainText>> as bus 0?</p><p class=MsoPlainText>> > > Presumably a non-PCI device number register?</p><p class=MsoPlainText>> ></p><p class=MsoPlainText>> > It looks like some PCI host controller registers do need to be set up</p><p class=MsoPlainText>> > (so far the information is related to "type 0" vs. "type 1"</p><p class=MsoPlainText>> > configuration accesses, although I'm not familiar with what that means).</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> So I've been spending some time implementing this along with some of the</p><p class=MsoPlainText>> things that Stephen and I discussed on IRC. However, each time I try to</p><p class=MsoPlainText>> access the type 0 configuration space through the special mapping that</p><p class=MsoPlainText>> Stephen dug up I get the following "imprecise external abort" oops:</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> [ 3.482802] tegra-pcie 80003000.pcie-controller: PCI host bridge to bus</p><p class=MsoPlainText>> 0000:00</p><p class=MsoPlainText>> [ 3.490107] pci_bus 0000:00: root bus resource [io 0x82000000-0x8200ffff]</p><p class=MsoPlainText>> [ 3.517456] pci_bus 0000:00: root bus resource [mem 0xb0000000-</p><p class=MsoPlainText>> 0xb7ffffff]</p><p class=MsoPlainText>> [ 3.530644] pci_bus 0000:00: root bus resource [mem 0xb8000000-0xbfffffff</p><p class=MsoPlainText>> pref]</p><p class=MsoPlainText>> [ 3.554191] pci_bus 0000:00: root bus resource [bus 00-ff]</p><p class=MsoPlainText>> [ 3.559693] Unhandled fault: imprecise external abort (0x406) at</p><p class=MsoPlainText>> 0x00034384</p><p class=MsoPlainText>> [ 3.559702] tegra-pcie 80003000.pcie-controller: AXI response decoding</p><p class=MsoPlainText>> error, signature: 00000001</p><p class=MsoPlainText>> [ 3.559708] tegra-pcie 80003000.pcie-controller: FPCI address: fe00000000</p><p class=MsoPlainText>> [ 3.582435] Internal error: : 406 [#1] PREEMPT SMP ARM</p><p class=MsoPlainText>> [ 3.587560] Modules linked in:</p><p class=MsoPlainText>> [ 3.590616] CPU: 1 Not tainted (3.7.0-next-20121219-00103-gb307de6-</p><p class=MsoPlainText>> dirty #42)</p><p class=MsoPlainText>> [ 3.591125] usb 1-1.1: new high-speed USB device number 3 using tegra-</p><p class=MsoPlainText>> ehci</p><p class=MsoPlainText>> [ 3.604944] PC is at tegra_pcie_read_conf+0x60/0xd4</p><p class=MsoPlainText>> [ 3.609823] LR is at ioremap_page_range+0xec/0x19c</p><p class=MsoPlainText>> [ 3.614605] pc : [<c0025034>] lr : [<c01dfcec>] psr: a0000093</p><p class=MsoPlainText>> [ 3.614605] sp : da645d08 ip : 90000000 fp : c06a8bf4</p><p class=MsoPlainText>> [ 3.626063] r10: 00000000 r9 : da645de0 r8 : dae89400</p><p class=MsoPlainText>> [ 3.631274] r7 : 00000000 r6 : 00000000 r5 : 00000004 r4 : 00000000</p><p class=MsoPlainText>> [ 3.637790] r3 : ffffffff r2 : 00000000 r1 : e0823000 r0 : e0822000</p><p class=MsoPlainText>> [ 3.644309] Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment</p><p class=MsoPlainText>> kernel</p><p class=MsoPlainText>> [ 3.651696] Control: 10c5387d Table: 1a6f404a DAC: 00000015</p><p class=MsoPlainText>> [ 3.657437] Process kworker/u:3 (pid: 55, stack limit = 0xda644238)</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> I beefed up the interrupt service routine a bit to output the full FPCI address</p><p class=MsoPlainText>> that the decoding error occurred and it always is the one configured for the</p><p class=MsoPlainText>> type 0 configuration space mapping. I assume that the "Internal error" is due</p><p class=MsoPlainText>> to the "imprecise external abort", but it would be good if somebody could</p><p class=MsoPlainText>> confirm that.</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> Basically I have modified tegra_pcie_setup_translations() to have this:</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> /* Bar 0: type 0 configuration space */</p><p class=MsoPlainText>> fpci_bar = 0xfe000000;</p><p class=MsoPlainText>> size = resource_size(pcie->cfg[0]);</p><p class=MsoPlainText>> axi_address = pcie->cfg[0]->start;</p><p class=MsoPlainText>> afi_writel(pcie, axi_address, AFI_AXI_BAR0_START);</p><p class=MsoPlainText>> afi_writel(pcie, size >> 12, AFI_AXI_BAR0_SZ);</p><p class=MsoPlainText>> afi_writel(pcie, fpci_bar, AFI_FPCI_BAR0);</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> /* Bar 1: type 1 configuration space */</p><p class=MsoPlainText>> fpci_bar = 0xfe100000;</p><p class=MsoPlainText>> size = resource_size(pcie->cfg[1]);</p><p class=MsoPlainText>> axi_address = pcie->cfg[1]->start;</p><p class=MsoPlainText>> afi_writel(pcie, axi_address, AFI_AXI_BAR1_START);</p><p class=MsoPlainText>> afi_writel(pcie, size >> 12, AFI_AXI_BAR1_SZ);</p><p class=MsoPlainText>> afi_writel(pcie, fpci_bar, AFI_FPCI_BAR1);</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> Where pcie->cfg[0] and pcie->cfg[1] are parsed from the following device</p><p class=MsoPlainText>> tree snippet:</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> reg = <0x80003000 0x00000800 /* PADS registers */</p><p class=MsoPlainText>> 0x80003800 0x00000200 /* AFI registers */</p><p class=MsoPlainText>> 0x90000000 0x10000000 /* type 0 configuration space */</p><p class=MsoPlainText>> 0xa0000000 0x10000000>; /* type 1 configuration space */</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> The complete code can be found in my tegra/next branch[0]. Beware that I</p><p class=MsoPlainText>> haven't cleaned up that branch in a while, so anything but looking at the end</p><p class=MsoPlainText>> result is not very recommended =). The branch contains patches</p><p class=MsoPlainText>> (untested) to make Harmony and TrimSlice initialize the PCIe from the</p><p class=MsoPlainText>> device tree, so anyone inclined to give this a spin should be able to do so. If</p><p class=MsoPlainText>> you want to try out the software emulated host bridge, you can just</p><p class=MsoPlainText>> uncomment the "#define EMULATE_HOST_BRIDGE 1" at the top of</p><p class=MsoPlainText>> arch/arm/mach-tegra/pcie.c.</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> Stephen: Do you think you could dig some more, try to find out what the</p><p class=MsoPlainText>> deal is with these "imprecise external aborts"? Perhaps something else</p><p class=MsoPlainText>> needs to be done in order to activate the type 0 mapping? Also, while</p><p class=MsoPlainText>> looking through some old downstream code from the Vibrante area I saw a</p><p class=MsoPlainText>> HyperTransport document mentioned that contains these mappings. It is</p><p class=MsoPlainText>> referred to as the "AFI IAS" (section 3.2.4 is mentioned specifically).</p><p class=MsoPlainText>> Any chance I could take a look at it?</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> Thierry</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> [0]: git://gitorious.org/thierryreding/linux.git</p><p class=MsoPlainText>> </p><p class=MsoPlainText>> * Unknown Key</p><p class=MsoPlainText>> * 0x7F3EB3A1</p></div></body></html>