Start with the link
Confirm power, reference clock, reset timing, lane wiring, and supported link generation/width. Check the controller's link status and LTSSM state using vendor documentation. A link that repeatedly retrains should be investigated before application-level benchmarks.
Confirm enumeration
On Linux, tools such as lspci -vv and the kernel log can show whether the root complex discovers the endpoint and what capabilities are negotiated. On RTOS or bare-metal systems, use the platform's configuration-space inspection tools.
Inspect BAR assignment
BARs describe endpoint resources. Confirm the BAR type and size, that the root complex assigned an address, and that the bridge windows cover the required range. Do not assume a BAR address is a CPU physical address without checking the platform's address translation and mapping rules.
Measure transactions carefully
Separate configuration traffic from memory reads/writes, account for transaction size and completion latency, and compare posted writes with non-posted reads. Small 32-bit reads can be dominated by round-trip latency rather than link bandwidth. Use aligned, repeated transfers and report both latency and throughput.
Checklist
- Verify reset, clocks, power, lane mapping, and link width/speed.
- Check LTSSM stability and negotiated capabilities.
- Confirm enumeration, BAR sizing/assignment, and address translation.
- Check IOMMU, inbound/outbound windows, and DMA configuration where relevant.
- Measure large transfers separately from small register accesses.
- Change one variable at a time and preserve logs.
Register names and diagnostic commands vary by controller and operating system. Use the SoC TRM, PCIe controller guide, and board schematic as primary references.
Final takeaway
Low-level engineering is about understanding the contract between hardware, firmware, and the operating system. Start with the specification, form a testable hypothesis, collect evidence, and validate behavior on the target platform.