Signal integrity testing

Testing the assembled, non-functional Enzian board for signal integrity turned out to be rather more work than we expected, much of it involving machining bits of metal – real hardware.

The high-speed signals we wanted to test on our board, in rough order of decreasing importance, were:

• The coherence lanes between the CPU and FPGA
• The 16 25Gb/s serial lines between the QSFP28 cages and the FPGA
• The DDR4 DIMM connections to the CPU
• The DDR4 DIMM connections to the FPGA
• The 8 10Gb/s serial lines between the QSFP28 cages and the twin 40 Gigabit Ethernet interfaces on the ThunderX
• The FPGA PCIe x16 connector
• The ThunderX PCIe x8 connector

Our wonderful friends over in the Microelectronics Design Center in the Electrical Engineering Department at ETH Zurich agreed to let us use their test gear, but we still needed to accurately attach probes to the rather small pads where the CPU and FPGA sit in order the measure the signals.

We sourced the probes themselves, but now we needed something to hold them, which involved building a test rig from scratch to accurately hold and move the PCB and the probes. ETH Zurich being the wonderful place that it is, we got permission to use the great metalworking shop in the Physics department.

Here are lots of panels marked out for drilling:

And here’s some drilling:

Here’s a picture of Daniel Schwyn with a lot of machined aluminium:

And here’s Michael Giardino assembling the rig:

In retrospect, we didn’t expect the sub-project to snowball quite as much as it did, but after a lot of work by David, Michael, and Daniel, we ended up with a nice completed test rig:

This finally allowed us to test some of the signals. These pictures make it fairly clear why we needed some kind of test rig:

Did it work? It did! We were actually pretty comfortably within tolerance for all the main links we tried, which were the coherence links and the QSFP28 cages.

The other set we really wanted to test was the 8 DDR4 DIMM connectors. Unfortunately, at that point Switzerland entered lockdown…