The Enzian 3 v1.5 rear panel

In this post, we’ll take a look at the back of an Enzian system.

Our 9 (currently) Enzian 3 machines are revision r1.5, and are all nicely mounted in 2U rack-mount cases. Here’s the back of one of them, with a few cables connected:

On the back of each case is this I/O panel:

This is rather different to a traditional PC. For example, there’s no video or audio output. Let’s look at all these connectors in turn:

QSFP-C1 and QSFP-C2: ThunderX-1 Ethernet

These two QSFP cages correspond to the two 40Gb/s Ethernet adapters on the Cavium/Marvell ThunderX-1 SoC. Linux has drivers for these NICs, and they are initialized by the boot firmware. They appear as regular network interfaces when you are logged into Linux on the Enzian.

Note that these are actually quite powerful – they are fairly full-featured SmartNICs.

QSFP-F0 - QSFP-F3: FPGA 100Gb/s networking

These four QSFP-28 cages bring out 4 x 25Gb/s bidirectional serial lines on the FPGA per connector. Each cage can be configured as 100Gb/s Ethernet, 4 x 25Gb/s Ethernet, or a variety of other custom protocols based on how the FPGA is programmed.

In fact, we can even go just a tad faster on these: since we’re using the XCVU9P-3 variant of the FPGA, which is the highest speed grade available (thanks, Xilinx!), we can clock these 16 serial lines at 32.75Gb/s, which would give us 131Gb/s per QSFP-28 connector with custom protocols.

Note that the connectors are not in regular order on the rear panel!

While you can use any or all of these QSFP-28 cages, they are all subtly different since they are connected to SerDes on different parts of the FPGA. This really depends on what the application is, or what hard IP blocks you’d like to be near to the transceivers.

USB 3.0

The ThunderX-1 has USB 3.0 support on the SoC in the form of two xHCI controllers. Each of these has a single physical USB 3.0 port on the CPU, and these are brought out on the back as USB type-A connectors.

microSD

This is a microSD slot for the ThunderX-1.

BMC ETH: Board Management Controller Ethernet

Enzian has a rather powerful Board Management Controller (BMC). As in any other server, the BMC handles power and clock sequencing, thermal management and fan control, remote management, etc.

Unlike regular servers, however, the Enzian BMC is open for other use cases. It runs Linaro Linux (currently), and you can log in remotely. It can be programmed to relay telemetry from the various board sensors (power, temperature, etc.), for example.

This RJ45 connector is 1 Gb/s Ethernet for the BMC. It allows ssh access to the Linux instance running on the BMC, and therefore can be used to remotely reset or power-cycle the rest of the Enzian.

BMC USB: Board Management Controller USB

This is a USB 2.0 port for the BMC, available to Linux running on the BMC. For example, a USB storage device plugged into here can be mounted inside the BMC and used to, for example, program the main FPGA.

UART: USB access to system UARTs

Note that this is a Type-B USB connector! If you plug a regular USB “Type-A to Type-B” cable between this and a laptop (say), then the laptop will see this as a quad UART device (specifically, a common-or-garden FTDI FT4232H).

For example, on a Linux PC lsusb -t -v will show something like this:

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 8: Dev 12, If 0, Class=Hub, Driver=hub/4p, 480M
        ID 05e3:0606 Genesys Logic, Inc. USB 2.0 Hub / D-Link DUB-H4 USB 2.0 Hub
        |__ Port 4: Dev 21, If 0, Class=Hub, Driver=hub/4p, 480M
            ID 05e3:0606 Genesys Logic, Inc. USB 2.0 Hub / D-Link DUB-H4 USB 2.0 Hub
            |__ Port 4: Dev 32, If 1, Class=Vendor Specific Class, Driver=ftdi_sio, 480M
                ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC
            |__ Port 4: Dev 32, If 2, Class=Vendor Specific Class, Driver=ftdi_sio, 480M
                ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC
            |__ Port 4: Dev 32, If 0, Class=Vendor Specific Class, Driver=ftdi_sio, 480M
                ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC
            |__ Port 4: Dev 32, If 3, Class=Vendor Specific Class, Driver=ftdi_sio, 480M
                ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC

Each of these 4 serial lines is routed through the BMC’s FPGA to a different console. The serial lines are distinguished over USB by the “Interface” of the USB device, and by default are wired up as follows:

1. The serial console for the BMC. This allows a serial login to the BMC’s OS.
2. The first UART console on the ThunderX-1 CPU. This is the main Linux tty for the Enzian CPU.
3. The second UART on the ThunderX-1 SoC. Currently unused.
4. The console UART on the XCVU9P FPGA.

As OS people, we really like serial ports for low-level access and debugging. This USB connector gives us four UARTs with a minimum of cabling.

JTAG: USB access to JTAG trees

This is the other Type-B USB connector, and this one looks like a dual UART device, a FTDI FT2232C:

/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
    |__ Port 8: Dev 12, If 0, Class=Hub, Driver=hub/4p, 480M
        ID 05e3:0606 Genesys Logic, Inc. USB 2.0 Hub / D-Link DUB-H4 USB 2.0 Hub
        |__ Port 3: Dev 18, If 1, Class=Vendor Specific Class, Driver=ftdi_sio, 480M
            ID 0403:6010 Future Technology Devices International, Ltd FT2232C/D/H Dual UART/FIFO IC
        |__ Port 3: Dev 18, If 0, Class=Vendor Specific Class, Driver=usbfs, 480M
            ID 0403:6010 Future Technology Devices International, Ltd FT2232C/D/H Dual UART/FIFO IC
        |__ Port 4: Dev 21, If 0, Class=Hub, Driver=hub/4p, 480M

This provides complete access to the JTAG tree of Enzian.

PCI Express panels

Next to the I/O shield on our rack-mount cases are the usual set of slots for PCIe cards. In the 2U cases we use, these accommodate half-height cards.

There are only two PCIe slots on the Enzian main board, a single PCIe Gen3 x16 slot for the FPGA, and also a PCIe Gen3 x8 slot for the Thunder-X1. The CPU’s SoC does not support PCIe x16, hence x8.

If you look closely at the photograph at the top, you can actually see an Intel Gigabit Ethernet card plugged into the CPU’s PCIe slot, which was helpful for remote access to the CPU when debugging the 40Gb/s Ethernet interfaces.