What is Enzian?

Enzian is a powerful computer for doing systems software research. You can use a single Enzian on its own, or connect a rack of them together.

Enzian is a cache-coherent 2-node asymmetric NUMA system where one node is a 48-core CPU and one node is an FPGA. It has a maximum of 640GiB of DDR4 RAM, and has up 480Gb/s of network bandwidth, both split between the two nodes. The board looks like this:

Enzian board overview, with labels

Current Enzian specifications

Node 0:

  • Marvell Cavium ThunderX-1 CN8890-NT CPU @ 2 GHz (48 x ARMv8.1 cores)
  • 128 GiB DDR4, 4x 32 GiB DIMMS @ 2133 MT/s
  • PCIe Gen3 x8 slot
  • 3 x NVMe connectors
  • 4 x SATA connectors
  • 2 x 40Gb/s Ethernet QSFP28 connectors
  • USB3, serial UARTs.
  • JTAG

Node 1:

  • Xilinx CVU9P FPGA
  • 512 GiB DDR4, 4x 128 GiB DIMMS @ 2133 MT/s or 64 GiB DDR4, 4x 16 GiB DIMMS @ 2400 MT/s
  • PCIe Gen3 x16 slot
  • 1 x NVMe connector
  • FMC connector
  • 16 x 25Gb/s serial lines in 4 x QSFP28 cages, configurable as 16 25Gb/s or 4 x 100Gb/s Ethernet.
  • JTAG

Coherent interconnect:

  • 24 x 10Gb/s lanes
  • MOESI-like directory-based protocol
  • Inter-processor interrupts
  • Cached and uncached (I/O) accesses
  • 128-byte cache line size
  • Shared physical address space

Baseboard Management Controller (BMC):

  • Enclustra Mercury SoM
  • Xilinx Zynq MPSoc CPU running Linux
  • Dedicated 1Gb/s RJ45 Ethernet
  • JTAG
  • All system serial ports brought out as USB UARTs

Software:

  • Ubuntu Linux 18.04 LTS, other OSes should boot.
  • ARM Trusted Firmware
  • Board development kit for low-level access

Form factor:

  • eATX-format motherboard (305 × 330 mm / 12 × 13 in)
  • 2U Rackmount case
  • Can accomodate double-width PCIe cards with riser.
  • 1000W Power supply.

Block diagram

Block diagram of
    Enzian V.3