FAQ

Q. How can I get {1,2,10,50,200} Enzian systems? When?

A. We have two Enzian prototypes right now, and they’re large ungainly systems put together from a variety of development boards and custom PCBs we’ve built. The production board design is happening, and we hope to have the first boards by the end of summer 2018.

Q. How much will an Enzian board cost?

A. We’ll see. The cost is dominated by the DDR4 and the FPGA. We’re hoping it will be roughly the cost of a conventional well-populated 2-socket server machine.

Q. Will I be able to partially reconfigure the FPGA from the CPU on the fly?

A. Yes.

Q. Can I plug a GPU in?

A. Yes, on either side, if you really must. The ThunderX has two PCIe x8 links, and the FPGA has an x16, both of which get put through the PCIe switch on the main board. It’ll increase the enclosure size from 2U to 3U, but the power and cooling should cope.

Q. What about other connectors, like cameras?

A. In addition to the network, USB, and PCIe on both sides, there’s also an FMC connector pinned out from the FPGA.

Q. What about intersocket interrupts?

A. These can be carried over ECI (the "Enzian Coherence Interface"): there are ECI messages for interrupts so the FPGA can raise an interrupt on any core. The CPU side is the standard ARM GIC arrangements.

Q. Is all the memory cache-coherent between the CPU and FPGA?

A. The short answer is that it’s up to you, but the full answer is a bit complicated since you should not think of the FPGA as just another processor. Your VHDL or Verilog on the FPGA sees all the coherency messages in ECI, our implementation of the ThunderX’s MOESI-like protocol. We expose these via a set of FIFOs, and handle the flow control, but what you do when you see a cache-line fetch is up to you. In many cases you’ll just want to perform the relevant load from the FPGA’s DDR4 (and we’ll supply Verilog for this), but there are a lot more interesting usecases.

Q. What software will run on the CPU side?

A. We boot Barrelfish (naturally), Linux, and FreeBSD on the CPU side today. In Linux and FreeBSD, there’s a simple kernel driver that maps the whole the FPGA’s section on the system address space into user space. For usecases that really want to treat Enzian as a conventional server with an FPGA on the side, this is fine and we won’t judge you. If you want to go fully heterogeneous and dynamically instantiate first-class cores on the FPGA, Barrelfish may be more to your taste.

Q. What’s the form factor?

A. As you can see from the photos, our two prototypes are pretty big, but the final board is an EATX form factor, and a 2U rackmount case, with front-to-back cooling and all the high-speed connectors on the rear I/O panel. You should be able to fit 15-20 of these in a single standard 19” rack.

Q. Why did you not use a Zynq, or whatever?

A. There are some nice FPGAs out there from Xilinx and Intel with hard CPU cores. However, they are not realistic processors from a server perspective (nor are they intended to be) – a cluster of four Cortex-A53 cores is typical, and that’s not in the same class as even a mature server part like the ThunderX-1.

Early on we did consider one of these MPSoCs as the FPGA side of the platform, but ultimately our choice of part was driven by the need to maximise transceivers - we went for bandwidth above all else, and this means a straight high-end FPGA.

Q. What other systems are there that look this?

A. We’re unaware of anything that looks like Enzian, which is why we’re building it. However, there are other interesting systems in this general space: the Intel HARP platform, for one. There are also a wide variety of platforms that put an FPGA on a PCIe card, including Microsoft’s Catapult, Amazon’s F1 cloud offering, and a lot of FPGA development boards. These are fine if you can live with PCIe. We didn’t want to.

Q. What about OpenCAPI / CCIX / Gen-Z / PCIe solutions?

A. We deliberately avoided all these standards. In some cases, we simply didn’t have access to them, or there was no available silicon. However, the principle reason is that we wanted to eliminate as many prior assumptions that might limit the platform. The CPU’s own coherency protocol itself has many assumptions in it of course, but it’s the closest we can get.

Q. How can I contribute to Enzian?

A. Right now we’ve only got two prototypes, so it’s a bit hard to work on the hardware (we are working on a simulator). Once the boards are available we’ll have a community site for people to contribute useful bits of software and hardware.

Q. Is Enzian open-source?

A. We’re releasing all of the software we write for Enzian as open source. It is our goal to also open-source as much of the FPGA doughnut as we can, including our implementaton of ECI, which is needed to understand how to talk to the CPU in more interesting ways than just shared memory. We will own the board design, so it may be possible and/or a good idea to release that as well.

Q. Why is there no high-bandwidth memory on Enzian?

A. We’d have liked it (and had a few usecases in mind), but when we started there were no available FPGAs with HBM on-chip, and we simply ran out of transcievers and pins after accomodating ECI, DDR, and the QSFP28 cages. You can plug HMC in, but you won’t get much more bandwidth than NVMe. We had to make a choice between a “slow” interface to HBM, and a full-speed interface to NVMe and PCIe x16 on the FPGA side, and we went for the latter.

Q. What about ThunderX-2?

A. ThunderX-2 wasn’t available when we first build the Enzian prototypes, and we wanted a known quantity to work with on the CPU side rather than a brand-new chip (this is the first time we’ve done this, after all…). Moving forward, ThunderX-2 is a much faster processor, we’d love a future revision of Enzian to use it, and we’re actively discussing this with Cavium. ThunderX-2 is very different from ThunderX-1, however, and the cache coherence protocols used in ThunderX-2 and ThunderX-1 are very different from each other, so this will involve significant reimplementation.

Q. Can I use this for machine learning or artificial intelligence?

A. Yes, if you like.

Q. Can I use this for blockchain operations or cryptocurrency mining?

A. You are not our target audience for this platform.