FAQ

How can I get {1,2,10,50,200} Enzian systems? When?

We have 14 working Enzian boards right now, and 5 more on the way. We’re talking to various people about how to make more of them, so that other groups can have their own, but in the meantime, please talk to us directly if you want to get access to them. We’re keep to get as many people as possible using them!

How much will an Enzian board cost?

The cost is dominated by the DDR4 and the FPGA, but apart from that the unit cost is not that expensive. Expect roughly the cost of a conventional well-populated 2-socket server machine.

Will I be able to partially reconfigure the FPGA from the CPU on the fly?

Yes.

Can I plug a GPU in?

Yes, on either side, if you really must. Enzian has a PCIe Gen3 x8 slot on the ThunderX (CPU) side, and the FPGA has an x16 slot. Both of these can accommodate an NVIDIA Kepler-class GPU card and fit into a 2-U rack-mount case with a suitable riser card.

What about other connectors, like cameras?

In addition to the network, USB, UARTS, NVMe and PCIe on both sides, there’s also an FMC connector pinned out from the FPGA.

What about inter-socket interrupts?

These can be carried over ECI (the “Enzian Coherence Interface”): there are ECI messages for interrupts so the FPGA can raise an interrupt on any core. The CPU side is the standard ARM GIC arrangements. In addition, we have low-speed GPIO pins connected which can bypass ECI but still allow suitable logic on the FPGA to raise an interrupt on the CPU.

Is all the memory cache-coherent between the CPU and FPGA?

The short answer is that it’s up to you, but the full answer is a bit complicated since you should not think of the FPGA as just another processor. Your VHDL or Verilog on the FPGA sees all the coherency messages in ECI, our implementation of the ThunderX’s MOESI-like protocol. We expose these via a set of FIFOs, and handle the flow control, but what you do when you see a cache-line fetch is up to you. In many cases you’ll just want to perform the relevant load from the FPGA’s DDR4 (and we’ll supply Verilog for this), but there are a lot more interesting use-cases.

If all this sounds a bit technical, we will also ship our “shell” for FPGAs with Enzian. It’s called Coyote, and we have an OSDI 2020 paper about it. It abstracts a lot of the CPU-FPGA interface, plus access to DRAM on both sides, at the cost of some flexibility.

What software will run on the CPU side?

We boot Ubuntu Linux 18.04 LTS and (naturally) Barrelfish on the CPU side. Ubuntu 20.04 LTS should be straightforward, as we only have to persuade Linux to ignore the other NUMA node. FreeBSD should also be straightforward and is on our to-do list.

In Linux, there’s a simple kernel driver that maps the whole the FPGA’s section on the system address space into user space, and we’re working on porting the more complex kernel driver for Coyote (see above) to Enzian. For use-cases that really want to treat Enzian as a conventional server with an FPGA on the side, this is fine and we won’t judge you. If you want to go fully heterogeneous and dynamically instantiate first-class cores on the FPGA, Barrelfish may be more to your taste.

What's the form factor?

Enzian is an eATX board, and fits into a standard (large) 2U rack-mount case. With the right riser card, this can also accommodate a double-width GPU plugged into either the CPU or FPGA, if you really want to try this. The coolers on the CPU and FPGA are designed for front-to-back cooling and all the networking connectors appear on the rear I/O panel; NVMe and SATA are on the front of the board and are suitable for rack-mount cases where the drives are at the front.

You should be able to fit 15-20 Enzians in a single standard 19” rack.

Why did you not use a Zynq, or whatever?

There are some nice FPGAs out there from Xilinx and Intel with hard CPU cores. However, they are not realistic processors from a server perspective (nor are they intended to be) – a cluster of four Cortex-A53 cores is typical, and that’s not in the same class as even a mature server part like the ThunderX-1.

Early on we did consider one of these MPSoCs as the FPGA side of the platform, but ultimately our choice of part was driven by the need to maximize transceivers - we went for bandwidth above all else, and this means a straight high-end FPGA.

What's the development environment?

On the software side, Linux is fairly straightforward, and on the FPGA, we support the standard Xilinx Vivado tools (and we’ll make the relevant support files available).

However, we also have a wider range of internal tools like an Enzian simulator which combines a Verilog simulator with a custom module for ARM’s FAST models configured to approximate the Marvell ThunderX-1 processor. We’re working to make these available, but we’re a small team and it takes time…

What other systems are there that look this?

We’re unaware of anything that looks like Enzian, which is why we built it. However, there are other interesting systems in this general space: the Intel HARP platform, for one. There are also a wide variety of platforms that put an FPGA on a PCIe card, including Microsoft’s Catapult, Amazon’s F1 cloud offering, and the latest Xilinx Alveo boards. These are fine if you can live with PCIe. We didn’t want to.

What about OpenCAPI / CCIX / Gen-Z / CXL / PCIe solutions?

We deliberately avoided all these standards (at least, the ones that existed when we started). In some cases, we simply didn’t have access to them, or there was no available silicon. However, the principal reason is that we wanted to eliminate as many prior assumptions that might limit the platform. The CPU’s own coherency protocol itself has many assumptions in it of course, but it’s the closest we can get.

How can I contribute to Enzian?

Talk to us! We’re working on a community site for people to contribute useful bits of software and hardware.

Is Enzian open-source?

We’re releasing all of the software we write for Enzian as open source. It is our goal to also open-source as much of the FPGA doughnut as we can, including our implementation of ECI, which is needed to understand how to talk to the CPU in more interesting ways than just shared memory. We will own the board design, so it may be possible and/or a good idea to release that as well, but (as we have discovered) the hardware industry has very different ideas about what can and cannot be released.

What about ThunderX-2?

ThunderX-2 wasn’t available when we first build the Enzian prototypes, and we wanted a known quantity to work with on the CPU side rather than a brand-new chip (this is the first time we’ve done this, after all…). Moving forward, ThunderX-2 (actually, ThunderX-3) is a much faster processor, we’d love a future revision of Enzian to use it, and we’d like to discuss this with Marvell. ThunderX-2/3 is very different from ThunderX-1, however, and the cache coherence protocols used in ThunderX-2/3 and ThunderX-1 are very different from each other, so this will involve significant re-implementation.

Why is there no high-bandwidth memory on Enzian?

We’d have liked it (and had a few use-cases in mind), but when we started there were no available FPGAs with HBM on-chip, and we simply ran out of transceivers and pins after accommodating ECI, DDR, and the QSFP28 cages. You can plug HMC in, but you won’t get much more bandwidth than NVMe. We had to make a choice between a “slow” interface to HBM, and a full-speed interface to NVMe and PCIe x16 on the FPGA side, and we went for the latter.

Can I use this for machine learning or artificial intelligence?

Yes, if you like. We’ve done work on ML acceleration, in conjunction with our colleague Ce Zhang.

Can I use this for blockchain operations or cryptocurrency mining?

You are not our target audience for this platform.

How can I cite Enzian?

For Enzian as a system, please use the APSLOS paper:

@inproceedings{10.1145/3503222.3507742,
    author = {Cock, David and Ramdas, Abishek and Schwyn, Daniel and Giardino, Michael and Turowski, Adam and He, Zhenhao and Hossle, Nora and Korolija, Dario and Licciardello, Melissa and Martsenko, Kristina and Achermann, Reto and Alonso, Gustavo and Roscoe, Timothy},
    title = {Enzian: An Open, General, CPU/FPGA Platform for Systems Software Research},
    year = {2022},
    isbn = {9781450392051},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3503222.3507742},
    doi = {10.1145/3503222.3507742},
    booktitle = {Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems},
    pages = {434–451},
    numpages = {18},
    keywords = {heterogeneous systems, FPGAs, cache coherence},
    location = {Lausanne, Switzerland},
    series = {ASPLOS 2022}
}