Gen-Z Interconnect Core Specification 1.0 Published
The first major release of the Gen-Z systems interconnect specification is now available. The Gen-Z Consortium was publicly announced in late 2016 and has been developing the technology as an open standard, with several drafts released in 2017 for public comment.
Gen-Z is one of several standards that emerged from the long stagnation of the PCI Express standard after the PCIe 3.0 release. Technologies like Gen-Z, CAPI, CCIX and NVLink seek to offer higher throughput, lower latency and the option of cache coherency, in order to enable much higher performance connections between processors, co-processors/accelerators, and fast storage. Gen-Z in particular has very broad ambitions to blur the lines between a memory bus, processor interconnect, peripheral bus and even straying into networking territory.
The Core Specification released today primarily addresses connecting processors to memory, with the goal of allowing the memory controllers in processors to be media-agnostic: the details of whether the memory is some type of DRAM (eg. DDR4, GDDR6) or a persistent memory like 3D XPoint are handled by a media controller at the memory end of a Gen-Z link, while the processor itself issues simple and generic read and write commands over the link. In this use case, Gen-Z doesn’t completely remove the need for traditional on-die memory controllers or the highest-performance solutions like HBM2, but Gen-Z can enable more scalability and flexibility by allowing new memory types to be supported without altering the processor, and by providing access to more banks of memory than can be directly attached to the processor’s own memory controller.
At the lowest level, Gen-Z connections look a lot like most other modern high-speed data links: fast serial links, bonding together multiple lanes to increase throughput, and running a packet-oriented protocol. Gen-Z borrows from both PCI Express and IEEE 802.3 Ethernet physical layer (PHY) standards to offer per-lane speeds up to the 56Gb/s raw speed of 50GBASE-KR, and will track the speed increases from future versions of those underlying standards. The PCIe PHY is incorporated more or less as-is, while the Ethernet PHY standards have been modified to allow for lower power operation when used for shorter links within a single system, such as communication between dies on a multi-chip module. Gen-Z allows for asymmetric links with more links and bandwidth in one direction than the other. The Gen-Z protocol supports various connection topologies like basic point to point links, daisy-chaining, and switched fabrics, including multiple paths of connection between endpoints. Daisy-chain links are estimated to add about 5ns of latency per hop, and switch latencies are expected to be on the order of 10ns for a small 8-port switch up to 50-60ns for a 64-port switch, so using Gen-Z for memory access is reasonable, especially where the somewhat slower persistent memory technologies are concerned. The Gen-Z protocol expresses almost everything in memory terms, but with each endpoint performing its own memory mapping and translation rather than attempting to form a unified single address space across a Gen-Z fabric that could scale beyond a single rack in a data center.
Wide Industry Participation
The Gen-Z Consortium launched with the support of a dozen major technology companies, but its membership has now grown to the point that it is easier to list the big hardware companies who aren’t currently involved: Intel and NVidia. Gen-Z has members from every segment necessary to build a viable product ecosystem: semiconductor design and IP (Mentor, Cadence, PLDA), connectors (Molex, Foxconn, Amphenol, TE), processors and accelerators (AMD, ARM, IBM, Cavium, Xilinx), switches and controllers (IDT, Microsemi, Broadcom, Mellanox), every DRAM and NAND flash memory manufacturer except Intel, software vendors (RedHat, VMWare), system vendors (Lenovo, HPE, Dell EMC). It is clear that most of the industry is paying attention to Gen-Z, even if most of them haven’t yet committed to bringing Gen-Z products to market.
At the SuperComputing17 conference in November, Gen-Z had a multi-vendor demo of four servers sharing access to two pools of memory through a Gen-Z switch. This was implemented with heavy use of FPGAs, but with the Core Specification 1.0 release we will start seeing Gen-Z show up in ASICs. The focus for now is on datacenter use cases with products potentially hitting the market in 2019.
In the meantime, it will be interesting to see where industry support concentrates between Gen-Z and competing standards. Many companies are members or supporters of more than one of the new interconnect standards, and there’s no clear winner at this time. Nobody is abandoning PCI Express, and it isn’t clear which new interconnect will offer the most compelling advantages over the existing ubiquitous standards or over proprietary interconnects. Gen-Z seems to have one of the widest membership bases and the widest target market, but it could still easily be doomed to niche status if it only receives half-hearted support from most of its members.