It’s been 46 years since Ethernet was conceived. From its humble beginnings, its impact has been monumental. Ethernet has constantly evolved to meet new demands and now outsells all other local area networks (LAN) by a mile.
Bob Metcalfe had an ingenious and practical idea in 1972 while working at PARC Xerox; he had a vision for an interconnected “Ethernet” (Figure 1). His intention was to provide shared access to a printer from a pool of computers (Figure 2). Though he was a visionary, it seems unlikely that Metcalfe could have imagined the impact his idea would have. As we all know now, Ethernet has in the past 40+ years evolved to be much more than that.
The initial PARC Xerox execution worked on a link speed of 3Mbps on a shared coaxial media. The path to a broad acceptance and mass deployment was secured in 1983 when an IEEE workgroup was set. This workgroup is called 802.3 and is responsible for the standardisation of optical fibre and copper media for various link speeds. Figure 3 shows the standardisation timeline initiated by the 10Mbps link speed in 1983 evolving through 100Mbps, GbE, 10GbE, 40GbE, 100GbE to 400GbE including future 50GbE and 200GbE speeds. Historically, the technologies deployed in the different Ethernet generations have been reused from other standards as with the Fibre Channel for the GbE generation.
Standardisation and Innovation
2010 saw the standardisation of 40GbE. This ushered in the dawn of a new age in the Ethernet ecosystem, as it broke the previously applied “rule” that the next generation link speed was 10x the previous speeds. The 40GbE standard simply deployed the already available 10Gbps transceiver technology from 10GbE on four channels, marking the beginning of the QSFP track QSFP -> QSFP+ -> QSFP28 ->. To learn more about QSFP, see Figure 4.
The development of new 25Gbps transceiver technology became necessary when the four-channel 100GbE standard came along in 2015. This was deployed for the first time with the CFP4 form factor and later in the QSFP28.
Deviating from the previous 10x link speed increase between neighbouring link speed standards constituted a radical shift for IEEE, but this was nothing compared to what was to come. With the introduction of the new 25Gbps transceiver technology, major players representing the full value chain from the device vendors to service providers formed an industry consortium promoting the use of the new 25Gbps transceiver technology in a single channel 25GbE Ethernet standard. They also promote a dual channel 50GbE Ethernet standard (see Figure 5 below).
The clear ambition with the new initiative was to bypass the IEEE bureaucracy and quickly “help themselves” to low-cost technology that, by simple means, would increase the bandwidths of existing datacentre fibre infrastructure. A truly revolutionary move that has never been seen before in the Ethernet “universe” and has been an eye-opener for IEEE.
Future-proof FPGA platforms
The next advancement of the Ethernet ethos was the evolution of FPGA technology in support of FPGA-based network acceleration cards (NACs). This has enabled the design of highly flexible hardware platforms. In turn, this supports a wide swathe of use cases, all with greater longevity.
A good place to start this topic is with a setting of expectations about a future-proof FPGA-based NAC. An NAC hardware platform based on FPGA and designed to be future-proof should support different FPGA size configuration, providing the customer with the right cost/feature ratio options; this enables competitive product offerings. It should also support Ethernet link speeds and types available currently and in the near future, through attractive front port connectivity (see Figure 1).
- Logical resource constraints in the FPGA
- Minimising BOM cost
- FPGA transceiver speed limitations
With regard to the FPGA limitations just mentioned, the PHY device compensates nicely – but it typically restricts the design in the number of supported Ethernet link speeds and potential types.
The 28nm FPGA process node brought with it 25G transceiver technology, but it was only made available in a few high-end device options, available at a substantial cost, compared to the mainstream device options. For good reasons, besides the cost factor, the initial 100G FPGA-based NAC offerings deployed PHY devices implementing the required so-called gearbox functionality and optional error correction functionality. As with the previous PHY application, the gearbox PHY restricts these designs in the number of supported Ethernet link speeds to just 100G.
20nm FPGA families have now been introduced by Xilinx and Intel (formerly Altera), so the FPGA technology is on par with the current and near-term future Ethernet link speeds, making the need for the PHY companion devices obsolete.
From the viewpoint of customers who need multi-link speed, A PHY-less, FPGA-based NAC design poses the following obvious benefits:
- Opportunity to source multiple product variants with the same NAC part number
- Collecting volume on one or few NAC part numbers, pleasing logistics and pricing
- Decreasing the number of required hardware qualification resources
- Limiting of the required knowledge base to one platform
- Ability to introduce multi-link speed product variants, eventually handling all major link speeds and types, on the same ports, through dynamic reconfiguration
Front Port Technology: The Flexible, Pluggable Future
After reviewing Ethernet origins and examining the evolution of future-proof FPGA platforms, we come to the current phase of its evolution: available pluggable front port technology and how it complements flexible properties of future-proof FPGA-based smart NICs.
Front port technology is flexible in these ways:
- Level of backward compatibility
- Number of link types supported
- Number of link rates supported
- Available lane breakout functionality
Multiple versions of IEEE/MSA-defined front port form factors that support link speeds up to 400G are available today. Only a reduced subset of these meet the physical outline limitations imposed by the PCIe standard. With focus on the high-end smart NIC use case, this article will explore the PCIe-compliant multilane-based subset.
Level of Backward Compatibility
If a front port module form factor supports deployment of at least one previous generation of pluggable modules, it is considered backward compatible. The multilane pluggable form factors are currently dominated by two families: the QSFPx family and the CFPx family (see Figures 1 and 2).
A cursory glance would yield the judgement that these two families provide the same functionality. However, a more careful inspection reveals significant differences in their offering, which has justified their co-existence thus far. The CFPx family is targeted at telecom use cases where performance is more important than price and power consumption. Whereas the CFPx family has no backward compatible members, the QSFPx family is fully backward compatible, enabling deployment of all previous pluggable family members, in a given form factor member. The latest addition to the QSFPx family is the QSFP-DD form factor, which brilliantly introduces a doubling of the available number of lanes on the system side without sacrificing backward compatibility.
Link Types Supported
The number of link types supported by a given form factor are determined by market demand and the power envelope of the form factor. The link types supported by the QSFPx form factor family is primarily defined by datacentre and enterprise applications. The SR link type is used massively in this segment, but LR and PSM volume is ramping due to the mega-sizing going on in the datacentres. With a primary focus on the telecom use cases, the CFPx form factor family members have historically provided the space needed to implement the current relevant link types, including the power-consuming long-haul distances.
The gap in the supported link types between the CFPx and the QSFPx family typically increases with each addition of new CFPx family members. Subsequently over time, the gap decreases as the technology allows the QSFPx form-factor to catch up. This is a phenomenon that is usually driven by commercial market demand.
Link Rates Supported
As if it weren’t enough that form factor backward compatibility enables multi-link speed support, but deployment of dual Ethernet rate modules also allows the smart NIC to support several Ethernet link speeds without physically changing the front port pluggable module. As an example, a QSFP28 port will enable deployment of a 100GBASE-SR4 dual-rate module in the following four modes:
- 1x 40G port
- 1x 100G port
- 4x 10G ports, breakout
- 4x 25G ports, breakout
There is a selection of dual-rate offerings available that support the QSFPx form factor family and add further flexibility to this form factor.
Available Lane Breakout Functionality
A standardised multi-lane-based pluggable front port form factor—the QSFP+—became available for the first time when the 40GBASE link-type family was introduced in 2010. This form factor deploys four 10G lanes on the system side in order to handle the 40G aggregate bandwidth on the media side. In a market environment that is constantly struggling to increase port density, it was quickly recognised that this form factor provided the highest available 10G port density at that point in time, resulting in the availability of the first breakout connectivity solutions. Figure 3 shows a breakout solution mapping directly between an SR4 port and four SR ports (MPO/LC), in this case 100G to 25G but also supporting 40G to 10G.
A Flexible Future
Flexibility is a critical asset for ports, and the information above draws the conclusion that the QSFP28 form factor is the most flexible form on the market today. These form factors have evolved just as Ethernet has evolved since its humble beginnings. FPGA technology has adapted to support FPGA-based network acceleration cards, paving the way to design very flexible hardware platforms. The ever-evolving network is well served by the flexibility that front port technology offers.
Claus Ek, hardware development manager, Napatech
Image source: Shutterstock/violetkaipa