Data center Archives - EFFECT Photonics

Pluggables have becomes mainstream in datacom applications.

Insights

February 8, 2023

A Story of Standards: From 400ZR and Open ROADM to OpenZR+

Coherent optical transmission has been crucial in addressing network operator problems for the last decade. In this time, coherent technology has expanded from a solution reserved only for premium long-distance links to one that impacts data center interconnects, metro, and access networks…

The development of the 400ZR standard by the Optical Internetworking Forum (OIF) proved to be a milestone in this regard. It was the result of several years of progress in electronic and photonic integration that enabled the miniaturization of 400G coherent systems into smaller pluggable form factors. With small enough modules to pack a router faceplate densely, the datacom sector could profit from an ideal solution for high-capacity data center interconnects. Telecom operators wanted to implement a similar solution in their metro links, so they combined the Open ROADM standardization initiative with the 400ZR initiative to develop the OpenZR+ agreement that better fits their use cases.

This article will elaborate on these standardization projects—400ZR, OpenROADM, and OpenZR+—and explain what use cases each was designed to tackle.

What Is 400G ZR?

To cope with growing bandwidth demands, providers wanted to implement the concept of IP over DWDM (IPoDWDM), in which tunable optics are integrated into the router. This integration eliminates the optical transponder shelf and the optics between the routers and DWDM systems, reducing the network capital expenditure (CAPEX). This is shown in the figure below.

Figure 1: Implementation of a traditional coherent transport architecture (left), with the router connected to a transponder line card and a Reconfigurable Optical Add/Drop Multiplexer (ROADM). In contrast, an IPoDWDM solution (right) has 400ZR pluggable transceivers that go directly into the router and use passive multiplexing filters. The IPoDWDM solution eliminates the line card and additional optics, reducing network CAPEX.

However, widely deploying IPoDWDM with coherent optics forced providers to face a router faceplate trade-off. Since DWDM modules have traditionally been much larger than the client optics, plugging DWDM modules into the router required sacrificing roughly half of the costly router faceplate capacity. This was unacceptable for datacom and telecom providers, who approached their suppliers and the Optical Internetworking Forum (OIF) about the need to develop a compact and interoperable coherent solution that addressed this trade-off.

These discussions and technology development led the OIF to publish the 400ZR implementation agreement in 2020. 400ZR specifies a (relatively) low-cost and interoperable 400 Gigabit coherent interface for a link with a single optical wavelength, using a double-polarization, 16-state quadrature amplitude modulation scheme (DP-16QAM). This modulation scheme uses sixteen different constellation points that arise from combining four different intensity levels and four phases. It doubles the usual 16-QAM transmission capacity by encoding information in two different polarizations of light.

The agreement specified a link reach of 80 kilometers without amplification and 120 kilometers with amplification. For forward error correction (FEC), the 400ZR standard supports a concatenated FEC (CFEC) method. CFEC combines inner and outer FEC codes to enhance the performance compared to a standard FEC code.

The 400ZR agreement does not specify a particular size or type of module, but its specifications targeted a footprint and power consumption that could fit in smaller modules such as the Quad Small Form-Factor Pluggable Double-Density (QSFP-DD) and Octal-Small Form-Factor Pluggable (OSFP). These form factors are small enough to provide the faceplate density that telecom and especially datacom operators need in their system architectures. So even if we often associate the 400ZR standard with QSFP-DD, other form factors, such as CFP2, can be used.

What Is Open ROADM?

In parallel with the 400ZR standardization efforts, telecom network operators had a different ongoing discussion.

Reconfigurable Optical Add-Drop Multiplexers (ROADMs) were a game-changer for optical communications when they entered the market in the 2000s. Before this technology, optical networks featured inefficient fixed routes and could not adapt to changes in traffic and demand. ROADMs allowed operators to remotely provision and manage their wavelength channels and bandwidth without redesigning the physical network infrastructure.

However, ROADMs were proprietary hardware with proprietary software. Changing the proprietary ROADM platform needed extensive testing and a lengthy integration process, so operators were usually reluctant to look for other platform alternatives. Besides, ROADMs still had several fixed, pre-defined elements that could have been configurable through open interfaces. This environment led to reduced competition and innovation in the ROADM space.

These trends drove the launch of the Open ROADM project in 2016 and the release of their first Multi-Source Agreement in 2017. The project aimed to disaggregate and open up these traditionally proprietary ROADM systems and make their provisioning and control more centralized through technologies such as software-defined networks (SDNs, explained in the diagram below).

Figure 2: In a traditional network paradigm (left), switches contain both the hardware that forwards traffic (the data plane), as well as the software that sets the rules of where to forward said traffic (the control plane). The SDN paradigm (right) separates the switching hardware from the software, effectively decoupling the data plane from the control plane. The language between this controller and the switches is an open protocol, with OpenFlow being the most commonly used one. The Open ROADM initiative proposed its own open protocol called NETCONF.

The Open ROADM project defined three disaggregated functions (pluggable optics, transponder, and ROADM), all controlled through an open standards-based API that could be accessed through an SDN controller. It defined 100G-400G interfaces for both Ethernet and Optical Transport Networking (OTN) protocols with a link reach of up to 500km. It also defined a stronger FEC algorithm called open FEC (oFEC) to support this reach. oFEC provides a greater enhancement than CFEC at the cost of more overhead and energy.

What Is OpenZR+?

The 400ZR agreement was primarily focused on addressing the needs of large-scale data center operators and their suppliers.

While it had some usefulness for telecom network operators, their transport network links usually span several hundreds of kilometers, so the interface and module power consumption defined in the 400ZR agreement could not handle such an extended reach. Besides, network operators needed extra flexibility when defining the transmission rate and the modulation type of their links.

Therefore, soon after the publication of the 400ZR agreement, the OpenZR+ Multi-Source Agreement (MSA) was published in September 2020. As the diagram below explains, this agreement can be seen as a combination of the 400ZR and Open ROADM standardization efforts.

Figure 3: Diagram of how different elements from the 400ZR and Open ROADM standardization projects were combined to form the OpenZR+ MSA. Diagram provided by the OpenZR+ website.

To better fit the telecom use cases of regional and long-haul transport links, OpenZR+ added a few changes to improve the link reach and flexibility over 400ZR:

Using the more powerful oFEC defined by the Open ROADM standard.
Multi-rate Ethernet that enables the multiplexing of 100G and 200G signals. This provides more options to optimize traffic in transport links.
Support for 100G, 200G, 300G, or 400G transport links using different modulation types (QPSK, 8QAM, or 16QAM). This enables further reach and capacity optimization for fiber links.
Higher dispersion compensation to make the fiber link more robust.

These changes allow QSFP-DD and OSFP modules to reach link lengths of up to 480 km (with optical amplifiers) at a 400G data rate. However, the FEC and dispersion compensation improvements that enable this extended reach come at the price of increased energy consumption. While the 400ZR standard targets a power consumption of 15 Watts, OpenZR+ standards aim for a power consumption of up to 20 Watts.

If operators need more performance, distances above 500 km, and support for OTN traffic (400ZR and OpenZR+ only support Ethernet), they must still use proprietary solutions, which are informally called 400ZR+. These 400ZR+ solutions feature larger module sizes (CFP2), higher performance proprietary FEC, and higher launch powers to achieve longer reach. These more powerful features come at the cost of even more power consumption, which can go up to 25 Watts.

Takeaways

The following table summarizes the use cases and characteristics of the approaches discussed in the article: 400ZR, Open ROADM, OpenZR+, and 400ZR+.

Technology	400ZR	Open ROADM	OpenZR+	400ZR+ Proprietary
Target Application	Edge Data Center Interconnects	Carrier ROADM Mesh Networks	Metro/Regional Carrier and Data Center Interconnects	Long-Haul Carrier
Target Reach @ 400G	120 km (amplified)	500 km (amplified)	480km (amplified)	1000 km (amplified)
Target Power Consumption	Up to 15 W	Up to 25 W	Up to 20 W	Up to 25W
Typical Module Option	QSFP-DD/OSFP	CFP2	QSFP-DD/OSFP	CFP2
Client Interface	400G Ethernet	100-400G Ethernet and OTN	100-400G Ethernet (Multi-rate)	100-400G Ethernet and OTN
Modulation Scheme	16QAM	QPSK, 8QAM, 16QAM	QPSK, 8QAM, 16QAM	QPSK, 8QAM, 16QAM
Forward Error Correction	CFEC	oFEC	oFEC	Proprietary
Standards / MSA	OIF	Open ROADM MSA	OpenZR+ MSA	Proprietary

400ZR is an agreement primarily focused on the needs of data center interconnects across distances of 80 – 120 kilometers. On the other hand, OpenROADM and OpenZR+ focused on addressing the needs of telecom carrier links, supporting link lengths of up to 500 km. These differences in reach are also reflected in the power consumption specs and the module form factors typically used. The 400ZR and OpenZR+ standards can only handle Ethernet traffic, while the Open ROADM and 400ZR+ solutions can handle both Ethernet and OTN traffic.

Tags: 400ZR, Data center, demand, disaggregation, extend, fiber, interoperable, Open ROADM, power, size, ZR+

The Light Path to a Coherent Cloud Edge Banner

Insights

June 15, 2022

Data Center Interconnects: Coherent or Direct Detect?

With the increasing demand for cloud-based applications, datacom providers are pushing forward with expanding their…

With the increasing demand for cloud-based applications, datacom providers are pushing forward with expanding their distributed computing networks. Therefore, they and telecom provider partners are looking for data center interconnect (DCI) solutions that are faster and more affordable than before to ensure that connectivity between metro and regional facilities does not become a bottleneck.

Energy usage, space, simplicity, and cost-effectiveness all impact the efficiency of DCI infrastructure. These solutions must consider watts per bit, rack space, and simplified provisioning and operating expenditure. Previously, direct detect technology could fulfill these requirements for short-reach DCIs inside data centers and campuses. However, achieving the reach and bandwidths required for edge and metro DCIs required external amplifiers and dispersion compensators that increased the cost and complexity of network operations.

Figure 1: Example of different data center interconnect (DCI) links. Intra-data center interconnects usually have lengths up to 2kms, campus interconnects up to 10km, and metro DCIs up to 100km, although there can be longer DCIs.

At the same time, advances in electronic and photonic integration allowed longer reach coherent technology to be miniaturized into QSFP-DD and OSFP form factors. This enabled the transport of 100G and 400G connections over a single wavelength and several hundreds of kilometers, which is ideal for edge and metro DCI networks. Provider operations teams found the simplicity of coherent pluggables very attractive. There was no need to install and maintain additional amplifiers and compensators as in direct detect: a single coherent transceiver plugged into a router could fulfill the requirements.

In the coming decade, the shorter-reach DCI links will also require upgrades to 400G, 800G, and Terabit speeds, and at those speeds, coherent technology comes close to matching the energy consumption of direct detect. This would make it competitive even for shorter links.

Coherent Dominates in Metro DCIs

The advances in electronic and photonic integration allowed coherent technology for metro DCIs to be miniaturized into QSFP-DD and OSFP form factors. This progress allowed the Optical Internetworking Forum (OIF) to create the 400ZR and ZR+ standards for 400G DWDM pluggable modules. With small enough modules to pack a router faceplate densely, the datacom sector could profit from a 400ZR solution for high-capacity data center interconnects of up to 80km. If needed, extended reach 400ZR+ pluggables can cover several hundreds of kilometers. As an example of their success, Cignal AI forecasts that 400ZR shipments will dominate in the edge applications, as shown in Figure 3.

Figure 2: Forecast of 100G port equivalents shipped for edge applications. These shipments are overwhelmingly 400ZR standard technology. Source: Cignal AI Transport Applications Report Q42021.

Further improvements in integration can further boost the reach and efficiency of coherent transceivers. For example, by integrating all photonic functions on a single chip, including lasers and optical amplifiers, EFFECT Photonics’ photonic System-On-Chip (SoC) technology can achieve higher transmit power levels and longer distances while keeping the smaller QSFP-DD form factor, power consumption, and cost.

Campus DCI Is The Battleground of Direct Detect and Coherent

The campus DCI segment, featuring distances below ten kilometers, was squarely the domain of direct detect products when the standard speed of these links was 100Gbps. No amplifiers nor compensators were needed for these shorter distances, so direct detect transceivers are as simple to deploy and maintain as coherent ones. However, at 400Gbps speeds, the power consumption of coherent technology is much closer to that of direct detect PAM-4 solutions.

This gap in power consumption is expected to disappear at 800Gbps, as shown in the figure below. For Terabit speeds, the prediction is that coherent transceivers will be more efficient. Furthermore, as the volume production of coherent transceivers increases, their price will also become competitive with direct detect solutions. Overall, coherent transceivers are expected to scale up better in future upgrades.

Figure 3. Coherent DSP power consumption is getting closer to that of PAM-4 direct detect solutions as link speed increases and will match it at 800G. Source: NeoPhotonics Blog

Direct Detect Dominates Intra Data Center Interconnects (For Now…)

Below Terabit speeds, direct detect technology (both NRZ and PAM-4) will likely dominate the intra-DCI space (also called data center fabric) in the coming years. In this space, links span less than two kilometers, and for particularly short links (< 300 meters), affordable multimode fiber (MMF) is frequently used.

Nevertheless, moving to larger, more centralized data centers (such as hyperscale data centers) is lengthening intra-DCI links. Instead of transferring data directly from one data center building to another, new data centers first move data to a central hub. So even if the building you want to connect to might be 200 meters away, the fiber runs to a hub that might be one or two kilometers away. In other words, intra-DCI links are becoming campus DCI links, which requires their single-mode fiber solutions.

On top of these changes, the upgrades to Terabit speeds in the coming decade will also see coherent solutions challenge the power consumption of direct detect transceivers. PAM-4 direct detect transceivers that fulfill the speed requirements require digital signal processors (DSPs) and more complex lasers that will be less efficient and affordable than previous generations of direct detect technology. With coherent technology scaling up in volume and having greater flexibility and performance, one can make the argument that it will reach cost-competitiveness in this space, too.

Takeaways

Unsurprisingly, the decision of using coherent or direct detect technology for DCIs boils down to the reach and capacity needs. Coherent is already established as the solution for metro DCIs and is already gaining ground in the campus DCI segment for 800G and Terabit speeds. With the move to Terabit speeds and scaling production volumes, it could also become cost-competitive inside the data center too. Overall, the datacom sector is moving towards coherent technology, and it pays off to have this in mind when upgrading data center links.

Tags: 800G, access networks, coherent, cost, cost-effective, Data center, distributed computing, edge and metro DCIs, integration, Intra DCI, license, metro, miniaturized, photonic integration, Photonics, pluggable, power consumption, power consumption SFP, reach, Terabit

Telecom/Datacom

Emerging Industries

DSP & FEC

Ultra Pure Light Sources

Ecosystems

Pluggable Modules – Direct Detect

Pluggable Modules – Coherent

Sub-Assemblies

ASICs

View All

Insights

News

Events

Careers

Knowledge Center

Company Info

Contact Us

Telecom/Datacom

Emerging Industries

DSP & FEC

Ultra Pure Light Sources

Ecosystems

Pluggable Modules – Direct Detect

Pluggable Modules – Coherent

Sub-Assemblies

ASICs

View All

Insights

News

Events

Careers

Knowledge Center

Company Info

Contact Us