Data center Archives - EFFECT Photonics

Insights

June 28, 2023

Coherent Lite and The Future Inside the Data Center

In the dynamic landscape of data centers, the demand for greater bandwidth and extended reach…

In the dynamic landscape of data centers, the demand for greater bandwidth and extended reach is rapidly increasing. As shown in the figure below, we can think about three categories of data center interconnects based on their reach

Intra-data center interconnects (< 2km)
Campus data center interconnects (<10km)
Metro data center interconnects (<100km)

Asset 6 data center interconnect — Figure 1: Example of different data center interconnect (DCI) links. Intra-data center interconnects usually have lengths up to 2kms, campus interconnects up to 10km, and metro DCIs up to 100km, although there can be longer DCIs.

Coherent optical technology has already established itself as the go-to solution for interconnecting data centers over long distances in metro areas.

However, within the confines of data centers themselves, intensity-modulated direct detect (IM-DD) technology remains dominant. Recognizing the limitations of IM-DD in meeting evolving requirements, the industry is exploring “Coherent Lite” solutions—a simplified implementation of coherent technology designed specifically for shorter-reach data center connections.

This article delves into the concept of coherent lite technology and its potential to address the escalating bandwidth demands within data centers.

Reducing Dispersion Compensation

The quality of the light signal degrades when traveling through an optical fiber by a process called dispersion. The same phenomenon happens when a prism splits white light into several colors. The fiber also adds other distortions due to nonlinear optical effects. These effects get worse as the input power of the light signal increases, leading to a trade-off. You might want more power to transmit over longer distances, but the nonlinear distortions also become larger, which beats the point of using more power. The DSP performs several operations on the light signal that try to offset these dispersion and nonlinear distortions.

Asset 7 dispersion in optical fibers — Figure 2: Explaining dispersion compensation in optical fibers. A pulse of light will spread out and distort due to dispersion effects.

However, shorter-reach connections require less dispersion compensation, presenting an opportunity to streamline the implementation of coherent solutions. Coherent lite implementations can reduce the use of dispersion compensation blocks. This significantly lowers system power consumption.

The Trade-Offs Between Fixed and Tunable Lasers

Coherent lite solutions also aim to replace tunable lasers with fixed lasers to reduce costs. The use of fixed lasers eliminates the need for wavelength tuning and associated control circuitry and algorithms, simplifying the implementation and reducing operational complexities.

While fixed lasers offer significant advantages, tunable lasers will push to remain competitive. As we described in a previous article, advances in tunable laser technology aim to further reduce package footprints and leverage electronic ecosystems to reduce cost. Such developments will allow tunable lasers to keep pace with the demands of coherent lite solutions, ensuring a viable alternative for shorter-reach data center connections.

Scaling Data Center Links

The increasing length and bandwidth of links within data centers is increasingly calling for the use of coherent technology. As bandwidth scales to 1.6 and 3.2 terabits, traditional direct detect technology faces challenges in keeping up with the growing distances. Intra data center links that were previously limited to 2 kilometers are now extending to 5 or even 10 kilometers, demanding more robust and efficient transmission technologies.

In this context, coherent lite technology provides an attractive middle ground for enabling extended-reach connections within data centers. By leveraging some aspects of coherent solutions, coherent lite technologies facilitate the reliable and efficient transport of data over longer distances.

Takeaways

As data centers evolve to accommodate escalating bandwidth demands, coherent lite technology emerges as a promising solution for communication links within these facilities. By reducing dispersion compensation, simplifying their laser setups, and enabling extended-reach transmission, coherent lite solutions address the limitations of traditional direct detect technology. These advancements pave the way for enhanced performance, and seamless scalability within data center environments.

Article PDF

Tags: 1.6T, 3.2T, bandwidth, coherent, Coherent Lite, Coherent Transmissions, Complexity, CWDM, Data center, datacom, dispersion compensation, Fixed Lasers, IM-DD (Intensity-Modulated Direct Detect), Interconnects, Intra DC, optical technology, power, reach, Scaling Data Center Links, Simplification, Single Wavelength

The Light Path to a Coherent Cloud Edge Banner

Insights

May 18, 2023

Data Center Interconnects: Coherent or Direct Detect?

Article first published 15 June 2022, updated 18 May 2023. With the increasing demand for…

Article first published 15 June 2022, updated 18 May 2023.

With the increasing demand for cloud-based applications, datacom providers are expanding their distributed computing networks. Therefore, they and telecom provider partners are looking for data center interconnect (DCI) solutions that are faster and more affordable than before to ensure that connectivity between metro and regional facilities does not become a bottleneck.

As shown in the figure below, we can think about three categories of data center interconnects based on their reach

Intra-data center interconnects (< 2km)
Campus data center interconnects (<10km)
Metro data center interconnects (<100km)

Figure 1: Example of different data center interconnect (DCI) links. Intra-data center interconnects usually have lengths up to 2kms, campus interconnects up to 10km, and metro DCIs up to 100km, although there can be longer DCIs.

Coherent 400ZR now dominates the metro DCI space, but in the coming decade, coherent technology could also play a role in shorter ranges, such as campus and intra-data center interconnects. As interconnects upgrade to Terabit speeds, coherent technology might start coming closer to direct detect power consumption and cost.

Coherent Dominates in Metro DCIs

The advances in electronic and photonic integration allowed coherent technology for metro DCIs to be miniaturized into QSFP-DD and OSFP form factors. This progress allowed the Optical Internetworking Forum (OIF) to create a 400ZR multi-source agreement. With small enough modules to pack a router faceplate densely, the datacom sector could profit from a 400ZR solution for high-capacity data center interconnects of up to 80km. Operations teams found the simplicity of coherent pluggables very attractive. There was no need to install and maintain additional amplifiers and compensators as in direct detection: a single coherent transceiver plugged into a router could fulfill the requirements.

As an example of their success, Cignal AI forecasted that 400ZR shipments would dominate edge applications, as shown in Figure 2.

Figure 2: Forecast of 100G port equivalents shipped for edge applications. These shipments are overwhelmingly 400ZR standard technology. Source: Cignal AI Transport Applications Report Q42021. This plot is the most recent update that includes this 100G Equivalent Port data, but more updated Cignal AI reports with similar data can be found here. — Figure 2: Forecast of 100G port equivalents shipped for edge applications. These shipments are overwhelmingly 400ZR standard technology. Source: Cignal AI Transport Applications Report Q42021. This plot is the most recent update that includes this 100G Equivalent Port data, but more updated Cignal AI reports with similar data can be found *here*.

Campus Interconnect Are the Grey Area

The campus DCI segment, featuring distances below 10 kilometers, was squarely in the domain of direct detect products when the standard speed of these links was 100Gbps. No amplifiers nor compensators were needed for these shorter distances, so direct detect transceivers are as simple to deploy and maintain as coherent ones.

However, as link bandwidths increase into the Terabit space, these direct detect links will need more amplifiers to reach 10 kilometers, and their power consumption will approach that of coherent solutions. The industry initially predicted that coherent solutions would be able to match the power consumption of PAM4 direct detect solutions as early as 800G generation. However, PAM4 developers have proven resourceful and have borrowed some aspects of coherent solutions without fully implementing a coherent solution. For example, ahead of OFC 2023, semiconductor solutions provider Marvell announced a 1.6Tbps PAM4 platform that pushes the envelope on the cost and power per bit they could offer in the 10 km range.

Following the coming years and how the PAM-4 industry evolves will be interesting. How many (power-hungry) features of coherent solutions will they have to borrow if they want to keep up in upcoming generations and speeds of 3.2 Tbps and beyond? Lumentum’s Chief Technology Officer, Brandon Collings, has some interesting thoughts on the subject in this interview with Gazettabyte.

Direct Detect Dominates Intra Data Center Interconnects (For Now…)

Below Terabit speeds, direct detect technology (both NRZ and PAM-4) will likely dominate the intra-DCI space (also called data center fabric) in the coming years. In this space, links span less than 2 kilometers, and for particularly short links (< 300 meters), affordable multimode fiber (MMF) is frequently used.

Nevertheless, moving to larger, more centralized data centers (such as hyperscale) is lengthening intra-DCI links. Instead of transferring data directly from one data center building to another, new data centers move data to a central hub. So even if the building you want to connect to might be 200 meters away, the fiber runs to a hub that might be one or two kilometers away. In other words, intra-DCI links are becoming campus DCI links requiring their single-mode fiber solutions.

On top of these changes, the upgrades to Terabit speeds in the coming decade will also see coherent solutions more closely challenge the power consumption of direct detect transceivers. PAM-4 direct detect transceivers that fulfill the speed requirements require digital signal processors (DSPs) and more complex lasers that will be less efficient and affordable than previous generations of direct detect technology. With coherent technology scaling up in volume and having greater flexibility and performance, one can argue that it will also reach cost-competitiveness in this space.

Takeaways

Unsurprisingly, using coherent or direct detect technology for data center interconnects boils down to reach and capacity needs. 400ZR coherent is already established as the solution for metro DCIs. In campus interconnects of 10 km or less, PAM-4 products remain a robust solution up to 1.6 Tbps, but coherent technology is making a case for its use. Thus, it will be interesting to see how they compete in future generations and 3.2 Tbps.

Coherent solutions are also becoming more competitive as the intra-data center sector moves into higher Terabit speeds, like 3.2Tbps. Overall, the datacom sector is moving towards coherent technology, which is worth considering when upgrading data center links.

Tags: 800G, access networks, coherent, cost, cost-effective, Data center, distributed computing, edge and metro DCIs, integration, Intra DCI, license, metro, miniaturized, photonic integration, Photonics, pluggable, power consumption, power consumption SFP, reach, Terabit

Pluggables have becomes mainstream in datacom applications.

Insights

February 8, 2023

A Story of Standards: From 400ZR and Open ROADM to OpenZR+

Coherent optical transmission has been crucial in addressing network operator problems for the last decade. In this time, coherent technology has expanded from a solution reserved only for premium long-distance links to one that impacts data center interconnects, metro, and access networks…

Coherent optical transmission has been crucial in addressing network operator problems for the last decade. In this time, coherent technology has expanded from a solution reserved only for premium long-distance links to one that impacts data center interconnects, metro, and access networks, as explained in the video below.

The development of the 400ZR standard by the Optical Internetworking Forum (OIF) proved to be a milestone in this regard. It was the result of several years of progress in electronic and photonic integration that enabled the miniaturization of 400G coherent systems into smaller pluggable form factors. With small enough modules to pack a router faceplate densely, the datacom sector could profit from an ideal solution for high-capacity data center interconnects. Telecom operators wanted to implement a similar solution in their metro links, so they combined the Open ROADM standardization initiative with the 400ZR initiative to develop the OpenZR+ agreement that better fits their use cases.

This article will elaborate on these standardization projects—400ZR, OpenROADM, and OpenZR+—and explain what use cases each was designed to tackle.

What Is 400G ZR?

To cope with growing bandwidth demands, providers wanted to implement the concept of IP over DWDM (IPoDWDM), in which tunable optics are integrated into the router. This integration eliminates the optical transponder shelf and the optics between the routers and DWDM systems, reducing the network capital expenditure (CAPEX). This is shown in the figure below.

Figure 1: Implementation of a traditional coherent transport architecture (left), with the router connected to a transponder line card and a Reconfigurable Optical Add/Drop Multiplexer (ROADM). In contrast, an IPoDWDM solution (right) has 400ZR pluggable transceivers that go directly into the router and use passive multiplexing filters. The IPoDWDM solution eliminates the line card and additional optics, reducing network CAPEX.

However, widely deploying IPoDWDM with coherent optics forced providers to face a router faceplate trade-off. Since DWDM modules have traditionally been much larger than the client optics, plugging DWDM modules into the router required sacrificing roughly half of the costly router faceplate capacity. This was unacceptable for datacom and telecom providers, who approached their suppliers and the Optical Internetworking Forum (OIF) about the need to develop a compact and interoperable coherent solution that addressed this trade-off.

These discussions and technology development led the OIF to publish the 400ZR implementation agreement in 2020. 400ZR specifies a (relatively) low-cost and interoperable 400 Gigabit coherent interface for a link with a single optical wavelength, using a double-polarization, 16-state quadrature amplitude modulation scheme (DP-16QAM). This modulation scheme uses sixteen different constellation points that arise from combining four different intensity levels and four phases. It doubles the usual 16-QAM transmission capacity by encoding information in two different polarizations of light.

The agreement specified a link reach of 80 kilometers without amplification and 120 kilometers with amplification. For forward error correction (FEC), the 400ZR standard supports a concatenated FEC (CFEC) method. CFEC combines inner and outer FEC codes to enhance the performance compared to a standard FEC code.

The 400ZR agreement does not specify a particular size or type of module, but its specifications targeted a footprint and power consumption that could fit in smaller modules such as the Quad Small Form-Factor Pluggable Double-Density (QSFP-DD) and Octal-Small Form-Factor Pluggable (OSFP). These form factors are small enough to provide the faceplate density that telecom and especially datacom operators need in their system architectures. So even if we often associate the 400ZR standard with QSFP-DD, other form factors, such as CFP2, can be used.

What Is Open ROADM?

In parallel with the 400ZR standardization efforts, telecom network operators had a different ongoing discussion.

Reconfigurable Optical Add-Drop Multiplexers (ROADMs) were a game-changer for optical communications when they entered the market in the 2000s. Before this technology, optical networks featured inefficient fixed routes and could not adapt to changes in traffic and demand. ROADMs allowed operators to remotely provision and manage their wavelength channels and bandwidth without redesigning the physical network infrastructure.

However, ROADMs were proprietary hardware with proprietary software. Changing the proprietary ROADM platform needed extensive testing and a lengthy integration process, so operators were usually reluctant to look for other platform alternatives. Besides, ROADMs still had several fixed, pre-defined elements that could have been configurable through open interfaces. This environment led to reduced competition and innovation in the ROADM space.

These trends drove the launch of the Open ROADM project in 2016 and the release of their first Multi-Source Agreement in 2017. The project aimed to disaggregate and open up these traditionally proprietary ROADM systems and make their provisioning and control more centralized through technologies such as software-defined networks (SDNs, explained in the diagram below).

Figure 2: In a traditional network paradigm (left), switches contain both the hardware that forwards traffic (the data plane), as well as the software that sets the rules of where to forward said traffic (the control plane). The SDN paradigm (right) separates the switching hardware from the software, effectively decoupling the data plane from the control plane. The language between this controller and the switches is an open protocol, with OpenFlow being the most commonly used one. The Open ROADM initiative proposed its own open protocol called NETCONF.

The Open ROADM project defined three disaggregated functions (pluggable optics, transponder, and ROADM), all controlled through an open standards-based API that could be accessed through an SDN controller. It defined 100G-400G interfaces for both Ethernet and Optical Transport Networking (OTN) protocols with a link reach of up to 500km. It also defined a stronger FEC algorithm called open FEC (oFEC) to support this reach. oFEC provides a greater enhancement than CFEC at the cost of more overhead and energy.

What Is OpenZR+?

The 400ZR agreement was primarily focused on addressing the needs of large-scale data center operators and their suppliers.

While it had some usefulness for telecom network operators, their transport network links usually span several hundreds of kilometers, so the interface and module power consumption defined in the 400ZR agreement could not handle such an extended reach. Besides, network operators needed extra flexibility when defining the transmission rate and the modulation type of their links.

Therefore, soon after the publication of the 400ZR agreement, the OpenZR+ Multi-Source Agreement (MSA) was published in September 2020. As the diagram below explains, this agreement can be seen as a combination of the 400ZR and Open ROADM standardization efforts.

Figure 3: Diagram of how different elements from the 400ZR and Open ROADM standardization projects were combined to form the OpenZR+ MSA. Diagram provided by the OpenZR+ website.

To better fit the telecom use cases of regional and long-haul transport links, OpenZR+ added a few changes to improve the link reach and flexibility over 400ZR:

Using the more powerful oFEC defined by the Open ROADM standard.
Multi-rate Ethernet that enables the multiplexing of 100G and 200G signals. This provides more options to optimize traffic in transport links.
Support for 100G, 200G, 300G, or 400G transport links using different modulation types (QPSK, 8QAM, or 16QAM). This enables further reach and capacity optimization for fiber links.
Higher dispersion compensation to make the fiber link more robust.

These changes allow QSFP-DD and OSFP modules to reach link lengths of up to 480 km (with optical amplifiers) at a 400G data rate. However, the FEC and dispersion compensation improvements that enable this extended reach come at the price of increased energy consumption. While the 400ZR standard targets a power consumption of 15 Watts, OpenZR+ standards aim for a power consumption of up to 20 Watts.

If operators need more performance, distances above 500 km, and support for OTN traffic (400ZR and OpenZR+ only support Ethernet), they must still use proprietary solutions, which are informally called 400ZR+. These 400ZR+ solutions feature larger module sizes (CFP2), higher performance proprietary FEC, and higher launch powers to achieve longer reach. These more powerful features come at the cost of even more power consumption, which can go up to 25 Watts.

Takeaways

The following table summarizes the use cases and characteristics of the approaches discussed in the article: 400ZR, Open ROADM, OpenZR+, and 400ZR+.

Technology	400ZR	Open ROADM	OpenZR+	400ZR+ Proprietary
Target Application	Edge Data Center Interconnects	Carrier ROADM Mesh Networks	Metro/Regional Carrier and Data Center Interconnects	Long-Haul Carrier
Target Reach @ 400G	120 km (amplified)	500 km (amplified)	480km (amplified)	1000 km (amplified)
Target Power Consumption	Up to 15 W	Up to 25 W	Up to 20 W	Up to 25W
Typical Module Option	QSFP-DD/OSFP	CFP2	QSFP-DD/OSFP	CFP2
Client Interface	400G Ethernet	100-400G Ethernet and OTN	100-400G Ethernet (Multi-rate)	100-400G Ethernet and OTN
Modulation Scheme	16QAM	QPSK, 8QAM, 16QAM	QPSK, 8QAM, 16QAM	QPSK, 8QAM, 16QAM
Forward Error Correction	CFEC	oFEC	oFEC	Proprietary
Standards / MSA	OIF	Open ROADM MSA	OpenZR+ MSA	Proprietary

400ZR is an agreement primarily focused on the needs of data center interconnects across distances of 80 – 120 kilometers. On the other hand, OpenROADM and OpenZR+ focused on addressing the needs of telecom carrier links, supporting link lengths of up to 500 km. These differences in reach are also reflected in the power consumption specs and the module form factors typically used. The 400ZR and OpenZR+ standards can only handle Ethernet traffic, while the Open ROADM and 400ZR+ solutions can handle both Ethernet and OTN traffic.

Tags: 400ZR, Data center, demand, disaggregation, extend, fiber, interoperable, Open ROADM, power, size, ZR+

Telecom/Datacom

Emerging Industries

DSP & FEC

Ultra Pure Light Sources

Ecosystems

Pluggable Modules – Direct Detect

Pluggable Modules – Coherent

Sub-Assemblies

ASICs

View All

Insights

Press Releases

In the News

Events

Careers

Knowledge Center

Company Info

Contact Us

Telecom/Datacom

Emerging Industries

DSP & FEC

Ultra Pure Light Sources

Ecosystems

Pluggable Modules – Direct Detect

Pluggable Modules – Coherent

Sub-Assemblies

ASICs

View All

Insights

Press Releases

In the News

Events

Careers

Knowledge Center

Company Info

Contact Us