Insights
The Evolution of Data Center Interconnects
The digital era’s rapid expansion requires advances in data center interconnects (DCIs) to support the burgeoning demands of cloud computing and data architecture.
For the sake of this article, let’s think broadly about three categories of data center interconnects based on their reach and location with relation to the data center
- Intra-data center interconnects (< 2km)
- Campus data center interconnects (<10km)
- Metro data center interconnects (<100km)
As data centers become more complex and AI increases its demands on them, the intra-data center sector is increasing in complexity and variety, but that will be the subject of a different article.
Coherent optical technology has established itself as the go-to solution for interconnecting data centers over longer distances, while direct detect continues to dominate the intra data center sector.
The Increasing Importance of Decentralizing Data Centers
Smaller data centers placed locally have the potential to minimize latency, overcome inconsistent connections, and store and compute data closer to the end-user. These benefits are causing the global market for edge data centers to explode, with PWC predicting that it will nearly triple from $4 billion in 2017 to $13.5 billion in 2024.
Meanwhile, the demands of new AI infrastructure are pushing data center power consumption to such a degree that the electrical power grid might be unable to sustain it. Longer data center interconnects enable a more decentralized system of data centers with branches in different geographical areas connected through high-speed optical fiber links to cope with the strain of data center clusters on power grids.
These trends push the data center industry to look for interoperable solutions for longer interconnects over distances of 80 to 120 km. The advances in electronic and photonic integration allowed coherent technology for metro DCIs to be miniaturized into QSFP-DD and OSFP form factors. This progress allowed the Optical Internetworking Forum (OIF) to create the 400ZR and ZR+ standards for 400G DWDM pluggable modules. With small enough modules to pack a router faceplate densely, the datacom sector could profit from a 400ZR solution for high-capacity data center interconnects of up to 80km.
After the success of 400ZR standardization, the data center industry and the OIF are starting to promote an 800ZR standard to enable the next generation of interconnects. In OFC 2024, we started seeing some demos from several vendors and the OIF on this new standards initiative.
Direct Detect and Coherent Technology in DCIs
For many years, there has been an expectation that the increasing capacity demands on data centers would reach a point where coherent technology would deliver a lower cost per bit and power per bit than direct detect technologies.
However, direct detect technology (both NRZ and PAM-4) continues to successfully overcome the challenges of coherent technology and will continue to dominate the intra-DCI space (also called data center fabric) in the coming years. In this space, links span less than two kilometers, and for particularly short links (< 300 meters), affordable multimode fiber (MMF) is frequently used.
The tendency towards data center decentralization will also impact the inside of data centers. Larger data centers can require longer interconnects from one building to another. So in a way, some links inside the data centers could become more like campus DCI links, which require single-mode fiber solutions, and in those spaces coherent might have a better chance of becoming competitive.
Takeaways
The decision to use coherent or direct detection technology for DCIs boils down to the reach and capacity needs. Coherent is already established as the solution for metro DCIs, and efforts are underway towards an 800ZR standard to follow up on the highly successful 400ZR interconnect standard. With the move to Terabit speeds and scaling production volumes, it was expected to become competitive in the data center sector, too, but for now, direct detect technology continues to dominate this sector.
From basic networking to sophisticated, AI-enhanced architectures, DCIs have become the backbone of the digital infrastructure, enabling the seamless operation of global cloud services and data centers.