Keeping Up with Hyperscale Data Center Connectivity Needs

Our world is being revolutionized by migration of computing power to “the Cloud” — the hyperscale data centers (HDCs) of internet giants like Google, Microsoft, Facebook, Amazon, and Apple. The scale of HDCs and their use of leading-edge technology gives them performance and cost advantages over even the highest-end enterprise networks. Enterprises are increasingly outsourcing their IT infrastructure to the Cloud to harness dynamically scalable best-in-class computing resources while avoiding investments in their own rapidly obsolescing data centers.


Typical current HDCs connect server computers via a multistage network with top-of-rack (TOR), leaf, and spine switches (Figure 1). While the short server-to-TOR switch connections are dominantly copper, the others are fiber-optic. Given rapid increases in data rates and distances as data centers increase in physical size, the fiber-optic connections are increasingly over single-mode (SM) fiber. Optical transceivers are the key components in these fiber-optic connections.

Figure 1. Typical HDC network architecture showing SM fibers connecting the levels of the switch hierarchy


Electronic ICs in servers and switches double in processing power every 18 months according to Moore’s Law, driving the need for comparable data rate improvements in the transceivers for DC optical interconnects. Cost and power consumption are vital factors in HDCs so transceivers must continually improve in normalized cost ($/Gbps) and power (watts/Gbps). For a state-of-the-art 3.2Tbps switch populated with 32 100-Gigabit Ethernet (100GE) transceivers, the 32 transceivers currently cost ~4x more than the rest of the switch, highlighting the importance of optical transceiver costs. Leveraging photonic integrated circuit (PIC) technologies is a key to meeting these cost and performance challenges.


Optical Transceivers

Current state-of-the-art data center optical interconnects carry 100GE inverse multiplexed over four 25Gbps NRZ optical links. As electronics advance, speed/power/cost optimization will soon favor 50 Gbps PAM4 and 100Gbps PAM4 over 25Gbps NRZ. Each of these evolutionary technology steps comes with IC, optical component, and system integration/packaging challenges.


Figure 2 shows the architecture of single WDM transceiver. The laser diode is made from InP; for a directly-modulated laser (DML), no separate modulator is needed. The modulator may be made from InP (monolithically integrated with the laser) or silicon photonics (SiPh). The receiver’s photodetector is made from InP or may be SiGe for SiPh PICs (Figure 3). The highly temperature-dependent refractive index (dn/dT), small mode size, and high optical loss of InP and SiPh PICs make them poor choices for implementing a wavelength mux or demux. By contrast, silica planar lightwave circuits (PLCs) do not suffer these impairments, yet leverage the cost and scale advantages of standard IC processing technologies. Unless the laser, modulator, and mux are all made from InP, a WDM transceiver requires hybrid optical integration (HOI) of multiple PIC technologies.


Figure 2. Architecture of transmit path of a WDM transceiver


Figure 3. Architecture of receive path of a WDM transceiver


Hybrid Optical Integration (HOI)

High-performance HOI requires: (1) low optical coupling losses; (2) compactness; (3) ruggedness; (4) low cost; (5) scalability to high-volume manufacturing; (6) scalability to high parallelism; and (7) flexible hybrid integration of different optical technologies. This long list of difficult requirements explains why HOI has been impeded for decades by lack of a suitable solution.


Achieving low optical coupling losses in SM transceivers is particularly challenging. Coupling from an InP laser (typical mode diameter ~2μm) to a SM waveguide or fiber with coupling losses of < 3dB requires alignment tolerances of < 1μm (Figure 4). This is far beyond the capabilities of standard electronic packaging technologies.


Figure 4. Optical coupling loss for a 2μm mode diameter vs. transverse misalignment


Mode size mismatch can also cause significant optical coupling losses. Butt coupling a 2μm mode to an ~8μm mode (typical of a silica waveguide or fiber) incurs > 6dB coupling loss. This mode mismatch loss can be reduced or eliminated by using a mode matching structure such as a lens or adiabatic taper to couple the different mode sizes.


Kaiam’s Optical Wire Bond (OWB™) technology finally provides a high-performance HOI solution. In OWB, one or more microlenses are mounted to micro-electromechanical systems (MEMS) platforms, as shown in Figure 5. Using simple equipment, the mechanical de-multiplication of the MEMS lever arm enables the microlenses to be automatically positioned in three dimensions with an accuracy of < 0.5μm while matching mode sizes. The MEMS platforms are then soldered in place by an integrated MEMS heater. OWB supports simultaneous coupling of four, eight, or more channels with coupling losses of < 2dB in a compact, rugged, low-cost assembly. This high parallelism combined with high performance is a key to meeting future transceiver needs.


Kaiam Optical Wirebond
Figure 5. Optical Wire Bond (OWB) coupling InP laser diodes to silica PLCs

The Future…

Moving forward, insatiable HDC bandwidth demands will require some combination of higher per-channel data rates, wavelength counts, and fiber counts. An optimal HOI solution, combined with a high-performance optical mux, demux and other optical components, will enable optical transceivers to keep up with these demands.


400Gbps pluggable modules will be needed by 2018 and 1.6Tbps optical interconnects will be needed just a few years later. Pluggables may give way to on-board optics or optics co-packaged with the switch chip ICs to improve density, power consumption, and cost. The same key technologies, including OWB and silica PLCs, are extensible to these higher speeds and new packaging architectures, and thus can meet the optical interconnect requirements of HDCs for the foreseeable future.

Rob Kalman

Dr. Rob Kalman has 20+ years of experience in successfully delivering leading edge optical communications technology and products. He was a founding member of Luminous Networks, ONI Systems, and Optivision, where he held various leadership roles in Engineering and Marketing. He has a Ph.D. in EE from Stanford.