English
InfiniBand is an open-standard, high-speed, low-latency interconnect technology designed for high-performance computing (HPC) and data centers. With the rise of AI, it has become the preferred networking solution for GPU servers in high-performance environments.
I. InfiniBand Specifications
InfiniBand has evolved through 9 major performance tiers, from the initial SDR (10Gbps) to the latest XDR (800Gbps), continuously advancing in bandwidth, latency, and energy efficiency for modern supercomputing and AI clusters.
Currently, HDR (200Gb/s) and NDR (400Gb/s) represent the most widely deployed generations in AI-related supercomputers. NVIDIA officially launched its XDR-based product family in 2025. Due to ongoing ecosystem and supply chain maturation, XDR is currently used only by a handful of leading enterprises for small-scale testing and validation. Mass commercialization is expected post-2026.
II. InfiniBand HDR Optical Modules
HDR (High Data Rate), the sixth-generation InfiniBand standard, delivers a theoretical single-port bandwidth of 200 Gb/s, using QSFP56 form factor. It supports 4-channel parallel transmission with each channel employing 50G PAM4 modulation technology, offering high density and energy efficiency.
2.1 HDR QSFP56 SR4 Module
- Application: Connects NVIDIA Quantum switches and ConnectX-6 NICs.
- Optical Interface: MPO/MTP-12 UPC connector.
- Distance: 70m with OM3 multimode fiber; 100m with OM4/OM5 fiber.
- Use Case: Short-distance interconnects within racks (e.g., server-to-switch links).
2.2 HDR QSFP56 FR4 Module
- Application: Connects NVIDIA Quantum switches, but with an LC-Duplex UPC optical interface.
- Optical Interface: Single-mode fiber (OS2).
- Distance: Up to 2km, ideal for long-distance connections across rooms or buildings.
- Use Case: Aggregation/core-layer switch interconnects.
- Application: NVIDIA ConnectX-7 NICs.
- Interface: MPO/MTP-12 APC.
- Distance: 30m (OM3 MMF); 50m (OM4/OM5 MMF).
- Use Case: Short-distance interconnects between servers and leaf switches within cabinets
- Application: ConnectX-7 NICs.
- Interface: MPO/MTP-12 APC (single-mode OS2 fiber).
- Distance: Up to 500m.
- Use Case: Medium-to-short-distance server-to-leaf links in large rack clusters.
- Application: NVIDIA Quantum-2 switches.
- Interface: Dual MPO/MTP-12 APC (dual-lane SR4).
- Bandwidth: 800Gbps total.
- Distance: 30m (OM3); 50m (OM4/OM5).
- Use Case: Primarily used for short-distance, high-speed interconnects between leaf nodes and spine switches within cabinets (leaf-to-spine).
- Application: Quantum-2 switches.
- Interface: Dual MPO/MTP-12 APC (OS2 SMF).
- Distance: Up to 500m.
- Use Case: Suitable for medium-to-short leaf-to-spine connections within a single data center, supporting larger-scale Fat-Tree topologies.
- Application: Designed for cross-room or campus-wide deployments.
- Interface: LC-Duplex UPC (OS2 SMF).
- Distance: 2km.
- Use Case: Specifically engineered for spine-to-core remote backbone interconnections, ensuring efficient data synchronization and low-latency communication between multi-room AI clusters.
III. InfiniBand NDR Optical Modules
NDR (Next Data Rate), the seventh-generation InfiniBand standard, delivers 400Gb/s per port (800Gb/s bidirectional), marking InfiniBand's entry into the 400G/800G era. It supports advanced modulation and new packaging protocols, meeting the demands of AI training clusters for extreme bandwidth and ultra-low latency.
3.1 400G Optical Modules
3.1.1 NDR OSFP SR4 & NSR QSFP112 SR4
3.1.2 NDR OSFP DR4 & NSR QSFP112 DR4
3.2 800G Optical Modules
3.2.1 NDR OSFP 2xSR4
3.2.2 NDR OSFP 2xDR4
3.2.3 NDR OSFP 2xFR4
IV. InfiniBand HDR vs NDR
| Parameter | HDR Modules | NDR Modules |
|---|---|---|
| Per-Port Bandwidth | 200Gb/s | 400Gb/s (800Gb/s bidirectional) |
| Form Factor | QSFP56 | 400G: OSFP/QSFP112 800G: OSFP |
| Latency | <1μs (typical) | <0.8μs (optimized paths) |
| Power Consumption | ≤5W | ≤16.5W |
| Transmission Distance | SR4: ≤100m (OM4/OM5) FR4: ≤2km (OS2) | SR4: ≤50m (OM4/OM5) DR4: ≤500m (OS2) FR4: ≤2km (OS2) |
| Target Scenarios | Mid-sized HPC clusters, enterprise data centers | Supercomputing centers, large-scale AI training clusters |
| Cost | Low | High |
V. HDR vs. NDR: How to Choose?
Selecting between InfiniBand HDR and NDR requires comprehensive consideration of workload characteristics, budget constraints, and future scalability plans. Below are tailored recommendations for typical scenarios:
5.1 Artificial Intelligence & Deep Learning Training
If you operate large-scale distributed AI training, particularly large models based on Transformer architectures (such as GPT-4, LLaMA, Gemini, etc.), InfiniBand NDR is strongly recommended. These applications heavily rely on collective communication operations like AllReduce and AllGather, making them extremely sensitive to network bandwidth and latency. NDR delivers 400G–800G throughput and sub-μs latency, drastically reducing gradient synchronization time, boosting GPU utilization and overall training efficiency.
Furthermore, NDR deeply integrated with NVIDIA's Magnum IO and SHARP™, which unlocks the potential of AI clusters.
5.2 High-Performance Computing (HPC)
For traditional HPC applications, HDR remains a cost-effective choice, proven in long-term deployments with high stability and low operational costs.
However, if your goal is to achieve exascale (10¹⁵ operations per second) computing or involves extreme parallel task scheduling and massive data exchange, transitioning to NDR is essential. Its higher aggregate bandwidth and lower communication overhead are critical factors for overcoming performance bottlenecks.
5.3 Budget-Constrained Deployments
For projects with limited budgets and moderate network performance requirements, HDR is the more economical choice. The HDR equipment market offers ample supply and a vibrant secondary market, resulting in significantly lower TCO (Total Cost of Ownership) compared to NDR. What's more, HDR fully supports RDMA over Converged Ethernet (RoCE) and native InfiniBand protocols, ensuring excellent interoperability and migration flexibility.
5.4 Future-Oriented Infrastructure Investments
If you plan to scale AI/HPC clusters within the next two years or anticipate emerging workloads (e.g., multimodal LLMs, real-time reinforcement learning, digital twin simulations), NDR is the strategic choice.
Despite higher initial investment, the NDR architecture offers superior horizontal scalability and reserves physical layer and protocol stack headroom for future evolution to XDR (800G→1.6T), helping avoid resource wastage from redundant short-term deployments.
Conclusion
InfiniBand NDR represents the pinnacle of high-speed interconnect technology, serving as the ideal solution for cutting-edge AI, hyperscale training clusters, and exascale HPC. Meanwhile, InfiniBand HDR remains the preferred choice for mid-sized deployments and cost-sensitive projects, leveraging its mature ecosystem, stable performance, and lower cost.
