8 × 400GbE RoCE Connectivity
Eight ConnectX-7 400GbE Ethernet links per DGX B200 provide high-bandwidth GPU-to-GPU communication.
400GbE RoCE Fabric for B200 AI Clusters
Build a scalable, 1:1 non-blocking RoCEv2 Ethernet fabric for B200 AI clusters with 400GbE connectivity.
Start Your B200 RoCE Fabric DesignShare your B200 cluster size, NIC configuration, target topology, and deployment requirements.
400GbE RoCE Fabric Overview
High-Performance Ethernet Networking for B200 AI Clusters
DGX B200 compute systems, NVIDIA Spectrum-4 SN5400 switches, and matched 400GbE Ethernet interconnects form a scalable RoCEv2 fabric for distributed AI workloads.
1:1 Non-Blocking Fat-Tree
Each SN5400 Leaf switch allocates 32 × 400GbE ports to servers and 32 × 400GbE ports to Spine uplinks, maintaining equal downlink and uplink capacity.
Lossless RoCE Traffic Control
PFC for the RoCE traffic class, together with ECN and QoS, helps control congestion and protect RDMA traffic across the Ethernet fabric.
B200 400GbE RoCE Solution Architecture
A two-level, 1:1 non-blocking RoCE fabric for B200 GPU clusters.
Cluster and Fabric Scale
- B200 GPU Servers
- 255servers
- 400Gb/s NICs
- 2,040400Gb/s NICs
- Leaf Switches
- 64switches
- Spine Switches
- 32switches
Optical Interconnect
- Optical Transceivers
- 2,040400G OSFP SR4 modules2,040400G QSFP-DD SR4 modules4,080400G QSFP-DD SR8 modules
- Optical Fiber Patch Cords
- 2,040MPO-12 APC patch cords2,040MPO-16 APC patch cords
400GbE RoCE Link Design
01 / Server–Leaf Link
- Protocol
- RoCEv2 over 400GbE Ethernet
- Link Rate
- 400GbE to 400GbE
- Interconnect Media
- MPO-12 APC multimode trunk cable
- Distance
- Up to 30 m on OM3 or 50 m on OM4
02 / Leaf–Spine Link
- Protocol
- RoCEv2 over 400GbE Ethernet
- Link Rate
- 400GbE to 400GbE
- Interconnect Media
- MPO-16 APC multimode trunk cable
- Distance
- Up to 70 m on OM3 or 100 m on OM4
Related Products
Components for the B200 400GbE RoCE fabric.
400GbE Ethernet RoCE Switches
- 64 × 400GbE QSFP-DD Ports
- SN5400 · Spectrum-4 · 2U
- Supports RoCEv2 Ethernet Fabrics
400GbE Ethernet RoCE Adapters
- 400GbE Ethernet Mode
- PCIe Gen5 x16
- Single-Port OSFP Interface
400GbE Ethernet Transceivers
- 400G QSFP-DD SR4 50m
- 400G QSFP-DD SR8 100m
- 400G OSFP SR4 50m
400GbE Ethernet AOC Cables
- OSFP-to-QSFP-DD
- QSFP-DD -to- QSFP-DD
- 1–50m Link Lengths
MTP/MPO Fiber Patch Cords
- MTP/MPO-12 APC
- MTP/MPO-16 APC
- OM3 / OM4 Multimode Fiber
WHY AICPLIGHT
AICPLIGHT Solution Support
Solution design, validated products, and dedicated project support.
Free Solution Design
A tailored solution architecture and matched product plan based on your requirements, site conditions, and deployment goals.
Tested Products for Deployment
Products are validated through demo testing. Optical transceivers undergo 100% inspection before shipment.
One Point of Contact
A dedicated AICPLIGHT contact supports the project from requirement review through delivery and ongoing maintenance.
FAQ
01What is RoCEv2, and why does a B200 cluster need it?
RoCEv2 enables RDMA over Ethernet, reducing CPU and kernel involvement in data transfers. For B200 AI training and inference clusters with substantial east-west GPU traffic, it delivers high-bandwidth, low-latency RDMA communication across a 400GbE Ethernet fabric.
02Why use a two-level, 1:1 non-blocking Fat-Tree topology?
DGX B200 systems continuously exchange GPU data across eight 400GbE links during distributed training and all-reduce operations. Each leaf switch uses 32 ports for server connections and 32 ports for spine uplinks, maintaining a 1:1 bandwidth ratio to prevent uplink oversubscription, congestion, and GPU wait time.
Lossless RoCE is enabled through PFC, ECN, and QoS configuration, while the 1:1 Fat-Tree ensures that fabric bandwidth is not oversubscribed.
03How many DGX B200 systems can this architecture support?
At full theoretical scale, the fabric can support up to 256 DGX B200 systems and 2,048 400GbE RoCE host links.
One server is reserved for management in this deployment, leaving 255 DGX B200 compute systems and 2,040 active 400GbE RoCE links. The final SU contains 31 compute systems.
04Which lossless and congestion-control settings are required for a RoCEv2 network?
ConnectX-7 adapters must operate in Ethernet mode, and all adapters and switches must use consistent RoCE traffic priority, MTU, and QoS mappings. Enable PFC only for the RoCE priority, together with ECN marking and NIC congestion-control settings.
05What tests are completed before delivery?
Before delivery, the topology, BOM, and cabling plan can be confirmed for the project, and selected optical transceivers can undergo optical performance testing. Link validation typically covers switch-to-NIC interoperability, 400GbE link establishment, RoCE traffic, stress testing, and link-stability verification.
CUSTOMER SUCCESS
Related AI Fabric Case Study
RELATED AI FABRIC CASE STUDY
Turnkey 128× B200 AI Platform for Fin-LLM Training
AICPLIGHT designed and delivered a 128-GPU NVIDIA B200 platform for a global financial services leader, including a Quantum-2 InfiniBand NDR computing fabric, optimized storage network, UFM management, and validated 800G/400G optical interconnects.
START YOUR PROJECT
Start Your Network Solution Design
Share your application, existing environment, and deployment requirements. AICPLIGHT will help assess the architecture and recommend suitable products.
We typically respond within 4 business hours.