Leaf-Spine Network Architecture for AI Data Centres

Leaf-spine (also called Clos fabric) is the dominant network topology for modern AI GPU clusters and hyperscale data centres. It replaces the older three-tier core/distribution/access model with a flat two-tier design that delivers uniform, low-latency bandwidth between any two servers regardless of where they sit in the fabric. For AI training clusters where every GPU server communicates with every other GPU server simultaneously during all-reduce operations, a leaf-spine fabric is not optional — it is the prerequisite for predictable scaling.

The Two-Tier Structure

A leaf-spine fabric has exactly two layers of switches. Every leaf switch connects upward to every spine switch — there are no direct leaf-to-leaf connections. Every server connects downward to exactly one leaf switch. This means that any two servers anywhere in the fabric are always exactly two hops apart: server → leaf → spine → leaf → server.

This uniform two-hop path has important consequences. First, latency between any two servers is identical and predictable — there are no long chains of hops that introduce variable delay. Second, adding more spine switches adds proportional east-west bandwidth without changing hop count or latency. Third, if a single spine switch fails, traffic redistributes across the remaining spine switches with no topology change — ECMP (Equal-Cost Multi-Path) routing automatically balances flows across all available paths.

Bandwidth Scaling

The key design parameter is the oversubscription ratio: the ratio of server-facing (downlink) bandwidth to spine-facing (uplink) bandwidth on each leaf switch. A 1:1 non-blocking leaf switch has as much uplink bandwidth as downlink bandwidth; traffic to any remote server is never throttled. A 4:1 oversubscribed leaf switch has four times more server-facing capacity than spine-facing capacity, which means high concurrent east-west traffic will be congested.

For AI training clusters, non-blocking or very low oversubscription (2:1 maximum) is required. NVIDIA's DGX SuperPOD reference architecture specifies a fully non-blocking InfiniBand fabric for exactly this reason: during an all-reduce step, all GPU servers transmit simultaneously at line rate, and any oversubscription creates a bottleneck that idles GPU compute.

Typical switch radix (port count) determines fabric scale:

• A 64-port 400G spine switch with 32 ports allocated as downlinks to leaf and 32 as uplinks provides 6.4 Tbps total fabric bandwidth per spine switch
• A fabric with 8 spine switches × 32 ports each = 256 leaf-facing ports = 256 × 400G = 102.4 Tbps aggregate east-west bandwidth
• Each leaf switch connecting 32 servers at 100G and having 32 × 400G uplinks to spine achieves a 1:1.28 oversubscription ratio — effectively non-blocking

InfiniBand Leaf-Spine vs Ethernet Leaf-Spine

AI clusters use two distinct leaf-spine implementations: NVIDIA InfiniBand fabric and Ethernet fabric. They are not interchangeable — each is optimised for a different communication pattern.

InfiniBand leaf-spine (using NVIDIA Quantum-2 QM9700 or Quantum-3 QM9790 switches) provides RDMA (Remote Direct Memory Access) natively, meaning GPU-to-GPU data transfer bypasses the CPU and OS entirely. InfiniBand also implements adaptive routing at the switch level, spreading flows dynamically across available spine paths without requiring the host to configure ECMP. For distributed training all-reduce traffic — dominated by a small number of very large flows — InfiniBand delivers consistently lower latency (sub-microsecond hop latency vs multi-microsecond for Ethernet) and higher throughput utilisation.

Ethernet leaf-spine (using Cisco Nexus 9000, Arista 7800R, Juniper QFX10000, or Nvidia Spectrum-4 SN5600 switches) is used for storage traffic, management networks, and increasingly for AI clusters deploying RoCEv2 (RDMA over Converged Ethernet). RoCEv2 provides RDMA semantics over standard Ethernet but requires careful QoS configuration (PFC — Priority Flow Control, ECN — Explicit Congestion Notification, DCQCN algorithm) to avoid head-of-line blocking and ensure lossless delivery. When configured correctly, RoCEv2 on Ethernet at 400G provides comparable training throughput to InfiniBand for many workloads at lower per-port cost.

BGP as the Leaf-Spine Routing Protocol

Modern leaf-spine fabrics run BGP (Border Gateway Protocol) as the underlay routing protocol rather than OSPF or IS-IS. BGP EVPN (Ethernet VPN) provides both layer-2 and layer-3 forwarding with ECMP load balancing across all spine paths. Each leaf switch peers with each spine switch in a point-to-point eBGP session — no OSPF flooding, no spanning tree. This approach scales to thousands of nodes, converges in seconds after a spine failure, and supports multi-tenancy via VXLAN encapsulation on the same physical fabric.

For AI clusters, the fabric typically uses a separate management network (1G or 10G Ethernet) for IPMI/BMC access and OS provisioning, a separate InfiniBand fabric for GPU-to-GPU training traffic, and a high-speed Ethernet fabric (100G or 400G per server) for storage (NFS, Lustre, GPFS) and general data movement.

Spine Switch Selection for AI Clusters

Spine switch selection depends on port speed, buffer size, and fabric protocol. Key platforms deployed in AI cluster spines:

• NVIDIA Quantum-2 QM9700 — 400G InfiniBand, 64 ports, purpose-built for GPU cluster fabrics, adaptive routing, 8192 virtual lanes. Haink has supplied this switch for sovereign AI GPU clusters in the Gulf region. Lead time from Hong Kong: 8–14 business days.
• NVIDIA Quantum-3 QM9790 — 800G InfiniBand, 64 ports, 409.6 Tbps switching capacity, designed for H200 and B200 GPU cluster scale-out.
• Arista 7800R3 — 400G Ethernet, modular spine, up to 576 × 400G ports per chassis, deep buffer (DRAM), widely used as Ethernet spine in large-scale AI data centres.
• Cisco Nexus 9364C-GX — 64 × 400G QSFP-DD ports, 51.2 Tbps, strong BGP-EVPN implementation, compatible with Cisco ACI fabric automation.
• Juniper QFX10008 — modular spine chassis, up to 9.6 Tbps per line card slot, excellent for mixed-traffic data centres combining AI training, storage, and front-end workloads.

Oversubscription by Workload Type

• AI distributed training (all-reduce intensive): 1:1 non-blocking required; any oversubscription creates measurable GPU idle time during communication phases
• AI inference serving: 4:1 to 8:1 acceptable; inference traffic is request-response with no synchronised bulk transfers
• General enterprise compute: 4:1 to 16:1 typical; most enterprise applications have bursty rather than sustained east-west traffic
• Storage fabric (NFS/NVMe-oF): 2:1 maximum recommended to avoid latency spikes during concurrent large file transfers

Common Design Mistakes

The most common leaf-spine design error in AI deployments is under-provisioning spine switches. A cluster may have non-blocking leaf switches but only two spine switches — creating a spine bottleneck when all leaf switches transmit simultaneously. The number of spine switches should equal the number of uplink ports on each leaf switch, ensuring every leaf-to-spine path has equal bandwidth.

The second most common error is mixing InfiniBand and RoCEv2 training traffic on the same leaf-spine fabric without proper traffic isolation. InfiniBand and Ethernet use different congestion control mechanisms; running both on the same Ethernet spine without QoS segmentation causes RoCEv2 flows to trigger PFC pause frames that inadvertently throttle unrelated traffic.

Related Resources

• Networking Hardware — Haink Supply
• Cisco vs Juniper Switching — Which for AI Data Centre?
• Network Infrastructure Supplier
• NVLink vs PCIe GPU Interconnect
• Case Study: Gulf Fintech Network Infrastructure
• Case Study: Telecom Data Hall Networking

Frequently Asked Questions

How many spine switches does an AI cluster need?

The number of spine switches should equal the number of uplink ports on each leaf switch to achieve a non-blocking fabric. If each leaf switch has 16 uplink ports to spine, you need 16 spine switches, each connected to every leaf. Fewer spine switches create oversubscription at the spine tier; more spine switches waste capacity. For a 32-server GPU cluster with 4 leaf switches × 8 uplinks each, you need 8 spine switches for a 1:1 non-blocking fabric.

Can I use a three-tier architecture for an AI cluster?

Three-tier (core/distribution/access) adds a third hop between servers and can create asymmetric paths and bandwidth bottlenecks between distribution and core layers. For AI training traffic that requires uniform bandwidth between all nodes simultaneously, three-tier architecture is not recommended. Leaf-spine is the standard for GPU clusters of any meaningful size. Very small clusters (under 8 servers) may operate with a single flat layer of switches without a spine tier.

What is the difference between InfiniBand and RoCEv2 for AI training?

Both provide RDMA for GPU-to-GPU data transfer. InfiniBand is a purpose-built protocol with native adaptive routing, guaranteed in-order delivery, and sub-microsecond latency — deployed as a separate fabric from Ethernet. RoCEv2 provides RDMA semantics over standard Ethernet, sharing the same physical infrastructure as storage and management traffic, at lower cost per port but with more complex QoS requirements. For large-scale training clusters (100+ GPUs), InfiniBand typically delivers higher sustained throughput due to adaptive routing and hardware congestion control. For moderate-scale deployments, RoCEv2 on 400G Ethernet is a practical and cost-effective alternative.