Chapter 3: Multi-Node Networking, Scaling & Performance Optimization

ConnectX-7 200 Gb/s Network Interface and RDMA Support

Each DGX Spark ships with an NVIDIA ConnectX-7 SmartNIC, a dedicated networking processor that handles 200 Gb/s Ethernet traffic. The ConnectX-7 supports RDMA over Converged Ethernet (RoCE), which allows one machine's GPU to read from or write to another machine's memory directly, bypassing the operating system's network stack entirely.

In a traditional network transfer, data passes through multiple copies: application to kernel buffer, kernel buffer to NIC send buffer, across the wire, then the reverse on the receiving side. Each copy adds latency. RDMA eliminates these intermediate copies -- the NIC reads directly from GPU memory on one node and writes directly into GPU memory on the other. For AI workloads where nodes synchronize tensor data millions of times during inference, this difference is transformative.

Each DGX Spark node exposes two QSFP ports through the ConnectX-7, providing four RoCE interfaces total across the two physical ports.

Direct Two-Node Scaling

The simplest multi-node configuration connects two DGX Sparks with a direct QSFP cable -- no switch required. A 0.5-meter QSFP cable between the ConnectX-7 ports creates a point-to-point 200 Gb/s link.

Four-Node Cluster Topologies

Scaling beyond two nodes requires an Ethernet switch. Each node needs a path to every other node, and a switch provides that star topology. Community-tested switches include the MikroTik CRS804-4DDQ and CRS812, both supporting 200 GbE QSFP connections.

A four-node cluster provides 512 GB of unified memory (4 x 128 GB), enough to host 700-billion-parameter class models such as Qwen3.5-397B. An alternative for three-node clusters exists: a switchless mesh topology using PP=3/TP=1, where each node connects directly to the other two.

Network Configuration, MTU Tuning, and GPUDirect RDMA

Tensor Parallelism (TP)

Tensor parallelism splits individual layers of a neural network across multiple GPUs. Each GPU computes a portion of every matrix multiplication, then partial results are combined via an all-reduce communication step. On DGX Spark, TP2 means the model is split across two nodes (256 GB total); TP4 means four nodes (512 GB total).

A practical TP2 deployment with vLLM:

vllm serve Qwen/Qwen3-Coder-Next \
  --port 8000 \
  --tensor-parallel-size 2 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --max-model-len 32768

Pipeline Parallelism (PP)

Pipeline parallelism assigns entire groups of layers to different GPUs. Node 1 handles layers 1-20, Node 2 handles layers 21-40, and so on. It communicates less frequently than tensor parallelism (only between stages, not at every layer), but introduces pipeline bubbles -- idle time when stages wait for data. PP is better suited for training workloads where micro-batching can fill the bubbles.

Scaling Benchmarks

Llama 3.3 70B NVFP4 (32K input, 1K output, batch=1):

Four-node TP4 results for Qwen3.5-397B-INT4:

Speculative Decoding

Speculative decoding uses a small, fast "draft" model to propose multiple token candidates (typically 3-12 tokens ahead), which the larger "target" model then verifies in a single forward pass. The key insight: verifying multiple tokens simultaneously is nearly as fast as generating a single token because verification can be parallelized within one forward pass.

On Blackwell GPUs, speculative decoding achieves 2-3x speedups (vs ~1.5x on Hopper). The most dramatic result: DFlash speculative decoding on Blackwell 6000 Pro reached ~429.69 tokens/s -- a 4.8x increase over the baseline 90.20 tokens/s.

FlashAttention and Kernel Fusion

FlashAttention processes attention in tiles without materializing the full N-by-N attention matrix, reducing memory usage from O(N^2) to O(N). On Blackwell, optimized FlashInfer kernels deliver up to 2.25x higher decode throughput, achieving 85-90% tensor core utilization.

Kernel fusion combines LayerNorm, matrix multiplications, activations, and bias additions into single CUDA kernels, eliminating intermediate tensor writes. TensorRT-LLM applies these fusions automatically, achieving 4x throughput over native PyTorch.

Quantization: FP4/FP8/INT8

Quantization is often a necessity on DGX Spark. A 200B-parameter model in BF16 requires ~400 GB, far exceeding the 128 GB per node. At 4-bit precision, the same model fits in ~100 GB, making single-node inference possible. NVFP4 is specifically optimized for Blackwell's tensor cores.

Prefill vs Decode Phases

Continuous batching allows new requests to enter the batch as soon as a slot opens, overlapping prefill for new requests with decode for existing ones. DGX Spark's 273 GB/s LPDDR5X bandwidth (far below HBM3e's ~8 TB/s) makes quantization and multi-node TP the highest-impact interventions for decode performance.

Nsight Systems and Nsight Compute

Nsight Systems provides a system-wide view of GPU activity, CPU activity, memory transfers, and kernel execution timelines. Nsight Compute provides kernel-level detail: occupancy, memory throughput, instruction throughput, and roofline proximity.

On DGX Spark, profiling reveals that during decode, bandwidth utilization drops to 55-60% of the 273 GB/s peak, with a contention floor around 80-90 GB/s due to concurrent memory accesses (weights, KV-cache, activations).

Memory Bandwidth: The Fundamental Bottleneck

DGX Spark's GB10 SoC uses LPDDR5X at 8533 MT/s, delivering approximately 273 GB/s peak bandwidth. For a 70B-parameter model in FP16, reading all weights (~140 GB) takes ~0.51 seconds at peak bandwidth -- this sets a hard floor on per-token latency.

Mitigation strategies, ranked by impact:

1. Quantize aggressively: FP16 to NVFP4 = 4x fewer bytes per weight
2. Scale to multiple nodes: Each node adds 273 GB/s bandwidth
3. Use sparse MoE models: 35B MoE with 3B active reads only ~6 GB per step
4. Fuse kernels: Eliminate intermediate tensor writes

KV-Cache Management

The KV-cache stores key and value tensors from previous tokens. It grows proportionally with sequence length and competes directly with model weight storage in the 128 GB unified memory. vLLM manages this via --gpu-memory-utilization (default 0.9; reduce to 0.7 for multi-node long-context serving).

Data Locality and NUMA-Aware Scheduling

Despite the unified memory architecture (Grace CPU + Blackwell GPU on NVLink-C2C), data locality still matters. Memory pages closer to GPU controllers are accessed with lower latency. NUMA-aware scheduling ensures:

• Pinning vLLM and NCCL processes to the correct NUMA node
• Loading model weights into GPU-local memory regions
• Avoiding memory migrations during inference

Optimal tile sizes for DGX Spark: 64x64 with occupancy 2 for the GB10's 48 SMs at SM 12.1 compute capability. Larger tiles that perform well on datacenter GPUs underperform on DGX Spark.