NVIDIA DGX Spark: Eight nodes pool 1 TB of memory for large AI models

What this is about

At GTC Berlin, NVIDIA announced official support for clusters of up to eight DGX Spark machines. DGX Spark is NVIDIA's compact AI computer for your desk. Link several together and their memory adds up: a single Spark has 128 GB of unified memory (CPU and GPU share the same memory), four nodes reach 512 GB, and eight reach 1 TB. That lets you run models locally that wouldn't even start on smaller systems. The numbers below come from the vendor itself. Setup is meant to run through NVIDIA Sync and its Cluster Assistant; support for eight nodes in Sync is due by the end of the year.

Running large models locally

The biggest advantage is memory. The full GLM 5.3 (753B) from Z.ai, a Mixture-of-Experts model (only part of the parameters are active per token), runs on the 8x 128 GB cluster. NVIDIA calls it one of the largest open models you can deploy locally. As an early example, GLM 5.3 Flash ran on eight nodes at about 80 tokens per second, which is enough for interactive development.

For the full model, NVIDIA gives these figures:

Prefill is reading in the prompt; decode is generating the answer token by token. More simultaneous users raise total throughput but lower per-user speed.

Faster communication with RoCEnante

With tensor parallelism (a model is split across several GPUs), a lot of time goes into collective operations like All-Reduce, in which all nodes reconcile intermediate results. For switch clusters, NVIDIA is testing a Spark-optimized backend called RoCEnante. Each rank sends its input directly to all the others and sums locally, controlled by a single GPU kernel. This saves communication steps compared with ring all-reduce, in which data travels from neighbor to neighbor over several rounds.

The price is more network traffic. For large messages, many nodes or high parallelism, NCCL (NVIDIA's standard library) is better. Measured gains were about 5% more prefill and 21% more decode with four nodes (NVFP4), plus roughly 18% more decode with eight nodes on the unquantized model. The backend is integrated into FlashInfer and automatically falls back to NCCL if its startup test fails. It works only with switch topologies. The original one-shot all-reduce idea came from the community (Jason Cook/el8).

Scaling from two to eight nodes

Thanks to ConnectX-7 network cards and RoCE (RDMA over Ethernet), the nodes exchange tensors efficiently. The test used the dense model Qwen3.8-27B in NVFP4 (4-bit floating point) without speculative decoding:

Decode scales noticeably, prefill only modestly. Scaling is clearly sublinear here: eight nodes deliver about 4.4x.

Ring or switch?

You can cable the nodes directly in a ring without a switch. That cuts cost and effort, and with four nodes NCCL uses ring algorithms anyway. NVIDIA recommends the ring for a simple, inexpensive four-node inference box. You need a switch with eight nodes if you want to split the cluster into independent groups (say, two sets of four), for MPI-style any-to-any patterns, and for the RoCEnante backend.

Model routing with NeMo Switchyard

Not every request needs the biggest model. NeMo Switchyard uses a lightweight classifier that distributes requests across model groups, and a load balancer spreads them across replicas. In the prototype, MiniCPM5 2B (OpenBMB) served as the classifier and handled simple tasks, while Qwen3.8 Flash Next handled hard ones. On a tool-use benchmark with 92 cases, the routed agent scored 91/100, matching the Qwen-only variant. With three Qwen replicas (two nodes each) and two MiniCPM replicas (one node each), total throughput is about 1.5x when half the requests go to Qwen, and about 3.0x at a quarter. The routing rules live in a YAML file, and NVIDIA provides an example.

Pre-training: near-linear scaling

In training, each node holds a full copy of the model, and only the gradients are synchronized once per step. NVIDIA trained a nanochat model (about 896.5M parameters, 20 layers) from scratch:

The code is on GitHub.

Our take

For developers who want to test large open models or multi-agent setups locally, an eight-node cluster is an interesting option, mainly because of the memory. The caveats: all figures are vendor claims, the post gives no prices for the cluster or switch, and 31 tokens/s for the full 753B model with a single user is usable but not fast. Inference decode scaling is also well below linear. For training small models, scaling is very good, but a 900-million-parameter model is far from frontier size. So the value proposition lies more in local prototyping than in production use.