12GB×2 の Blackwell + WSL2 で exllamav3 の Tensor Parallel を動かすまで 「WSL では NCCL 無理」の壁と gather カーネルのレースThis article documents how to enable exllamav3 Tensor Parallel inference on…

Running large language models locally on consumer hardware increasingly means combining more than one mid-range GPU, and this write-up walks through the practical obstacles of doing exactly that: enabling exllamav3's Tensor Parallel inference across two 12 GB Blackwell-generation cards inside a WSL2 environment. The exercise matters because it sits at the intersection of three trends — the rise of quantized inference engines, the growing viability of multi-GPU setups for enthusiasts, and the reality that many hobbyists run their AI stack on Windows through the Windows Subsystem for Linux rather than on bare-metal Linux.

By way of background, exllamav3 is a high-performance inference library optimized for its EXL3 quantization format, designed to squeeze large models onto limited VRAM while retaining acceptable quality. Tensor Parallel, the technique at the center of this article, splits individual weight matrices across multiple GPUs so that each device handles a slice of every layer. This differs from simple layer-wise or pipeline splitting, where whole layers are assigned to separate cards. Tensor parallelism can improve throughput and lets a model that would not fit on a single 12 GB card span two of them, but it demands frequent, low-latency communication between the GPUs to combine partial results after each parallelized operation.

That communication requirement is where the first wall appears. Multi-GPU inference frameworks typically rely on NCCL, NVIDIA's Collective Communications Library, to handle the all-reduce and all-gather operations that stitch tensor-parallel shards back together. NCCL, however, is not fully supported under WSL2. The virtualized GPU path that WSL exposes does not provide the peer-to-peer and device-to-device transport features NCCL expects, so attempts to initialize an NCCL communicator inside WSL commonly fail or hang. The article frames this bluntly as the "NCCL is impossible on WSL" barrier, and it is a well-known limitation that has affected other projects attempting distributed workloads on the subsystem.

The workaround, as described, is to avoid NCCL entirely and route the cross-GPU synchronization through an alternative communication path. In practice this appears to mean falling back to a non-NCCL backend for the collective operations — for example, staging data through host memory or using a custom gather implementation rather than the library's optimized collectives. This sidesteps the incompatibility but shifts the burden onto exllamav3's own kernels to move and combine tensors between the two Blackwell devices, which is where the second problem surfaces.

That second issue is a race condition in the gather kernel. When partial outputs from each GPU are gathered and recombined, correct results depend on all contributing operations completing before the gathered buffer is read. If synchronization is missing or incorrectly placed, one GPU can begin reading or writing shared data before the other has finished, producing corrupted activations, non-deterministic output, or outright crashes. The article documents diagnosing and addressing this timing hazard, likely by adding or correcting stream synchronization so that the gather step waits for the relevant kernels and memory transfers to finish. Race conditions of this kind are notoriously intermittent, which makes them difficult to reproduce and easy to mistake for other faults, so pinning the behavior to the gather path is a meaningful part of the story.

The Blackwell context adds another layer of nuance. As a newer architecture, Blackwell cards can require recent CUDA toolkits, driver versions, and framework builds, and support in inference libraries may still be maturing. Combining bleeding-edge hardware with the constraints of WSL2 compounds the risk that a given code path has not been widely tested, which is one reason low-level bugs like the gather race can go unnoticed until someone runs this specific configuration.

For readers weighing a similar build, the takeaways are pragmatic rather than universal. Dual 12 GB GPUs can serve as an affordable route to running larger quantized models with tensor parallelism, but the software path is not turnkey on WSL2, and users should expect to substitute non-NCCL communication and to verify output correctness carefully. Those who can run native Linux may avoid the NCCL limitation altogether, and anyone attempting this on Windows should treat the described fixes as a starting point that may shift as exllamav3 and Blackwell support continue to evolve.