Running Qwen3.5-27B Across Multiple GPUs Over LAN Achieves Practical Speed for Local Inference

A practitioner in the LocalLLaMA community discovered the power of llama.cpp's RPC server feature, achieving practical inference speeds by splitting Qwen3.5-27B Q5 quantization across a 4070Ti GPU and AMD RX6800 over LAN, reaching 13 tokens/second with a 32K token prompt. This discovery has significant implications for practitioners who are "GPU poor" but have access to multiple discrete GPUs across a home or small office network.

The approach leverages llama.cpp's RPC server capabilities to transparently distribute model inference across heterogeneous hardware. Previously, running 27B dense models required high-end single GPUs or very low inference speeds; this technique enables mixed NVIDIA/AMD setups to achieve reasonable performance.

This is a practical breakthrough for budget-conscious deployers who can repurpose older gaming GPUs or distribute compute across existing machines. It expands the viable hardware configurations for local LLM inference significantly and demonstrates that sophisticated multi-GPU orchestration no longer requires specialized frameworks—llama.cpp now handles the complexity transparently.

Source: r/LocalLLaMA · Relevance: 8/10