Researchers and engineers running huge mixture-of-experts models on limited local GPU hardware.

KTransformers is an open-source hybrid CPU-GPU inference and fine-tuning framework that offloads MoE expert weights to RAM so models like DeepSeek-671B can run on a single consumer GPU.

Self-hosted, run locally from source or pip on Linux workstations with a GPU and large system RAM.

Released by Tsinghua's MADSys Lab, with SOSP 2025 publication and active development into 2026.

It exists to make trillion-parameter-class MoE inference feasible without datacenter-scale GPU memory.

Built on PyTorch and integrates optimized kernels from llama.cpp and Marlin.

Open source, Apache-2.0 license, self-hosted and runs fully local; no API key or external data transfer required.