KTransformers
KVCache.AIKTransformers runs very large language models on limited hardware by splitting inference between CPU and GPU, keeping routed experts in DRAM with AMX- and AVX-optimized kernels while the GPU handles the rest. It made DeepSeek-R1 671B runnable on a single 24GB GPU with 382GB of DRAM, supports NVIDIA, AMD, Intel and Ascend backends, and its kt-kernel is being integrated into SGLang. LoRA and full fine-tuning of MoE models through LLaMA-Factory sit alongside the inference path.
Openness
5 high confidence- license
- Apache-2.0(OSI)
- source
- public
- core features withheld
- no — research project
The LICENSE body is the byte-stock Apache-2.0 text (digest identical to the canonical text vLLM ships). The repository is public and very active, and the README describes a research project with no commercial edition, hosted tier or withheld component.
- https://api.github.com/repos/kvcache-ai/ktransformers recorded 2026-09-01
Repo metadata for kvcache-ai/ktransformers - archived false, license Apache-2.0, 19,430 stars, pushed 2026-09-01.
- https://raw.githubusercontent.com/kvcache-ai/ktransformers/main/LICENSE recorded 2026-10-07
LICENSE body, read in full: stock Apache License Version 2.0 text, unmodified (digest matches the canonical Apache-2.0 text).
- https://raw.githubusercontent.com/kvcache-ai/ktransformers/main/README.md recorded 2026-10-07
README, read for a paid tier or gated component (none named): "a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing"; kt-kernel Inference and SFT entry points; DeepSeek-R1 671B on single 24GB GPU / 382G DRAM; Ascend NPU, Intel Arc, ROCm, AMX support; SGLang integration roadmap.
Adoption
3 low confidenceGitHub stars are the adoption signal for KTransformers, since the product is installed mainly through hardware-specific source builds. Its PyPI package is confirmed as the project's own, but its downloads are a minority channel next to source installs, so it is left undeclared and stars carry the reading.
- https://api.github.com/repos/kvcache-ai/ktransformers recorded 2026-09-01
Repo metadata - stargazers_count = 19,430 - for kvcache-ai/ktransformers.
- https://pypistats.org/api/packages/ktransformers/recent recorded 2026-09-01
"last_month":2223 - the PyPI channel figure the note above declines to use.
Capability
4 medium confidenceKTransformers sits one tier below vLLM, level with LMDeploy and mistral.rs. It can do something vLLM cannot, serve 671B-class MoE models on a single consumer GPU by scheduling experts onto the CPU, and spans four accelerator vendors, but it is not built for datacenter throughput and increasingly delivers its serving path through an SGLang integration rather than standing alone.
- https://raw.githubusercontent.com/kvcache-ai/ktransformers/main/README.md recorded 2026-10-07
Updates timeline and capabilities: DeepSeek-R1/V3 on single 24GB GPU + 382G DRAM at 3~28x speedup; AMX-Int8/BF16 kernels; FP8 GPU kernel; 3-layer prefix cache; multi-concurrency; ROCm, Intel Arc, Ascend NPU support; SGLang integration; GLM-5.3, MiniMax-M3, Kimi-K2.5 day-0 support.
Verified 2026-09-27