Model components / Inference code

KTransformers

KTransformers runs very large language models on limited hardware by splitting inference between CPU and GPU, keeping routed experts in DRAM with AMX- and AVX-optimized kernels while the GPU handles the rest. It made DeepSeek-R1 671B runnable on a single 24GB GPU with 382GB of DRAM, supports NVIDIA, AMD, Intel and Ascend backends, and its kt-kernel is being integrated into SGLang. LoRA and full fine-tuning of MoE models through LLaMA-Factory sit alongside the inference path.

PyPI package verified as the project's own but not declared - minority channel, 2,223 downloads/30d against 19,430 stars; adoption reads the stars route. Verified 2026-09-01 via GitHub, the LICENSE body, the repository README, the PyPI project JSON and pypistats.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public
core-gated: ungated(research project

The LICENSE body is the byte-stock Apache-2.0 text (digest identical to the canonical text vLLM ships). The repository is public and very active, and the README describes a research project with no commercial edition, hosted tier or withheld component.

https://raw.githubusercontent.com/kvcache-ai/ktransformers/main/LICENSE recorded 2026-09-01
LICENSE body, read in full: stock Apache License Version 2.0 text, unmodified (digest matches the canonical Apache-2.0 text).
https://api.github.com/repos/kvcache-ai/ktransformers recorded 2026-09-01
Repo metadata for kvcache-ai/ktransformers - archived false, license Apache-2.0, 19,430 stars, pushed 2026-09-01.
https://raw.githubusercontent.com/kvcache-ai/ktransformers/main/README.md recorded 2026-09-01
README, read for a paid tier or gated component (none named): "a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing"; kt-kernel Inference and SFT entry points; DeepSeek-R1 671B on single 24GB GPU / 382G DRAM; Ascend NPU, Intel Arc, ROCm, AMX support; SGLang integration roadmap.

Adoption

3 low confidence

3.0

19,430 GitHub stars, in the >10K band of the stars scale, which caps at level 3. A PyPI package exists and is the project's own, but at 2,223 downloads in the trailing 30 days it is a minority channel for a product installed through hardware-specific source builds, so banding on it would measure the wrong channel; it is left undeclared and the stars route wins.

https://api.github.com/repos/kvcache-ai/ktransformers recorded 2026-09-01
Repo metadata - stargazers_count = 19,430 - for kvcache-ai/ktransformers.
https://pypistats.org/api/packages/ktransformers/recent recorded 2026-09-01
"last_month":2223 - the PyPI channel figure the note above declines to band on.

Capability

4 medium confidence

4.0

One band below the vllm anchor, level with lmdeploy and mistral-rs. It owns a capability the anchor does not have - serving 671B-class MoE models on a single consumer GPU by scheduling experts onto CPU - and spans four accelerator vendors, but it is not the datacenter throughput frontier and its serving path is now partly delivered through SGLang integration rather than standing alone.

https://raw.githubusercontent.com/kvcache-ai/ktransformers/main/README.md recorded 2026-09-01
Updates timeline and capabilities: DeepSeek-R1/V3 on single 24GB GPU + 382G DRAM at 3~28x speedup; AMX-Int8/BF16 kernels; FP8 GPU kernel; 3-layer prefix cache; multi-concurrency; ROCm, Intel Arc, Ascend NPU support; SGLang integration; GLM-5.3, MiniMax-M3, Kimi-K2.5 day-0 support.

Verified 2026-09-01