llm-d project

license

Apache-2.0(OSI)

source

public

core features withheld

no — CNCF project

The LICENSE body is the byte-stock Apache-2.0 text. The repository is public and active, the guides and Helm charts are in-tree, and no component sits behind a license key or enterprise edition; the vendors' paid platforms that bundle llm-d are separate products, the langchain shape.

• https://api.github.com/repos/llm-d/llm-d recorded 2026-09-01

  Repo metadata for llm-d/llm-d - archived false, license Apache-2.0, 4,375 stars, homepage https://www.llm-d.ai, pushed 2026-09-01.

• https://raw.githubusercontent.com/llm-d/llm-d/main/LICENSE recorded 2026-10-07

  LICENSE body, read in full: stock Apache License Version 2.0 text, unmodified (digest matches the canonical Apache-2.0 text).

• https://raw.githubusercontent.com/llm-d/llm-d/main/README.md recorded 2026-10-07

  README, read for a paid tier or withheld component (none named): "a high-performance distributed inference serving stack optimized for production deployments on Kubernetes"; CNCF Sandbox, founded by Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA; intelligent routing, tiered KV cache, prefill/decode disaggregation, wide expert-parallelism, SLO-aware autoscaling, batch APIs.

llm-d ships through container images and Helm charts, channels with no published download count, leaving GitHub stars as the only readable adoption signal. Production deployments named in the README, at Tesla, Red Hat, AWS, Oracle and in Google's own benchmarks, suggest the star count understates real use, though confirming that would need its own separate read.

• https://api.github.com/repos/llm-d/llm-d recorded 2026-09-01

  Repo metadata - stargazers_count = 4,375 - for llm-d/llm-d.

llm-d sits one tier below vLLM, level with Dynamo, since both are serving stacks whose forward pass belongs to someone else's engine. Its cross-accelerator production benchmarks give it the widest validation surface among the orchestration layers, but without an engine of its own it does not reach vLLM's tier.

• https://raw.githubusercontent.com/llm-d/llm-d/main/README.md recorded 2026-10-07

  Performance highlights: 3x throughput / 2x TTFT with prefix-cache-aware routing (Llama 3.1 70B, 4x MI300X, Tesla/Red Hat); up to 70% higher tokens/sec with P/D disaggregation vs standard vLLM (AWS B200); 50k tok/s wide-EP on 16x16 B200; 13.9x with hierarchical KV offload; themes - routing, KV-cache management, serving large models, operational excellence and batch processing.