Back to Gap Map Model components / Inference code

NVIDIA Dynamo

open source / Overall score: 3.0

Dynamo is NVIDIA's datacenter-scale inference serving stack, the orchestration layer that turns clusters running vLLM, SGLang or TensorRT-LLM into one coordinated system. It disaggregates prefill and decode into independently scalable pools, routes requests by KV cache overlap, offloads KV blocks across GPU, CPU, SSD and remote storage, and autoscales against latency SLAs. It is built in Rust and Python and deploys on Kubernetes.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI
source: public
core features withheld: no — the published repo is the whole stack

The LICENSE body carries a NOTICE about vendored MIT-licensed test data, then the full unmodified Apache-2.0 text covering the rest of the codebase. The repository is public and active, and no component sits behind a license key; NVIDIA sells NIM and enterprise support as separate products beside it.

https://api.github.com/repos/ai-dynamo/dynamo recorded 2026-09-01
Repo metadata for ai-dynamo/dynamo - archived false, license NOASSERTION (the API could not classify the NOTICE-prefixed file), 7,938 stars, homepage https://docs.nvidia.com/dynamo/latest, pushed 2026-09-01.
https://raw.githubusercontent.com/ai-dynamo/dynamo/main/LICENSE recorded 2026-10-07
LICENSE body, read in full: a NOTICE that test-data files under ./lib/llm/tests/data/deepseek-v3.2 are MIT-licensed derivatives of the DeepSeek-V3.2 repository, "The rest of this codebase is licensed under the Apache License 2.0", followed by the complete stock Apache-2.0 text with END OF TERMS AND CONDITIONS at the standard place and nothing appended after the appendix.
https://raw.githubusercontent.com/ai-dynamo/dynamo/main/README.md recorded 2026-10-07
README, SPDX header "Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES ... SPDX-License-Identifier: Apache-2.0"; "The open-source, datacenter-scale inference stack"; read for a paid tier or withheld component of Dynamo itself - none named; Apache-2.0 badge; PyPI ai-dynamo badge; NGC container catalog linked as packaging.

Adoption

2 medium confidence

2.0

PyPI downloads of the ai-dynamo package, which resolves to this repository, are the measured adoption signal. Dynamo also ships as NGC container images, a channel with no published count, so the figure is a floor on total use rather than the whole of it.

https://pypi.org/pypi/ai-dynamo/json recorded 2026-09-01
project_urls Repository = https://github.com/ai-dynamo/dynamo.git; version 1.4.2; summary "Distributed Inference Framework".
https://pypistats.org/api/packages/ai-dynamo/recent recorded 2026-09-01
"last_month":58749

Capability

4 medium confidence

4.0

Dynamo is one step below vLLM. Its cluster-scale serving surface (SemiAnalysis reports 7x throughput per GPU on GB200 NVL72, and Baseten reports 2x faster time-to-first-token) is the broadest orchestration layer in the category, but it coordinates someone else's engine, vLLM, SGLang or TensorRT-LLM, and cannot run inference on its own, which keeps it a step behind vLLM.

https://raw.githubusercontent.com/ai-dynamo/dynamo/main/README.md recorded 2026-10-07
Core capabilities table: Disaggregated Prefill/Decode, KV-Aware Routing, KV Block Manager, ModelExpress weight streaming, SLA Planner, Grove gang scheduling, fault tolerance; "it doesn't replace SGLang, TensorRT-LLM, or vLLM, it turns them into a coordinated multi-node inference system"; key results 7x throughput/GPU (GB200 NVL72), 2x TTFT (Baseten).

Verified 2026-09-24