AI Potluck
Contribute
Back to Gap Map Model components / Inference code

vLLM

vLLM (UC Berkeley / vLLM team)
open source / Overall score: 4.5(leading)

vLLM is a serving engine for large language models built on PagedAttention, which manages the KV cache as virtual memory pages to reduce memory waste during batched inference. It supports continuous batching, tensor and pipeline parallelism, speculative decoding, and quantization, exposing an OpenAI-compatible API server for self-hosted deployment. It originated in the Sky Computing Lab at UC Berkeley and is community-governed under vllm-project.

Openness

5 high confidence
5.0
license
Apache-2.0(OSI)
source
public
core features withheld
no

vLLM is Apache-2.0 across the repository. The published tree is the full engine with no commercial edition or enterprise-licensed directory, so nothing is gated out of the source.

  • https://github.com/vllm-project/vllm recorded 2026-10-13

    Public repository vllm-project/vllm, described as "A high-throughput and memory-efficient inference and serving engine for LLMs"; full source tree with no commercial edition or enterprise-licensed directory - the page's only "Enterprise"/"Pricing" strings are GitHub's own site navigation; README states it was originally developed in the Sky Computing Lab at UC Berkeley and is now maintained by a diverse community.

  • https://raw.githubusercontent.com/vllm-project/vllm/main/LICENSE recorded 2026-10-13

    "Apache License" "Version 2.0, January 2004" full text, unmodified.

Adoption

4 high confidence
4.0

PyPI downloads are vLLM's primary adoption signal, and a large GitHub star count corroborates but does not substitute for that measure.

Capability

5 high confidence
5.0

vLLM is the serving stack behind AMD's MLPerf Inference v6.0 submission on Instinct MI355X, including a multi-node Llama 3.1 405B run at 97-98% scaling efficiency, and it supports more than 200 model architectures on Hugging Face. It is the reference point for capability in this category, against which the other engines here are described.

  • https://github.com/vllm-project/vllm recorded 2026-08-09

    "vLLM seamlessly supports 200+ model architectures on Hugging Face"

  • https://raw.githubusercontent.com/vllm-project/vllm/main/README.md recorded 2026-09-29

    README feature list: "Continuous batching of incoming requests, chunked prefill, prefix caching", "Tensor, pipeline, data, expert, and context parallelism for distributed inference", "OpenAI-compatible API server, plus Anthropic Messages API and gRPC support", "Efficient multi-LoRA support for dense and MoE layers", and "vLLM seamlessly supports 200+ model architectures on Hugging Face".

  • https://rocm.blogs.amd.com/artificial-intelligence/mlperf-inference-v6.0/README.html recorded 2026-09-29

    AMD Instinct MI355X MLPerf Inference v6.0 submission; "tuning experiments were conducted on vLLM parameters and used to refine the scheduler configurations", and the Llama 3.1 405B multi-node run used "an AMD MLPerf Stack with" vLLM at "98% scaling efficiency".

Verified 2026-09-24