AI Potluck
Contribute
Back to Gap Map Model components / Inference code

Text Generation Inference

Hugging Face
open source / Overall score: 3.0

Text Generation Inference (TGI) is Hugging Face's LLM serving toolkit, a Rust, Python, and gRPC server purpose-built for the HF model ecosystem. It supports continuous batching, tensor parallelism, Flash Attention, token streaming, and quantization (GPTQ, AWQ, EETQ), and has powered Hugging Face's Inference Endpoints and Hugging Chat. Its final release was v3.3.7 (2025-12-19) and the repository was archived 2026-03-21, with Hugging Face pointing new deployments to vLLM, SGLang, llama.cpp, and MLX.

The project is archived, so the entry is read against its final release rather than an active repository.

Openness

5 high confidence
5.0
license
Apache-2.0(OSI)
source
public
core features withheld
no — Apache-2.0 across the repo, no ee/ or enterprise directory and no license key
note
was briefly under restrictive HFOIL license in 2023, reverted to Apache-2.0

Text Generation Inference's LICENSE is Apache-2.0 with nothing withheld: there is no enterprise directory, no closed package and no license key, and Hugging Face's hosted Inference Endpoints run this code rather than hold features back from it. The repository carries a "Public archive" label, but the source remains fully public despite being archived.

  • https://github.com/huggingface/text-generation-inference recorded 2026-09-22

    "This repository was archived by the owner on Mar 21, 2026. It is now read-only." with a "Public archive" label, confirming the source remains publicly readable. Licensed Apache-2.0. Top-level directories are .github, assets, backends, benchmark, clients/python, docs, integration-tests, launcher, load_tests, nix, proto, router, server -- no ee/ or enterprise directory and no license-key gate. The README notes text-generation-inference is in maintenance mode and points users at vLLM and SGLang. Hugging Face's own products (HuggingChat, Inference API, Inference Endpoints) are named as places TGI runs in production, not as tiers withholding features from this source.

  • https://github.com/huggingface/text-generation-inference/blob/main/LICENSE recorded 2026-09-22

    Apache License Version 2.0, January 2004; Copyright 2022 Hugging Face

Adoption

3 medium confidence
3.0

README states production use at Hugging Face and explicitly recommends vLLM/SGLang/llama.cpp/MLX going forward, confirming a real but sunsetting footprint.

  • https://github.com/huggingface/text-generation-inference recorded 2026-08-09

    'Used in production at Hugging Face to power Hugging Chat, the Inference API and Inference Endpoints.' and 'This approach is now adopted by downstream inference engines, which we contribute to and recommend using going forward: vllm, SGLang, as well as local engines with inter-compatibility such as llama.cpp or MLX.'

  • https://huggingface.co/docs/text-generation-inference/index recorded 2026-09-29

    The docs index says "Text Generation Inference is used in production by multiple projects, such as" and lists Hugging Chat and nat.dev; it also says the engines "which we contribute to and recommend using going forward" are vllm, SGLang, llama.cpp and MLX.

  • https://raw.githubusercontent.com/huggingface/text-generation-inference/main/README.md recorded 2026-09-29

    The main-branch README says "Used in production at [Hugging Face](https://huggingface.co)" and "text-generation-inference is now in maintenance mode.", and recommends vllm and SGLang going forward.

Capability

3 medium confidence
3.0

Development has stopped: the repository is archived and the feature set is frozen at continuous batching, tensor parallelism, Flash Attention, token streaming, GPTQ/AWQ/EETQ quantization, OpenTelemetry tracing and Prometheus metrics, while vLLM and SGLang keep advancing. That holds it mid-pack rather than at the frontier.

Verified 2026-09-22