TensorRT-LLM
NVIDIANVIDIA's optimized LLM inference library providing maximum throughput on NVIDIA GPUs through kernel fusion, in-flight batching, FP8/INT4 quantization, and multi-GPU tensor parallelism. Python API for defining and compiling models into optimized TensorRT engines. Used by major cloud providers and inference services as the backend for NVIDIA GPU deployments. Source code is Apache-2.0, but the value proposition is tightly coupled to NVIDIA hardware and CUDA ecosystem.
Actively maintained; repo last updated 2026-06-01 with ongoing releases (AutoDeploy Q2/Q3 milestone). Confirmed to exist as of June 2026.
Openness
5 high confidence- license
- Apache-2.0(OSI, stated in LICENSE
- source
- public
- core-gated
- ungated
- caveat
- runs only on NVIDIA GPUs (vendor-hardware lock, not a license restriction)
Apache-2.0 across the repo. The LICENSE opens "This project is licensed under the Apache 2.0 license" and then appends the terms of the third-party code it bundles, which is why GitHub's classifier reports NOASSERTION. Nothing is withheld from the published source: NVIDIA sells NIM and enterprise support as separate products rather than gating features out of this one. Score corrected from 4 to 5 on 2026-07-30. In this ladder 4 pairs with open_core and never with open_source, and the recorded 4 was carrying the NVIDIA-GPU-only caveat, which is a hardware lock rather than a license restriction and answers no openness dimension.
- https://github.com/NVIDIA/TensorRT-LLM recorded 2026-06-04
flagship phase-C verification source
- https://github.com/NVIDIA/TensorRT-LLM/blob/main/LICENSE recorded 2026-06-04
flagship phase-C verification source
Adoption
3 high confidenceProduction inference optimizer/runtime for NVIDIA GPUs; used in MLPerf submissions (HGX H100 GPT-J ~3x gains). Described in 2026 as the 'specialist's tool', narrower adoption than vLLM/SGLang due to compilation overhead and NVIDIA-only scope. ~13.8k stars. Level 3 reflects significant but more specialized usage; no clean user/download count verified.
- https://github.com/NVIDIA/TensorRT-LLM recorded 2026-06-04
flagship phase-C verification source
- https://www.yottalabs.ai/post/best-llm-inference-engines-in-2026-vllm-tensorrt-llm-tgi-and-sglang-compared recorded 2026-06-04
flagship phase-C verification source
- https://theaiengineer.substack.com/p/vllm-vs-ollama-vs-sglang-vs-tensorrt recorded 2026-06-04
flagship phase-C verification source
Capability
5 high confidenceMaximum NVIDIA-hardware performance; featured in MLPerf Inference (HGX H100 ~3x GPT-J gain attributed to TensorRT-LLM). Top raw throughput/latency on NVIDIA silicon among 2026 contenders, at the cost of compilation overhead.
- https://www.hpcwire.com/2024/03/28/mlperf-inference-4-0-results-showcase-genai-nvidia-still-dominates/ recorded 2026-06-04
flagship phase-C verification source
- https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/ recorded 2026-06-04
flagship phase-C verification source
Unchanged since 2026-07-30 (last edited, not re-checked)