TensorRT-LLM

license

Apache-2.0(OSI, stated in LICENSE

source

public

core features withheld

yes — the default build links a prebuilt internal_cutlass_kernels static library that ships without source, and the package requires NVIDIA's proprietary nvidia-cutlass-dsl

caveat

runs only on NVIDIA GPUs (vendor-hardware lock, not a license restriction)

closed-component

internal_cutlass_kernels(a prebuilt static library unpacked from an in-tree tarball and linked unless a builder supplies its source)

The repository is Apache-2.0 and carries the C++ runtime and most kernels as source, but the default build does not come from that source alone. Unless a builder supplies the internal kernels' source, CMake unpacks a prebuilt internal_cutlass_kernels static library from the tree and links it, and the Python package requires NVIDIA's proprietary nvidia-cutlass-dsl. A closed component that the default build requires makes this open core. NVIDIA sells NIM and enterprise support as separate products beside it.

• https://github.com/NVIDIA/TensorRT-LLM recorded 2026-08-09

  TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.

• https://pypi.org/pypi/nvidia-cutlass-dsl/json recorded 2026-10-08

  PyPI metadata for nvidia-cutlass-dsl carries the classifier "License :: Other/Proprietary License".

• https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/cpp/tensorrt_llm/CMakeLists.txt recorded 2026-10-08

  cpp/tensorrt_llm/CMakeLists.txt, under "if(NOT INTERNAL_CUTLASS_KERNELS_PATH)", imports "tensorrt_llm_internal_cutlass_kernels_static" as a STATIC IMPORTED library unpacked from a tarball, failing with "The internal_cutlass_kernels library is truncated or incomplete." when the file is missing.

• https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/LICENSE recorded 2026-10-08

  Repository LICENSE: "This project is licensed under the Apache 2.0 license, whose full license text is available below."

• https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/requirements.txt recorded 2026-10-08

  requirements.txt lists "nvidia-cutlass-dsl[cu13]==4.8.0.dev0" as an install requirement.

• https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/setup.py recorded 2026-10-08

  setup.py parses requirements.txt into "required_deps" and passes "install_requires=required_deps".

Independent comparisons frame TensorRT-LLM as the specialist choice: the highest raw throughput on standardized NVIDIA fleets, but narrower reach than vLLM or SGLang because of its compilation overhead and single-vendor lock-in. No standalone user or download count exists, so the rating reflects that specialized, GPU-locked production use rather than a broad count.

• https://github.com/NVIDIA/TensorRT-LLM recorded 2026-08-09

  "stargazerCount":14342

• https://theaiengineer.substack.com/p/vllm-vs-ollama-vs-sglang-vs-tensorrt recorded 2026-09-29

  The article says TensorRT-LLM "squeezes maximum performance from NVIDIA hardware but demands 1-2 weeks of setup time and locks you into a single vendor."

• https://www.yottalabs.ai/post/best-llm-inference-engines-in-2026-vllm-tensorrt-llm-tgi-and-sglang-compared recorded 2026-09-29

  The page says "Because TensorRT-LLM is built specifically for NVIDIA hardware, it can achieve extremely high performance in environments where infrastructure is standardized around NVIDIA GPU clusters."

NVIDIA's MLPerf Inference v4.0 recap credits TensorRT-LLM with nearly 3x more throughput on the GPT-J test than the prior round. An independent H100 benchmark on Llama 3.3 70B puts it ahead of vLLM and SGLang on raw throughput, though with a cold start of about 28 minutes against roughly a minute for the others.

• https://developer.nvidia.com/blog/nvidia-h200-tensor-core-gpus-and-nvidia-tensorrt-llm-set-mlperf-llm-inference-records/ recorded 2026-09-29

  NVIDIA's MLPerf Inference v4.0 post has a section headed "TensorRT-LLM nearly triples LLM inference performance" and says H100 "nearly tripled performance on the GPT-J LLM test."

• https://www.hpcwire.com/2024/03/28/mlperf-inference-4-0-results-showcase-genai-nvidia-still-dominates/ recorded 2026-08-09

  delivered nearly 3x more performance on the GPT-J test compared to the prior round thanks to our TensorRT-LLM software

• https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/ recorded 2026-09-29

  The benchmark guide recommends TensorRT-LLM "if you have a single model in long-term production and throughput is paramount" and lists its engine build at "~28 minutes (engine compilation)"