Model components / Inference code

AWS Neuron

Proprietary SDK and runtime for running inference on AWS's custom Inferentia2 and Trainium chips. The Neuron Compiler converts models from PyTorch/JAX into optimized executables for AWS custom silicon. Offers significant cost savings vs GPU instances for supported model architectures. Integrated with SageMaker and EC2. AWS's play to break NVIDIA GPU dependency for inference workloads, custom silicon with a proprietary software stack.

Neuron SDK 2.30.0 (released ~May 22, 2026; 86 releases on GitHub). Serves LLM/DL inference on AWS Inferentia/Trainium accelerators (Inf2/Trn instances). Confirmed live June 2026.

Openness

1 high confidence

1.0

license: mixed/proprietary-core
runtime: proprietary(Neuron Runtime+Compiler not open sourced)
open-glue: Linux kernel driver(Apache-2.0),vLLM-Neuron integration(open, in vLLM org),NKI library, samples(MIT-0)
hardware-lock: AWS-Inferentia/Trainium-only

Headline inference path (Neuron Compiler + Neuron Runtime) is proprietary AWS, only runnable on AWS silicon; surrounding glue (driver, vLLM integration, NKI, samples) is open but does not make the engine open_source. Scored closed.

https://github.com/aws-neuron/aws-neuron-sdk recorded 2026-06-04
SDK repo lists samples/docs (MIT-0/Unknown) only; runtime+compiler not in an OSI-licensed repo; 2.30.0 release ~May 22 2026
https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/what-is-neuron.html recorded 2026-06-04
open components (driver Apache-2.0, vLLM integration) named; Compiler/Runtime absent from open list; vLLM-V1 inference on Inferentia/Trainium

Adoption

3 low confidence

3.0

Used across AWS Inf2/Trn fleet and integrated with HuggingFace/vLLM/PyTorch; no clean download or user count published. Level 3 reflects reported AWS-scale traction rather than a verified usage_volume figure.

https://aws.amazon.com/ai/machine-learning/neuron/ recorded 2026-06-04
Neuron positioned as production SDK for Inferentia/Trainium with framework integrations

Capability

4 medium confidence

4.0

Strong large-scale LLM inference feature set on AWS silicon (EP, disaggregated serving, spec decode). No MLPerf Inference figure surfaced for Neuron this run, so feature_matrix not benchmark. Not a frontier C5 (hardware-locked, narrower than vLLM/SGLang/TensorRT-LLM coverage).

https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/what-is-neuron.html recorded 2026-06-04
vLLM V1 with EP, disaggregated inference, speculative decoding on Trainium/Inferentia

Unchanged since 2026-06-24 (last edited, not re-checked)