Model components / Inference code

Groq Inference

Groq Inference runs open models on Groq's custom Language Processing Unit (LPU) hardware, built from the ground up for AI inference rather than adapted from a general-purpose GPU. Its deterministic, statically scheduled architecture keeps model weights in on-chip SRAM rather than in off-chip memory. The GroqCloud service exposes open models, including gpt-oss, Llama, and Whisper, through an OpenAI-compatible API.

Verified 2026-08-09 via the GroqDocs models and overview pages and Groq's LPU architecture blog.

Openness

1 high confidence

1.0

engine: proprietary(LPU stack)
source: closed
access: managed-cloud-API-only
license: Proprietary

GroqCloud is a managed, API-only service. The groq GitHub org publishes 29 repositories, among them the groq-python and groq-typescript client SDKs, the openbench and harbor eval tools, demo apps and infrastructure tooling, but no inference-engine repository, so the engine itself is unpublished. No license is granted to the underlying product either: the limited license in groq.com's terms of use covers only the marketing website, not GroqCloud or the LPU stack.

https://console.groq.com/docs/overview recorded 2026-08-09
Fast LLM inference, OpenAI-compatible. Simple to integrate, easy to scale. Start building in minutes.
https://github.com/orgs/groq/repositories recorded 2026-08-09
Full list of the groq GitHub org's 41 public repos, read across both pages of the listing (page 1: groq-python, groq-typescript, groq-mcp-server, openbench, harbor, groq-desktop-beta, dispatch, ...; page 2: groqflow, mlagility, groq-api-cookbook, groq-gradio, ...) - none is the LPU inference engine or serving runtime; all are client SDKs, demo apps, eval harnesses, model-build toolflow front ends, or infra tooling.
https://groq.com/terms-of-use recorded 2026-08-09
Subject to your compliance with these Terms, we grant you a limited, non-exclusive, non-transferable, license to access and use the Websites for your personal, non-commercial use only.

Adoption

4 medium confidence

4.0

The groq.com homepage names no enterprise customers, so the band rests on a vendor-reported developer count: a Groq press release claims "more than five million developers", which falls in the 1M-10M range. That figure is self-reported and not independently corroborated.

https://groq.com/newsroom/groq-raises-usd650m-to-scale-its-ai-inference-cloud-business recorded 2026-08-09
Today, Groq operates 13 data centers across North America, Europe, the Middle East and APAC. The company serves more than five million developers and thousands of AI-native companies that consume trillions of tokens each week.

Capability

5 medium confidence

5.0

On Artificial Analysis's gpt-oss-120b provider benchmark Groq measures 476.8 output tokens per second, ranking third of 18 API providers behind Cerebras at 1,942.0 and SambaNova at 705.7. That still places it firmly in the purpose-built-silicon speed class rather than among the GPU-hosted majority of the field.

https://artificialanalysis.ai/models/gpt-oss-120b/providers recorded 2026-08-09
the top providers are Cerebras (1,942.0 t/s), SambaNova (705.7 t/s), and Groq (476.8 t/s). Speed varies significantly across providers, with a 5011% difference between the fastest and slowest.

Verified 2026-08-09