AI Potluck
Contribute
Back to Gap Map Model components / Inference code

Together Inference Engine

Together AI
closed / Overall score: 4.0(strong)

Together Inference Engine is a serving stack that runs open models with FlashAttention, adaptive speculative decoding (ATLAS), and custom CUDA kernels. It offers serverless endpoints across a large catalog of open models alongside dedicated instances that reserve GPU capacity for a single model. Together AI builds it, and its team also publishes open kernel research including FlashAttention and ThunderKittens.

Openness

1 high confidence
1.0
engine
proprietary(Together Inference Engine)
source
closed(though Together publishes OSS research e.g. FlashAttention/ThunderKittens separately)
access
managed-cloud-API-only
license
Proprietary

No self-hostable implementation of the Together Inference Engine exists; its dedicated and serverless plans offer only managed cloud API access to "Together's proprietary inference engine." Its terms of service grant just a non-sublicensable right to access the service, and its open-source research, such as FlashAttention and ThunderKittens, is separate from the engine itself.

  • https://www.together.ai/dedicated-model-inference recorded 2026-09-23

    "Reserved, isolated GPU endpoints running Together's proprietary inference engine for production workloads."

  • https://www.together.ai/inference recorded 2026-09-23

    "Pay-per-use access to 200+ open-source models across chat, image, video, code, and voice modalities. OpenAI-compatible API, sub-second latency, adaptive speculative decoding, and batch processing at up to 50% lower cost. No infrastructure to manage."

  • https://www.together.ai/terms-of-service recorded 2026-09-23

    "the Company hereby grants you a non-exclusive, non-sublicensable right and license to use the Company IP as permitted by this Agreement. Company reserves all of its intellectual property and other proprietary rights not expressly granted to you herein."

Adoption

4 medium confidence
4.0

Together's homepage features a named-customer case study on Cursor and a hero logo row naming Salesforce, ElevenLabs, DeepMind, Mozilla and Decagon. No developer or user count is disclosed, so adoption is read from that reported enterprise breadth rather than a measured figure.

  • https://www.together.ai/ recorded 2026-09-29

    Homepage carries the story-card title "How Cursor partnered with Together AI to deliver real-time, low-latency inference at scale" and hero logo images "salesforce_black.svg", "elevenlabs_black.svg", "deepmind_black.svg", "mozilla_black.svg" and "decagon_black.svg".

Capability

4 low confidence
4.0

Together's FlashAttention-4 and ATLAS posts report attention kernels faster than cuDNN and Triton and up to 4x faster LLM inference. Both are Together's own benchmarks, and no independent benchmark is cited for its serving.

Verified 2026-09-23