Together AI: Run 200+ Open Models via OpenAI-Compatible API
Together AI provides serverless inference for 200+ open-source models including Llama 3.1 405B at $3.50/1M tokens, with fine-tuning, batch jobs, and an OpenAI-compatible SDK.
Together AI aggregates 200+ open-source models - Llama, Mistral, Qwen, Gemma, DeepSeek, and more - behind a single OpenAI-compatible API. Rather than managing GPU infrastructure or maintaining separate integrations per model, you switch models with a one-line change.
This is particularly useful for teams that want to:
Compare multiple models on the same task without re-engineering
Access large models (405B+) that require multi-node GPU clusters
Fine-tune open models without managing training infrastructure
Run batch inference jobs at lower cost than real-time endpoints
Serverless vs Dedicated Endpoints
Serverless: Pay per token, no provisioning. Cold starts possible on less popular models. Best for development and variable workloads.
Dedicated: Reserve GPU capacity for consistent latency. Required for SLA-sensitive production. Priced per GPU-hour.
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
from together import Together
client = Together(api_key="your-together-api-key")
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain the CAP theorem in simple terms."}
],
max_tokens=1024,
temperature=0.7,
)
print(response.choices[0].message.content)
OpenAI SDK Drop-In Replacement
Together AI is compatible with the OpenAI Python SDK - just change base_url and api_key:
from openai import OpenAI
client = OpenAI(
api_key="your-together-api-key",
base_url="https://api.together.xyz/v1"
)
# Now use any Together model with the familiar OpenAI interface
response = client.chat.completions.create(
model="Qwen/Qwen2.5-72B-Instruct-Turbo",
messages=[{"role": "user", "content": "Translate to French: Hello, how are you?"}]
)
This means zero code changes for teams migrating from OpenAI - just swap credentials and model names.
Training data must be in OpenAI's JSONL format (messages array per line). Fine-tuned models are private and available immediately after training completes.
Batch Inference
For offline workloads (nightly processing, dataset annotation, bulk translation), batch jobs are cheaper than real-time and don't count against rate limits:
For Llama 3.1 8B at high volume, Groq wins on price. For 405B or Qwen 2.5, Together AI is often the only option with good availability.
FlashAttention and Performance
Together AI's infrastructure uses FlashAttention and continuous batching by default - you get optimized throughput without configuration. The full model list shows available models, context sizes, and pricing.
Summary
Together AI is the most complete open-model inference platform: 200+ models, OpenAI compatibility, fine-tuning, batch processing, and dedicated endpoints. For teams building on open-source models, it eliminates the need to manage GPU infrastructure. Start at together.ai and explore the full API at docs.together.ai.
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
530
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
Open Code Review – An AI-powered code review CLI tool: A Practical Overview
Open Code Review is an open-source CLI tool from Alibaba that uses AI to review code changes. It runs locally, supports multiple LLMs, and costs about $0.01 per review. Here's a practical breakdown.
