The ARC-AGI benchmark tests abstract reasoning using visual pattern puzzles that humans solve easily but AI has historically failed at. Prior to o3, the best models scored around 32% (o1) and 34% (Claude 3.5).
o3 achieved 87.5% on the semi-private ARC-AGI evaluation - a jump so dramatic it made headlines and prompted the ARC Prize team to call it "a genuine breakthrough."
Benchmark Scores
Benchmark
o3
o1
GPT-4o
ARC-AGI
87.5%
32.0%
5.0%
AIME 2024
96.7%
74.4%
13.4%
SWE-Bench
71.7%
48.9%
38.0%
GPQA Diamond
87.7%
78.3%
53.6%
The SWE-Bench score of 71.7% is particularly significant - it means o3 can resolve nearly 3 in 4 real GitHub issues autonomously.
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
Unlike standard models where inference cost is fixed, o3 uses test-time compute scaling: you can tell it how much "thinking budget" to spend. More thinking = higher accuracy = higher cost.
Tasks where o3's performance gains don't justify the cost premium
Existing pipelines already tuned for o1 behavior
When you need faster time-to-first-token
API Access
o3 and o3-mini are available through the OpenAI API. Access for o3 (full) is currently prioritized for Tier 4-5 API users. o3-mini has broader availability.
o3 is significantly more expensive than GPT-4o. For most production applications, o3-mini at medium reasoning effort hits the best accuracy/cost balance. Run a cost analysis before deploying o3 at scale - the quality gains are real, but so is the bill.
Summary
o3 represents a step-change in AI reasoning capability, particularly for math, science, and complex coding. o3-mini with configurable compute tiers makes this capability accessible at reasonable cost. Track benchmark results and access details at the ARC Prize blog.
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
538
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
What Is GPT-5.6 Sol Ultra Will Be in Codex? A Practical Overview
GPT-5.6 Sol Ultra is a rumored model optimized for code generation, integrated into Codex. We analyze the claims, potential capabilities, and what developers should expect.
