The ARC-AGI benchmark tests abstract reasoning using visual pattern puzzles that humans solve easily but AI has historically failed at. Prior to o3, the best models scored around 32% (o1) and 34% (Claude 3.5).
o3 achieved 87.5% on the semi-private ARC-AGI evaluation - a jump so dramatic it made headlines and prompted the ARC Prize team to call it "a genuine breakthrough."
Benchmark Scores
Benchmark
o3
o1
GPT-4o
ARC-AGI
87.5%
32.0%
5.0%
AIME 2024
96.7%
74.4%
13.4%
SWE-Bench
71.7%
48.9%
38.0%
GPQA Diamond
87.7%
78.3%
53.6%
The SWE-Bench score of 71.7% is particularly significant - it means o3 can resolve nearly 3 in 4 real GitHub issues autonomously.
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
Unlike standard models where inference cost is fixed, o3 uses test-time compute scaling: you can tell it how much "thinking budget" to spend. More thinking = higher accuracy = higher cost.
Tasks where o3's performance gains don't justify the cost premium
Existing pipelines already tuned for o1 behavior
When you need faster time-to-first-token
API Access
o3 and o3-mini are available through the OpenAI API. Access for o3 (full) is currently prioritized for Tier 4-5 API users. o3-mini has broader availability.
o3 is significantly more expensive than GPT-4o. For most production applications, o3-mini at medium reasoning effort hits the best accuracy/cost balance. Run a cost analysis before deploying o3 at scale - the quality gains are real, but so is the bill.
Summary
o3 represents a step-change in AI reasoning capability, particularly for math, science, and complex coding. o3-mini with configurable compute tiers makes this capability accessible at reasonable cost. Track benchmark results and access details at the ARC Prize blog.
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
530
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
What Is OpenAI Frontier Models and Codex on AWS? A Practical Overview
OpenAI's frontier models and Codex are now available on AWS through Amazon Bedrock and SageMaker. This post covers what's included, how it works, and the practical tradeoffs for teams considering this integration.
