OLMo 2: Allen AI's Fully Open LLM (Weights + Data + Code)

OLMo 2 is the only major LLM where you can reproduce the entire training run: weights, 3T-token Dolma dataset, training code, and evaluation suite are all public.

Mahmudul Haque Qudrati

CEO & ML Engineer

April 16, 2026

7 min read

// tags

#olmo-2#allen-ai#fully-open#dolma#training-data

FIG. ART-20

7 min read

“

OLMo 2: Allen AI's Fully Open LLM (Weights + Data + Code)

// reading plan

sections

369

words

min read

// Machine Learning

Data Labeling Guide: How to Annotate Training Data Efficiently and Accurately

Labeling tools, quality control with inter-annotator agreement, active learning to cut labeling costs by 60-80%, and programmatic labeling with Snorkel.

9 min read

// Machine Learning

Building a Dataset for ML: What Makes Good Training Data

True Openness vs. Open Weights

The word "open" in AI is heavily overloaded. Llama 3 releases model weights with a usage license — the training data, data processing pipeline, and training code remain proprietary. OLMo 2 from Allen AI is different: every component is public. You can download the weights, replicate the training data from Dolma, run the exact training code from the GitHub repository, and verify results with the same evaluation suite.

OLMo 2 Variants

Allen AI released two sizes: OLMo 2 7B (1124 checkpoint) and OLMo 2 13B. Both were trained on Dolma, a 3T-token open dataset built from Common Crawl, Wikipedia, books, GitHub code, and academic papers — all with documented filtering and deduplication pipelines.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "allenai/OLMo-2-1124-7B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
)

inputs = tokenizer("The key difference between supervised and unsupervised learning is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

OLMo-Instruct Fine-Tunes

The base OLMo 2 models are pretrained only — not instruction-tuned. Allen AI also releases OLMo-2-Instruct fine-tunes trained on open instruction datasets, which are more practical for conversational applications while maintaining the full reproducibility guarantee.

Benchmark Performance

OLMo 2 7B outperforms Llama 3.1 8B on several reasoning and knowledge benchmarks (ARC-Challenge, HellaSwag, MMLU) while being comparable on coding. This is notable because Llama 3.1 had access to substantially more compute and a larger, curated (but private) training dataset.

The Dolma Dataset

Dolma is worth examining independently of OLMo. The 3T-token corpus is one of the largest fully documented and reproducible pretraining datasets available:

Common Crawl (cleaned with CCNet pipeline)
Wikipedia and Wikibooks (all languages, deduplicated)
Project Gutenberg (books with expired copyright)
OpenWebMath (mathematical text from the web)
RedPajama-v1 GitHub (code across 30+ languages)
Semantic Scholar (scientific papers)

Why Reproducibility Matters for Research

The scientific value of OLMo 2 is that researchers can run ablation studies on training data composition, compare different tokenization strategies, or test curriculum learning schedules without guessing what Meta or Google did. For AI safety research, interpretability work, and data contamination studies, having the full training pipeline available is essential.

OLMo 2: Allen AI's Fully Open LLM (Weights + Data + Code)

Related Articles

Data Labeling Guide: How to Annotate Training Data Efficiently and Accurately

Building a Dataset for ML: What Makes Good Training Data

True Openness vs. Open Weights

OLMo 2 Variants

OLMo-Instruct Fine-Tunes

Benchmark Performance

The Dolma Dataset

Why Reproducibility Matters for Research

Links

The workspace your team
actually needs

AI & ML insights, weekly

Mahmudul Haque Qudrati

Ollama Complete Guide 2026: Run Any LLM Locally in 5 Minutes

OLMo 2: Allen AI's Fully Open LLM (Weights + Data + Code)

Related Articles

Data Labeling Guide: How to Annotate Training Data Efficiently and Accurately

Building a Dataset for ML: What Makes Good Training Data

True Openness vs. Open Weights

OLMo 2 Variants

OLMo-Instruct Fine-Tunes

Benchmark Performance

The Dolma Dataset

Why Reproducibility Matters for Research

Links

The workspace your teamactually needs

AI & ML insights, weekly

Mahmudul Haque Qudrati

Ollama Complete Guide 2026: Run Any LLM Locally in 5 Minutes

The workspace your team
actually needs