OLMo 2: Allen AI's Fully Open LLM (Weights + Data + Code)

Updated
7 min read
OLMo 2: Allen AI's Fully Open LLM (Weights + Data + Code)
TL;DR

OLMo 2 is the only major LLM where you can reproduce the entire training run: weights, 3T-token Dolma dataset, training code, and evaluation suite are all public.

True Openness vs. Open Weights

The word "open" in AI is heavily overloaded. Llama 3 releases model weights with a usage license - the training data, data processing pipeline, and training code remain proprietary. OLMo 2 from Allen AI is different: every component is public. You can download the weights, replicate the training data from Dolma, run the exact training code from the GitHub repository, and verify results with the same evaluation suite.

OLMo 2 Variants

Allen AI released two sizes: OLMo 2 7B (1124 checkpoint) and OLMo 2 13B. Both were trained on Dolma, a 3T-token open dataset built from Common Crawl, Wikipedia, books, GitHub code, and academic papers - all with documented filtering and deduplication pipelines.

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "allenai/OLMo-2-1124-7B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
)

inputs = tokenizer("The key difference between supervised and unsupervised learning is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

OLMo-Instruct Fine-Tunes

The base OLMo 2 models are pretrained only - not instruction-tuned. Allen AI also releases OLMo-2-Instruct fine-tunes trained on open instruction datasets, which are more practical for conversational applications while maintaining the full reproducibility guarantee.

Benchmark Performance

OLMo 2 7B outperforms Llama 3.1 8B on several reasoning and knowledge benchmarks (ARC-Challenge, HellaSwag, MMLU) while being comparable on coding. This is notable because Llama 3.1 had access to substantially more compute and a larger, curated (but private) training dataset.

The Dolma Dataset

Dolma is worth examining independently of OLMo. The 3T-token corpus is one of the largest fully documented and reproducible pretraining datasets available:

  • Common Crawl (cleaned with CCNet pipeline)
  • Wikipedia and Wikibooks (all languages, deduplicated)
  • Project Gutenberg (books with expired copyright)
  • OpenWebMath (mathematical text from the web)
  • RedPajama-v1 GitHub (code across 30+ languages)
  • Semantic Scholar (scientific papers)

Why Reproducibility Matters for Research

The scientific value of OLMo 2 is that researchers can run ablation studies on training data composition, compare different tokenization strategies, or test curriculum learning schedules without guessing what Meta or Google did. For AI safety research, interpretability work, and data contamination studies, having the full training pipeline available is essential.

#olmo-2#allen-ai#fully-open#dolma#training-data

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.