MPT-7B: MosaicML's Commercial-Ready LLM With ALiBi Attention

Why MPT-7B Mattered in 2023

When MosaicML released MPT-7B in May 2023, the open-weight LLM landscape was dominated by models with restrictive licenses. Llama required a research application. GPT-NeoX and Bloom had training code open but practical limitations. MPT-7B was trained entirely on public data, released under Apache 2.0, and designed with production inference in mind from day one.

ALiBi: Attention With Linear Biases

Most transformer models use learned positional embeddings or RoPE to encode token positions. MPT-7B uses ALiBi (Attention with Linear Biases), which adds a fixed linear penalty to attention scores based on distance rather than learning position embeddings. The penalty is -m × distance, where m is a per-head constant.

The key consequence: models trained with ALiBi can generalize to sequence lengths longer than their training context without fine-tuning. An MPT-7B trained on 2K context can be used at 4K or 8K context (with quality degradation, but not catastrophic failure). Positional embedding models typically break entirely outside their training context.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "mosaicml/mpt-7b-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)

prompt = """Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Summarize the key differences between REST and GraphQL APIs.

### Response:"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=300, temperature=0.1)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training at Scale: 1T Tokens in 9.5 Days

MosaicML trained MPT-7B on 1 trillion tokens using 440 A100 GPUs in 9.5 days - an efficiency benchmark that influenced subsequent open model training runs. They used their own MosaicML Streaming Dataset library for efficient data loading and FlashAttention + Triton kernels for compute efficiency.

MPT Fine-Tunes

The base MPT-7B model spawned several practical variants:

• MPT-7B-Instruct - fine-tuned on databricks-dolly-15k and Anthropic HH for helpfulness
• MPT-7B-Chat - optimized for multi-turn conversation with commercial license
• MPT-7B-StoryWriter - fine-tuned for long-form narrative generation at 65K context

Commercial License Importance

The Apache 2.0 license was not a footnote - it was the point. Teams building products on Llama had legal uncertainty. MPT-7B gave companies a clear, commercial-safe foundation. This triggered a wave of similar releases and established "Apache 2.0" as a standard expectation for commercially useful open models.

Historical Context

MPT-7B held the top position on the HuggingFace Open LLM Leaderboard briefly in mid-2023 before being surpassed by Llama 2 and then the Mistral family. Its lasting contribution is establishing the template for efficient, commercially licensed open LLMs that the entire ecosystem now follows.

Links

• HuggingFace model page
• Databricks/MosaicML blog