Why MPT-7B Mattered in 2023
When MosaicML released MPT-7B in May 2023, the open-weight LLM landscape was dominated by models with restrictive licenses. Llama required a research application. GPT-NeoX and Bloom had training code open but practical limitations. MPT-7B was trained entirely on public data, released under Apache 2.0, and designed with production inference in mind from day one.
ALiBi: Attention With Linear Biases
Most transformer models use learned positional embeddings or RoPE to encode token positions. MPT-7B uses ALiBi (Attention with Linear Biases), which adds a fixed linear penalty to attention scores based on distance rather than learning position embeddings. The penalty is -m × distance, where m is a per-head constant.
The key consequence: models trained with ALiBi can generalize to sequence lengths longer than their training context without fine-tuning. An MPT-7B trained on 2K context can be used at 4K or 8K context (with quality degradation, but not catastrophic failure). Positional embedding models typically break entirely outside their training context.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "mosaicml/mpt-7b-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_name,
trust_remote_code=True,
torch_dtype="auto",
device_map="auto",
)
prompt = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Summarize the key differences between REST and GraphQL APIs.
### Response:"""
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=300, temperature=0.1)
print(tokenizer.decode(output[0], skip_special_tokens=True))
