MusicGen: Meta's Text-to-Music Model That Runs Locally
Meta's MusicGen generates 30-second music clips from text descriptions or melody conditioning using an EnCodec audio tokenizer and autoregressive transformer - fully open and self-hostable.
Mahmudul Haque Qudrati
CEO & ML Engineer
May 15, 2026
7 min read
One AI engineering post, weekly
LLM benchmarks, prompt techniques, and token-cost breakdowns — not another AI news roundup.
Suno and Udio produce impressive music but are closed APIs with usage limits and licensing ambiguity. MusicGen from Meta's FAIR team is fully open: Apache 2.0 license, self-hostable on a single GPU, and capable of high-quality instrumental generation from text prompts or melody conditioning.
Architecture: EnCodec + Transformer
MusicGen uses a two-stage approach:
-
EnCodec - Meta's audio codec model converts raw audio waveforms into discrete tokens across multiple codebooks (4 - 8 codebooks at different quality levels). A 30-second clip at 32 kHz becomes a sequence of approximately 1500 tokens per codebook.
-
Transformer decoder - an autoregressive model generates the EnCodec token sequence conditioned on text embeddings (from a frozen T5 encoder). The model generates all codebooks in a single forward pass using a "delay pattern" that interleaves tokens from different codebooks.
from transformers import AutoProcessor, MusicgenForConditionalGeneration
import torch
import scipy.io.wavfile
processor = AutoProcessor.from_pretrained("facebook/musicgen-stereo-large")
model = MusicgenForConditionalGeneration.from_pretrained(
"facebook/musicgen-stereo-large",
torch_dtype=torch.float16,
).to("cuda")
descriptions = [
"upbeat jazz piano trio, 120 BPM, walking bass, brushed drums, no vocals",
"ambient electronic, slow evolving pads, 60 BPM, cinematic, minor key",
]
inputs = processor(text=descriptions, padding=True, return_tensors="pt").to("cuda")
# Generate 15 seconds of audio (256 tokens ≈ 5 seconds, so 750 for 15s)
audio_values = model.generate(**inputs, max_new_tokens=750)
# Save as WAV
sampling_rate = model.config.audio_encoder.sampling_rate # 32000
for i, audio in enumerate(audio_values.cpu().numpy()):
scipy.io.wavfile.write(f"output_{i}.wav", rate=sampling_rate, data=audio.T)
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
Melody Conditioning
MusicGen can continue or harmonize with an existing melody:
from transformers import AutoProcessor, MusicgenForConditionalGeneration
import torchaudio
processor = AutoProcessor.from_pretrained("facebook/musicgen-melody")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-melody").to("cuda")
# Load reference melody
melody_waveform, sr = torchaudio.load("melody_reference.wav")
inputs = processor(
audio=melody_waveform.numpy(),
sampling_rate=sr,
text=["orchestral arrangement, strings, cinematic"],
return_tensors="pt",
padding=True,
).to("cuda")
audio_values = model.generate(**inputs, max_new_tokens=500)
Model Sizes and Quality Trade-offs
| Variant | Parameters | VRAM | Quality |
|---|---|---|---|
| musicgen-small | 300M | ~4GB | Adequate for prototyping |
| musicgen-medium | 1.5B | ~8GB | Good for most use cases |
| musicgen-large | 3.3B | ~16GB | High quality |
| musicgen-stereo-large | 3.3B | ~16GB | Stereo output, recommended |
MusicGen vs Suno/Udio
Suno and Udio generate vocals and lyrics in addition to instrumentals - MusicGen is instrumental only. On pure instrumental quality, MusicGen large is competitive with early Suno but falls short of Suno v3/v4. The trade-offs: MusicGen is open, self-hostable, and has no usage limits or licensing concerns about using generated music commercially.
Real Use Cases
Background music for YouTube videos (avoiding copyright claims), game soundtrack generation for indie developers, podcast intro/outro generation, and rapid music prototyping for composers who want to hear a harmonic idea before writing notation.
Links
#musicgen#meta#music-generation#audio-ai#encodec
Mahmudul Haque Qudrati
CEO & ML Engineer
Visionary leader with extensive experience in machine learning and software development. Drives strategic innovation and business growth.
More from Mahmudul
GPT-6 Astra AGI: How to Actually Use It and Get Its Best Performance
8 min read
DeepSeek vs MiniMax vs Kimi in Coding Benchmarks: Real Scores, Real Prices, Real Tradeoffs
8 min read
GPT-6 Astra Benchmarks, Pricing, and Safety: What Developers Need to Know
5 min read
Claude Fable 5.1: Same Price, 75% Cheaper Cache, and Real Tradeoffs
4 min read
Related Articles
7 min readONNX: Export Any ML Model and Run It Anywhere
ONNX (Open Neural Network Exchange) is the universal model format - export from PyTorch, scikit-learn, or HuggingFace and run 3x faster inference with ONNX Runtime on CPU or GPU.
9 min readGradient Descent Explained: How Machine Learning Models Actually Learn
Gradient descent is the engine behind every modern ML model. Here is how it works, why learning rate matters, and when to use Adam over SGD.
10 min readSupervised Learning Explained: How Models Learn from Labeled Examples
Supervised learning is the most widely used ML paradigm. Here is exactly how the train-measure-adjust loop works, where labels come from, and when the approach breaks down.
// discussion
Comments