SOLAR 10.7B: How Depth Upscaling Makes a 10B Model Beat 30B Models

What Is Depth Upscaling?

Depth upscaling is a model merging technique developed by Upstage. Instead of training a large model from scratch, it starts with a pretrained model (in this case Llama 2 13B), duplicates its middle layers, and fine-tunes the resulting larger model. The duplicate layers start with the same weights as the originals - a warm initialization that requires far less training compute than starting from random weights.

The result for SOLAR 10.7B:

1. Take Llama 2 13B (32 transformer layers)
2. Remove the last 8 layers
3. Concatenate two copies of the first 24 layers (total: 48 layers)
4. Fine-tune on high-quality data

The output has 10.7B parameters - slightly fewer than 13B because the embedding layer is shared - but depth that would normally require a 30B+ model to achieve.

HuggingFace Leaderboard Performance

When SOLAR 10.7B was released in December 2023, it entered the top-10 of the HuggingFace Open LLM Leaderboard despite being the smallest model in that tier. The key results at time of release:

The TruthfulQA score (76.8%) is particularly striking - Llama 2 70B scores 44.9% on the same benchmark. This reflects the quality of fine-tuning data as much as architecture.

Korean and English Bilingual Strength

Upstage is a South Korean AI company, and SOLAR 10.7B was trained with strong Korean language data alongside English. This makes it notable among open-source models for Korean language tasks:

• Korean MMLU: outperforms models twice its size that were not specifically trained for Korean
• Korean instruction following: the instruct variant handles polite/formal Korean register correctly
• Code-switching (Korean + English in same conversation): handled gracefully

Using the Instruct Variant

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_UPSTAGE_KEY",
    base_url="https://api.upstage.ai/v1/solar",
)

response = client.chat.completions.create(
    model="solar-1-mini-chat",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the depth upscaling technique in simple terms."},
    ],
)
print(response.choices[0].message.content)

Self-Hosting with Ollama

ollama pull solar
ollama run solar

At 10.7B parameters, SOLAR runs comfortably on a machine with 16GB VRAM or 32GB unified memory (MacBook Pro M2). In Q4_K_M quantization it requires about 7GB, making it viable on consumer GPUs like the RTX 3080 10GB.

Apache 2.0 License

SOLAR 10.7B is licensed under Apache 2.0 - fully permissive for commercial use without attribution requirements or usage restrictions. This is an important distinction from Llama 2's custom license (which has user-count thresholds) and makes SOLAR suitable for building commercial products.

When to Choose SOLAR

• You need a 10B model that punches above its weight class on English and Korean
• You want Apache 2.0 commercial licensing without restrictions
• You are running on hardware that fits 7 - 16GB VRAM
• You want a model that demonstrates the depth upscaling technique for your own fine-tuning research

Links

• HuggingFace model page
• SOLAR technical paper
• Upstage AI