SOLAR 10.7B: How Depth Upscaling Makes a 10B Model Beat 30B Models

Updated
7 min read
SOLAR 10.7B: How Depth Upscaling Makes a 10B Model Beat 30B Models
TL;DR

Upstage's SOLAR 10.7B uses depth upscaling - duplicating and fine-tuning Llama 2 layers - to create a model that outperforms 30B-class models on the HuggingFace leaderboard while remaining practical to serve.

What Is Depth Upscaling?

Depth upscaling is a model merging technique developed by Upstage. Instead of training a large model from scratch, it starts with a pretrained model (in this case Llama 2 13B), duplicates its middle layers, and fine-tunes the resulting larger model. The duplicate layers start with the same weights as the originals - a warm initialization that requires far less training compute than starting from random weights.

The result for SOLAR 10.7B:

  1. Take Llama 2 13B (32 transformer layers)
  2. Remove the last 8 layers
  3. Concatenate two copies of the first 24 layers (total: 48 layers)
  4. Fine-tune on high-quality data

The output has 10.7B parameters - slightly fewer than 13B because the embedding layer is shared - but depth that would normally require a 30B+ model to achieve.

HuggingFace Leaderboard Performance

When SOLAR 10.7B was released in December 2023, it entered the top-10 of the HuggingFace Open LLM Leaderboard despite being the smallest model in that tier. The key results at time of release:

5 rows · Click column to sort
Average (4-task)74.267.960.1
ARC66.567.359.9
HellaSwag88.187.381.3
MMLU65.568.964.2
TruthfulQA76.844.945.5

The TruthfulQA score (76.8%) is particularly striking - Llama 2 70B scores 44.9% on the same benchmark. This reflects the quality of fine-tuning data as much as architecture.

Korean and English Bilingual Strength

Upstage is a South Korean AI company, and SOLAR 10.7B was trained with strong Korean language data alongside English. This makes it notable among open-source models for Korean language tasks:

  • Korean MMLU: outperforms models twice its size that were not specifically trained for Korean
  • Korean instruction following: the instruct variant handles polite/formal Korean register correctly
  • Code-switching (Korean + English in same conversation): handled gracefully

Using the Instruct Variant

python
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_UPSTAGE_KEY",
    base_url="https://api.upstage.ai/v1/solar",
)

response = client.chat.completions.create(
    model="solar-1-mini-chat",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the depth upscaling technique in simple terms."},
    ],
)
print(response.choices[0].message.content)

Self-Hosting with Ollama

bash
ollama pull solar
ollama run solar

At 10.7B parameters, SOLAR runs comfortably on a machine with 16GB VRAM or 32GB unified memory (MacBook Pro M2). In Q4_K_M quantization it requires about 7GB, making it viable on consumer GPUs like the RTX 3080 10GB.

Apache 2.0 License

SOLAR 10.7B is licensed under Apache 2.0 - fully permissive for commercial use without attribution requirements or usage restrictions. This is an important distinction from Llama 2's custom license (which has user-count thresholds) and makes SOLAR suitable for building commercial products.

When to Choose SOLAR

  • You need a 10B model that punches above its weight class on English and Korean
  • You want Apache 2.0 commercial licensing without restrictions
  • You are running on hardware that fits 7 - 16GB VRAM
  • You want a model that demonstrates the depth upscaling technique for your own fine-tuning research
#solar#upstage#depth-upscaling#korean#open-source

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.