Phi-3 Vision: Microsoft's 4.2B Multimodal Model for Edge Devices

Updated
7 min read
Phi-3 Vision: Microsoft's 4.2B Multimodal Model for Edge Devices
TL;DR

Phi-3 Vision packs chart understanding, document analysis, and image reasoning into 4.2 billion parameters - small enough to run on a mobile device with CoreML or ONNX, yet scoring 59.8% on MMMU.

Why Size Matters: The Edge Deployment Case

Most multimodal models require 7B+ parameters to achieve useful performance on chart and document understanding. Phi-3 Vision achieves competitive results at 4.2B total parameters (3.8B language model + 0.4B CLIP vision encoder), making it viable for:

  • On-device inference on iPhone (Apple Neural Engine via CoreML)
  • Android deployment (ONNX Runtime with NNAPI)
  • Embedded systems with limited VRAM
  • Air-gapped environments where data cannot leave the device

Benchmark Numbers

Click any column to sort
4 rows
MMMU59.8%36.2%75.1%
TextVQA70.9%58.2%78.0%
DocVQA82.0%29.0%87.2%
ChartQA81.4%18.2%78.1%

The most striking result is DocVQA (document visual question answering): 82.0% vs LLaVA-1.5 7B's 29.0%. Phi-3 Vision's training data included heavy emphasis on documents and charts, which is exactly the content mix that enterprise users encounter most frequently.

On ChartQA, Phi-3 Vision (81.4%) actually outperforms GPT-4V (78.1%) - remarkable for a model one-tenth the size.

Running Locally

python
from PIL import Image
import requests
from transformers import AutoModelForCausalLM, AutoProcessor
import torch

model_id = "microsoft/Phi-3-vision-128k-instruct"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="cuda",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    _attn_implementation="eager",
)

# Load a chart image
image = Image.open("quarterly_revenue_chart.png")

messages = [
    {"role": "user", "content": "<|image_1|>\nWhat is the revenue trend shown in this chart? Provide specific numbers."},
]

prompt = processor.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(prompt, [image], return_tensors="pt").to("cuda")

output = model.generate(**inputs, max_new_tokens=512, temperature=0.0)
print(processor.decode(output[0], skip_special_tokens=True))

CoreML Export for iOS

bash
# Export to CoreML format for on-device iOS inference
pip install coremltools

python -c "
import coremltools as ct
# Microsoft provides pre-exported CoreML packages via the ONNX Runtime iOS package
# See: https://github.com/microsoft/onnxruntime
"

Microsoft maintains official ONNX and CoreML exports of the Phi-3 family. The iOS package handles tokenization, image preprocessing, and the vision encoder - you write Swift code that calls the exported model package.

Azure AI Integration

Phi-3 Vision is available as a managed endpoint on Azure AI Studio with serverless deployment (pay-per-token, no reserved capacity):

python
import os
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import SystemMessage, UserMessage, ImageContentItem, TextContentItem
from azure.core.credentials import AzureKeyCredential

client = ChatCompletionsClient(
    endpoint=os.environ["AZURE_AI_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["AZURE_AI_KEY"]),
)

response = client.complete(
    messages=[
        UserMessage(content=[
            TextContentItem(text="Extract all data points from this chart as a table:"),
            ImageContentItem(image_url={"url": "https://your-storage.blob.core.windows.net/chart.png"}),
        ])
    ],
    model="Phi-3-vision-128k-instruct",
)
print(response.choices[0].message.content)

When to Choose Phi-3 Vision

  • You need multimodal capability on a device with less than 8GB VRAM
  • Document and chart understanding is the primary use case
  • You need on-device inference for privacy (no data leaves the device)
  • You want Azure-native deployment with Microsoft's compliance certifications
#phi-3-vision#microsoft#multimodal#edge#chart-understanding

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.