CogVLM2: Open-Source Video and Image Understanding With Long Context

Updated
7 min read
CogVLM2: Open-Source Video and Image Understanding With Long Context
TL;DR

Zhipu AI's CogVLM2 introduces a Visual Expert Module that gives visual tokens their own weight matrices, enabling richer image and video understanding than shared-weight alternatives.

Beyond Shared Attention Weights

Most vision-language models process visual and text tokens through the same attention and FFN layers. CogVLM2 takes a different approach: a Visual Expert Module adds a separate set of QKV projection weights and FFN weights exclusively for visual tokens. Text tokens are processed normally; visual tokens travel through both the shared weights and the expert weights, giving the model dedicated capacity for visual reasoning.

CogVLM2 Image: Resolution and Architecture

CogVLM2-Image (8B parameters) processes images at up to 1344×1344 pixels - among the highest native resolutions for a model in this size class. The visual expert runs on top of a Llama 3 8B language backbone, with a SigLIP vision encoder handling image tokenization.

python
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("THUDM/cogvlm2-llama3-chat-19B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "THUDM/cogvlm2-llama3-chat-19B",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)

image = Image.open("screenshot.png").convert("RGB")
query = "What UI components are visible in this screenshot and what do they do?"

input_by_model = model.build_conversation_input_ids(
    tokenizer, query=query, images=[image], template_version="chat"
)
inputs = {
    "input_ids": input_by_model["input_ids"].unsqueeze(0).to(model.device),
    "token_type_ids": input_by_model["token_type_ids"].unsqueeze(0).to(model.device),
    "attention_mask": input_by_model["attention_mask"].unsqueeze(0).to(model.device),
    "images": [[input_by_model["images"][0].to(model.device).to(torch.bfloat16)]],
}
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)

CogVLM2-Video: Temporal Understanding

CogVLM2-Video extends the architecture to handle video input by sampling frames at regular intervals and embedding them as a sequence of visual token groups. The model attends across all frames simultaneously, enabling temporal reasoning: tracking object motion, identifying state changes, understanding cause-and-effect across frames.

Supported input formats include direct video files (MP4, AVI) via the video loading utilities in the CogVLM repository, or manually sampled frame tensors.

GLM-4 Language Backbone

The video variant uses a GLM-4 language backbone rather than Llama, providing stronger Chinese language capability alongside English - an advantage for teams building multilingual visual applications.

Benchmark Comparisons

CogVLM2-Image competes closely with InternVL2-8B and LLaVA-1.6-34B on OCRBench and document understanding tasks, while the Visual Expert Module gives it a measurable advantage on tasks requiring fine-grained visual attribute recognition. For video tasks, CogVLM2-Video outperforms open alternatives on EgoSchema and VideoChatGPT benchmarks.

Practical Use Cases

The high image resolution makes CogVLM2 particularly suited for: UI/UX feedback (screenshot analysis), medical image annotation, technical diagram interpretation, and retail product image extraction where small visual details matter.

#cogvlm2#video-understanding#zhipu-ai#vlm#multimodal

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.