CogVLM2: Open-Source Video and Image Understanding With Long Context

Beyond Shared Attention Weights

Most vision-language models process visual and text tokens through the same attention and FFN layers. CogVLM2 takes a different approach: a Visual Expert Module adds a separate set of QKV projection weights and FFN weights exclusively for visual tokens. Text tokens are processed normally; visual tokens travel through both the shared weights and the expert weights, giving the model dedicated capacity for visual reasoning.

CogVLM2 Image: Resolution and Architecture

CogVLM2-Image (8B parameters) processes images at up to 1344×1344 pixels - among the highest native resolutions for a model in this size class. The visual expert runs on top of a Llama 3 8B language backbone, with a SigLIP vision encoder handling image tokenization.

import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("THUDM/cogvlm2-llama3-chat-19B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "THUDM/cogvlm2-llama3-chat-19B",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)

image = Image.open("screenshot.png").convert("RGB")
query = "What UI components are visible in this screenshot and what do they do?"

input_by_model = model.build_conversation_input_ids(
    tokenizer, query=query, images=[image], template_version="chat"
)
inputs = {
    "input_ids": input_by_model["input_ids"].unsqueeze(0).to(model.device),
    "token_type_ids": input_by_model["token_type_ids"].unsqueeze(0).to(model.device),
    "attention_mask": input_by_model["attention_mask"].unsqueeze(0).to(model.device),
    "images": [[input_by_model["images"][0].to(model.device).to(torch.bfloat16)]],
}
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)

CogVLM2-Video: Temporal Understanding

CogVLM2-Video extends the architecture to handle video input by sampling frames at regular intervals and embedding them as a sequence of visual token groups. The model attends across all frames simultaneously, enabling temporal reasoning: tracking object motion, identifying state changes, understanding cause-and-effect across frames.

Supported input formats include direct video files (MP4, AVI) via the video loading utilities in the CogVLM repository, or manually sampled frame tensors.

GLM-4 Language Backbone

The video variant uses a GLM-4 language backbone rather than Llama, providing stronger Chinese language capability alongside English - an advantage for teams building multilingual visual applications.

Benchmark Comparisons

CogVLM2-Image competes closely with InternVL2-8B and LLaVA-1.6-34B on OCRBench and document understanding tasks, while the Visual Expert Module gives it a measurable advantage on tasks requiring fine-grained visual attribute recognition. For video tasks, CogVLM2-Video outperforms open alternatives on EgoSchema and VideoChatGPT benchmarks.

Practical Use Cases

The high image resolution makes CogVLM2 particularly suited for: UI/UX feedback (screenshot analysis), medical image annotation, technical diagram interpretation, and retail product image extraction where small visual details matter.

Links

• HuggingFace model page
• CogVLM GitHub