SAM 2: Meta's Real-Time Video Segmentation Model

SAM 2 vs SAM 1: The Video Gap

The original Segment Anything Model (SAM 1) was image-only. You could apply it frame-by-frame to video, but without temporal consistency — each frame started fresh, and the same object might get different masks across frames. Meta AI's SAM 2 solves this with a streaming memory bank that maintains object state across frames.

Key architectural additions:

• Memory encoder: Encodes frame features and masks into memory tokens
• Memory bank: Stores recent frames and prompted frames as spatial memories
• Memory attention: Cross-attends current frame features to memory bank before decoding

The result: prompt an object once at frame 0 (with a click or bounding box), and SAM 2 tracks and segments it through the rest of the video.

Prompting Modes

SAM 2 inherits SAM 1's flexible prompting:

• Points: Click to include (+) or exclude (-) regions
• Bounding boxes: Coarse box → precise mask
• Masks: Refine a previous mask
• Text (via integration with CLIP): Experimental natural language prompting

For video, you prompt on one or more keyframes and the model propagates forward and backward in time.

SA-V Dataset

HuggingFace's SAM 2 was trained on SA-V, a dataset 35x larger than SA-1B (the original SAM dataset). SA-V contains 51,000 videos with 643,000 masklet annotations — objects tracked across entire video sequences. The diversity spans sports, cooking, outdoor scenes, and social media content.

Python API

import torch
from sam2.build_sam import build_sam2_video_predictor

predictor = build_sam2_video_predictor(
    "sam2_hiera_large.yaml",
    "sam2_hiera_large.pt",
    device="cuda"
)

# Load video frames (as JPEG directory or tensor)
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(video_path="./video_frames/")

    # Prompt object at frame 0 with a point
    frame_idx, obj_ids, mask_logits = predictor.add_new_points_or_box(
        inference_state=state,
        frame_idx=0,
        obj_id=1,
        points=[[350, 200]],  # (x, y) in pixels
        labels=[1],           # 1=foreground, 0=background
    )

    # Propagate through video
    for frame_idx, obj_ids, mask_logits in predictor.propagate_in_video(state):
        masks = (mask_logits > 0.0).cpu().numpy()
        print(f"Frame {frame_idx}: {masks.sum()} pixels segmented")

Real-Time Performance

SAM 2 Hiera-L runs at ~45 FPS on an A100 GPU for 1024x1024 frames. The smaller Hiera-S variant achieves ~140 FPS. For real-time applications (30fps video), Hiera-B+ is sufficient on a consumer RTX 4080.

Use Cases

Video editing: Rotoscope a subject for background replacement without frame-by-frame masking.

Medical imaging: Track tumors or instruments across CT/MRI scan slices.

Robotics: Identify and track objects for manipulation tasks without domain-specific training.

Sports analytics: Track individual players through occlusion, camera cuts, and fast motion.

The SAM 2 GitHub includes demo notebooks for all major use cases and a Gradio web demo for rapid prototyping.