SAM 2: Architecture, Video Segmentation Memory & Production Implementation Guide

From Static Images to Streaming Video: The Evolution of Segment Anything

When Meta AI published the original Segment Anything Model (SAM 1) in 2023, it revolutionized zero-shot image segmentation. With a single point, bounding box, or freeform prompt, SAM 1 isolated objects with unprecedented boundary accuracy.

Yet, when computer vision engineers attempted to deploy SAM 1 into production video pipelines - such as surgical robotics, autonomous vehicle perception, or VFX rotoscoping - they immediately ran into a fundamental architectural wall: SAM 1 had no concept of time.

Applying SAM 1 frame-by-frame resulted in severe temporal flicker, object identity loss during occlusions, and staggering computational costs.

Segment Anything Model 2 (SAM 2) resolves this gap by introducing a unified foundation model designed from the ground up for both static images and streaming video. By coupling real-time vision transformers with a streaming memory bank and memory attention mechanism, SAM 2 tracks objects across thousands of video frames in real time, recovering target entities even after prolonged visual occlusions.

In this evergreen technical guide, we break down the internal architecture of SAM 2, explore how its memory attention mechanism maintains temporal consistency, provide a complete Python video inference implementation, and benchmark performance across model variants.

1. Architectural Foundation: How SAM 2 Works

Unlike hybrid tracking frameworks that bolt an optical flow or Kalman filter tracker onto a static segmentation model, SAM 2 treats images as single-frame videos. The architecture consists of five core components:

   +-------------------------------------------------------+
   |                  SAM 2 Architecture                   |
   +-------------------------------------------------------+
        [ Video Frame Input ]  ---> [ Image Encoder (Hiera) ]
                                          |
                                          v
   [ User Prompt (Point/Box) ] ---> [ Prompt Encoder ]
                                          |
                                          v
   [ Past Frame Memories ]    ---> [ Memory Attention ]
                                          |
                                          v
                                   [ Mask Decoder ]
                                     /          \
                                    v            v
                           [ Output Mask ]  [ Memory Encoder ]
                                                 |
                                                 v
                                        [ Memory Bank (FIFO) ]

1. Image Encoder (Hiera Backbone)

SAM 2 replaces the heavy ViT (Vision Transformer) backbone of SAM 1 with Hiera, a hierarchical vision transformer that is both faster and more parameter-efficient. Hiera processes frames at multiscale feature levels, outputting compact spatial tokens to downstream modules.

2. Prompt Encoder

Accepts sparse prompts (positive/negative foreground clicks, bounding boxes) or dense prompts (masks). It maps coordinates into continuous embedding spaces using Fourier positional encodings.

3. Memory Attention

The intellectual heart of SAM 2. The memory attention module conditions current frame visual features against historical representations stored in the memory bank using stacked cross-attention blocks.

4. Mask Decoder

Synthesizes the prompt embeddings and conditioned frame representations to predict target segmentation masks and an associated IoU confidence score.

5. Memory Encoder & Memory Bank

Whenever a frame is segmented, the memory encoder compresses the predicted mask and visual features into a low-dimensional memory token. This token is written into a FIFO (First-In, First-Out) Memory Bank, alongside a set of compact object pointer tokens.

2. Temporal Consistency: The Memory Attention Mechanism

How does SAM 2 maintain identity coherence when an object turns around, changes shape, or gets blocked by another entity?

Handling Dynamic Occlusions

When an object is occluded, SAM 2 does not discard its representation:

1. Historical Keyframe Persistence: The memory bank retains representations from early prompt frames as permanent anchors, while continuously sliding recent temporal frames through FIFO buffers.
2. Occlusion Prediction Head: SAM 2 contains an auxiliary head that explicitly predicts whether the target object is visible in the current frame. If an object is hidden behind a wall, SAM 2 suppresses mask emission rather than guessing incorrectly.
3. Re-Identification: As soon as the object reappears, the memory cross-attention layers detect matching spatial features against the persistent memory tokens, instantly resuming accurate segmentation without requiring manual user re-prompting.

3. Production Python Implementation: Step-by-Step

Let's implement a complete video segmentation pipeline using the official Segment Anything 2 Python SDK.

Prerequisites & Installation

pip install torch torchvision
pip install git+https://github.com/facebookresearch/segment-anything-2.git
pip install opencv-python decord

Video Segmentation Script

# sam2_video_pipeline.py
import torch
import numpy as np
import cv2
import os
from sam2.build_sam import build_sam2_video_predictor

# 1. Initialize device and predictor
device = "cuda" if torch.cuda.is_available() else "cpu"
checkpoint = "checkpoints/sam2_hiera_large.pt"
model_cfg = "sam2_hiera_l.yaml"

predictor = build_sam2_video_predictor(model_cfg, checkpoint, device=device)

# 2. Initialize video state from directory of extracted frames
video_dir = "./extracted_frames" # Directory containing 00000.jpg, 00001.jpg, etc.
inference_state = predictor.init_state(video_path=video_dir)

# 3. Add prompt on Keyframe 0 (Target object at coordinates x=450, y=320)
ann_frame_idx = 0  # Initial frame
ann_obj_id = 1     # Tracking Object ID 1

points = np.array([[450, 320]], dtype=np.float32)
labels = np.array([1], dtype=np.int32) # 1 = foreground click, 0 = background click

_, out_obj_ids, out_mask_logits = predictor.add_new_points_or_box(
    inference_state=inference_state,
    frame_idx=ann_frame_idx,
    obj_id=ann_obj_id,
    points=points,
    labels=labels,
)

print(f"✅ Initialized prompt on frame {ann_frame_idx}. Mask generated.")

# 4. Propagate segmentation across the entire video timeline
print("🚀 Propagating masks across video frames...")
video_segments = {} # frame_idx -> {obj_id -> mask}

for out_frame_idx, out_obj_ids, out_mask_logits in predictor.propagate_in_video(inference_state):
    video_segments[out_frame_idx] = {
        out_obj_id: (out_mask_logits[i] > 0.0).cpu().numpy()
        for i, out_obj_id in enumerate(out_obj_ids)
    }

print(f"🎉 Successfully segmented {len(video_segments)} frames.")

# 5. Export annotated video with OpenCV
frame_files = sorted([f for f in os.listdir(video_dir) if f.endswith('.jpg')])
first_frame = cv2.imread(os.path.join(video_dir, frame_files[0]))
h, w, _ = first_frame.shape

out = cv2.VideoWriter("segmented_output.mp4", cv2.VideoWriter_fourcc(*'mp4v'), 30, (w, h))

for idx, frame_file in enumerate(frame_files):
    frame = cv2.imread(os.path.join(video_dir, frame_file))
    if idx in video_segments:
        mask = video_segments[idx][ann_obj_id][0] # Boolean 2D mask
        
        # Apply semi-transparent green overlay
        colored_mask = np.zeros_like(frame)
        colored_mask[mask] = [0, 255, 0]
        frame = cv2.addWeighted(frame, 1.0, colored_mask, 0.45, 0)

    out.write(frame)

out.release()
print("🎬 Saved annotated video to segmented_output.mp4")

4. Model Family & Real-World Hardware Benchmarks

Meta released SAM 2 in four distinct model sizes to address diverse deployment constraints, from edge robotics to cloud VFX servers:

5. Bidirectional Correction Workflow

One of SAM 2's most powerful enterprise capabilities is interactive multi-frame refinement. If the model begins to drift on frame 120 due to extreme motion blur:

1. The user pauses on frame 120.
2. The user clicks a corrective negative point on the background.
3. SAM 2 immediately re-conditions its memory bank and propagates updates bidirectionally (both forward to frame 200 and backward to frame 80), repairing the entire timeline with zero re-computation of the base video embeddings.

6. Industrial Applications in Modern Engineering

SAM 2 expands computer vision possibilities across mission-critical industries:

• Medical Imaging & Surgical Video: Tracking surgical instruments and anatomical organs through dynamic endoscopic video feeds.
• Autonomous Drones & Robotics: Pixel-precise spatial tracking of dynamic obstacles without training bespoke per-object classifiers.
• Automated Video Post-Production: Zero-touch rotoscoping for background replacement, object removal, and visual effects.

Building Intelligent Vision Systems

Deploying real-time video perception requires balancing transformer backbone latency with memory retention efficiency. By combining Hiera's lightweight representations with bidirectional memory attention, SAM 2 sets an enduring benchmark for real-time video segmentation.

If you are developing enterprise computer vision pipelines, multi-camera tracking systems, or edge AI infrastructure, discover our custom Computer Vision & AI Engineering Services, browse our technical teardowns across the Machine Learning category, and explore our production case studies.

Official References & Further Reading

• Meta AI SAM 2 Research Paper
• SAM 2 Official Open-Source GitHub Repository
• SA-V Benchmark Dataset Overview