Meta SAM 3: Text Prompt Segmentation for Images and Video, Plus SAM 3.1

Short answer: Meta SAM 3 is an open-weight vision model that segments every instance of a concept you describe with a short text phrase, an exemplar box, or classic click prompts, in still images and video. Weights sit on Hugging Face facebook/sam3 with official code in facebookresearch/sam3. A March 27, 2026 update, SAM 3.1, adds object multiplexing for faster multi-object video on one GPU. If you already use SAM 2 for masks, SAM 3 is the upgrade path when you need open-vocabulary "find all yellow school buses" style queries, not just point-and-track.

SAM 3 sits beside Meta's 3D line, which we cover in GenIA on SAM 3D Objects. SAM 3 is the 2D and video masking foundation that powers consumer edits and downstream research stacks.

TL;DR

What problem does SAM 3 solve?

Segmentation models used to split into two camps. Interactive models like SAM 1 and SAM 2 excel when a human clicks a point or box and wants a tight mask, including through time in video. Detector-style models take a fixed label set and return boxes or masks per class.

Meta's SAM 3 paper frames a middle task: Promptable Concept Segmentation. You give a short noun phrase ("yellow school bus"), an exemplar crop, or SAM-style clicks, and the model should return segmentation masks and persistent IDs for every object instance that matches the concept, including in cluttered scenes.

That matters for builders because many pipelines still chain a detector, a tracker, and a hand-tuned class list. SAM 3 tries to collapse detection plus instance masks plus tracking behind one prompt interface. Meta also ties it to creative tools — pixelate every face, highlight every player, trail every ball — which is why Edits and Vibes show up in the launch story on Meta's SAM 3 page.

For comparison with how other labs use segmentation inside agents, see OpenVLHarness for multimodal agents and CodeCapBench, which used SAM 3.1 for spatio-temporal IoU.

Vision object detection lens over a pear, illustrating Meta SAM 3 open-vocabulary segmentation targets

How SAM 3 is built (without the math wall)

Meta's publication page describes SAM 3 as one backbone shared by an image-level detector and a memory-based video tracker. A presence head decouples recognition from localization, which helps when many similar objects appear.

Training scale is part of the story. Meta says a data engine produced on the order of four million unique concept labels, including hard negatives, across images and video. They also shipped the SA-Co (Segment Anything with Concepts) benchmark so outsiders can test open-vocabulary prompts instead of only COCO-style classes.

Meta's announcement blog (November 19, 2025) claimed roughly a 2× gain over prior systems on PCS in both images and video, while keeping SAM 2-style click refinement. Treat that multiplier as vendor-reported, not independent.

SAM 3.1: why video builders should care

Video is where naive multi-object tracking burns GPU hours. Meta's March 27, 2026 SAM 3.1 note says SAM 3 previously ran one forward pass per tracked object. SAM 3.1 introduces object multiplexing: up to 16 objects processed together with shared memory, which Meta positions as a drop-in replacement for SAM 3 video checkpoints on Hugging Face facebook/sam3.1 (RELEASE_SAM3p1.md in the GitHub repo).

Meta's headline number: on a single H100, throughput for medium object counts moves from about 16 FPS to about 32 FPS. That is still not phone-real-time for every scene, but it is the difference between batch post-production and near-real-time preview for creator tools.

If you evaluate video agents, pair this with our Viggle Remake open video-to-video agent coverage — different task, same question of whether masks are cheap enough to run inside an agent loop.

How to run SAM 3 locally

Meta documents two paths: the sam3 Python package in GitHub and transformers on Hugging Face.

Transformers image example (from the model card on Hugging Face):

from transformers import Sam3Processor, Sam3Model
import torch
from PIL import Image
import requests

device = "cuda" if torch.cuda.is_available() else "cpu"
model = Sam3Model.from_pretrained("facebook/sam3").to(device)
processor = Sam3Processor.from_pretrained("facebook/sam3")

image_url = "http://images.cocodataset.org/val2017/000000077595.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")

inputs = processor(images=image, text="ear", return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model(**inputs)

results = processor.post_process_instance_segmentation(
    outputs,
    threshold=0.5,
    mask_threshold=0.5,
    target_sizes=inputs.get("original_sizes").tolist()
)[0]

GitHub image path uses build_sam3_image_model() and Sam3Processor from the sam3 package; video uses build_sam3_video_predictor() with session APIs for start_session and add_prompt. Read INSTALL.md in the repo before you pin versions — SAM 3.1 expects the latest facebookresearch/sam3 commit.

Hardware reality: Meta's speed claims assume datacenter GPUs. On a laptop CPU you can prototype, but PCS on long video is not pleasant without CUDA. For local vision stacks that avoid cloud APIs, compare llama.cpp-style local inference guides — different modality, same budgeting question.

License and gating

The Hugging Face model card requires you to agree to share information with Meta and accept their license before weights download. That is stricter than a plain Apache repo. If you need fully permissive weights for a commercial robot stack, read the text before you bake SAM 3 into a shipping pipeline. Research and internal demos are the easy cases.

A ball tracked across empty video frames, illustrating Meta SAM 3 multi-object video segmentation

Where SAM 3 fits in Meta's product map

Meta is explicit that SAM 3 is not only a paper. The SAM 3 research landing page lists:

• Segment Anything Playground for no-code trials on images and video templates.
• Instagram Edits for creator effects on selected people or objects.
• Vibes on the Meta AI app for similar generative edits.
• Partnerships such as Conservation X Labs for wildlife monitoring datasets.

For feed-ranking context on how Meta thinks about user-controlled AI, see Facebook Your Algorithm. SAM 3 is the computer-vision half; ranking is the distribution half.

Downstream researchers already borrow SAM masks. Our CodeCapBench post notes object-level spatio-temporal IoU with SAM 3.1 tracking against reference annotations — a pattern you can copy when you build video eval harnesses.

Limitations and honest bench skepticism

Open vocabulary is not magic. Meta's 75–80% of human figure on SA-Co is useful for comparing releases, but your warehouse, medical, or factory footage can sit far from Meta's training mixture. Plan a small gold set of phrases your product actually uses.

Concept collisions. Short phrases like "truck" or "person in red" remain ambiguous. SAM 3 supports exemplar boxes and follow-up clicks precisely because first-pass PCS can miss rare instances or bleed into look-alikes.

Video ID switches. Faster multiplexing reduces cost; it does not delete classic tracker failure modes when objects occlude each other. For robotics stacks that need metric depth, pair masking with projects like NVIDIA MetricTrail rather than masks alone.

3D is a sibling, not a substitute. Need a mesh from one photo? That is SAM 3D and GenIA, not SAM 3. Many pipelines will use SAM 3 masks to crop before a 3D model runs.

What people ask after reading the launch

Does SAM 3 replace YOLO or Grounding DINO? Not automatically. Detectors still win when you need fixed COCO classes at the lowest latency on edge hardware. SAM 3 wins when the class list is open-ended and you want masks, not boxes, for every instance of a phrase. Many teams will keep a detector for proposals and use SAM 3 only on hard language-conditioned edits.

Do I need the Playground? No for engineering, yes for product discovery. Meta's Segment Anything Playground is the fastest way to see whether your phrase triggers false positives before you wire APIs. Templates include privacy-style edits such as pixelating faces or license plates, which mirror how Meta describes Instagram Edits.

What about fine-tuning? Meta announced a Roboflow partnership so teams can annotate and fine-tune SAM 3 on domain data, per the November 2025 launch post. Fine-tuning is the realistic path if SA-Co averages hide failure on your SKUs.

Will SAM 3 run on Apple Silicon? The Hugging Face card targets PyTorch with CUDA for serious video. MPS may run short image demos, but Meta's FPS claims assume H100-class hardware. For on-device segmentation trade-offs, watch Apple's separate research line in tri-modal masked diffusion, which is generative, not PCS.

How does this relate to agent tooling? Agents that call vision tools still need guardrails on what they blur, crop, or upload. If your agent pipeline auto-masks people in user uploads, log the prompt, mask area, and frame hash alongside model version so compliance can reproduce decisions. That pattern mirrors how we discuss audit trails in agent sandbox comparisons.

SAM 1, SAM 2, and SAM 3 at a glance

The row that matters for builders is the last: SAM 3 is the first in the family where language is a first-class prompt, not an afterthought wrapped around a detector.

What to try this week

1. Accept the license on Hugging Face and run the "ear" COCO val image example to verify your GPU stack.
2. Pick ten phrases your product cares about and log false merges (two objects, one mask) and false splits.
3. If you track video, pull SAM 3.1 checkpoints and compare FPS and ID stability against SAM 3 on the same clip.
4. Wire masks into an agent only with human-readable overlays stored alongside prompts — useful for debugging PCS in production.

Related reading

• GenIA and SAM 3D Objects
• CodeCapBench and SAM 3.1 tracking metrics
• OpenVLHarness multimodal agent memory
• Facebook Your Algorithm feed controls
• NVIDIA MetricTrail depth from video
• Scenario 3D asset collection on Hugging Face

Specifications, checkpoint names, and FPS claims are accurate as of November 9, 2026. Check facebookresearch/sam3 and the Hugging Face model card before you pin versions in production.