- tags
- Computer vision, Foundation models, Object recognition, Vision Language Models, Grounding, Synthetic training data
- source
- (Carion et al. 2025)
Summary
SAM 3 (Segment Anything Model 3) is Meta’s third installment of the SAM family of Foundation models for Computer vision. The headline contribution is a new task — Promptable Concept Segmentation (PCS) — which generalizes the SAM 1/2 Promptable Visual Segmentation (PVS) task from “segment one object given a click/box/mask” to “segment, identify, and track every instance of a visual concept given a short noun phrase, an image exemplar, or both, across an image or short video”. Concepts are restricted to atomic noun phrases (e.g. “striped cat”, “yellow school bus”); more compositional or referring queries are handled by combining SAM 3 with an external MLLM as a tool.
Architecturally, SAM 3 is a dual encoder–decoder transformer with a shared Perception Encoder backbone feeding (i) a Object recognition-style DETR detector conditioned on text and exemplar tokens, and (ii) a SAM 2-style memory-based tracker for video. The key model novelty is a learned global presence token that decouples recognition (“is the concept present anywhere in the image?”) from localization (“where exactly?”); each proposal query then only solves the conditional localization problem p(match | NP present), and the final score is a product of the global presence score and the per-query localization score. This decoupling particularly helps in the open-vocabulary setting with hard negatives, where forcing every query to do global recognition is counter-productive.
The other half of the contribution is a four-phase, human + AI in-the-loop data engine that produces SA-Co — a dataset of 4M unique concept labels, 52M masks across 5.2M images, plus 24.8M concept labels and 134K video–NP pairs in video. By fine-tuning Llama 3.2 as both a Mask Verifier and an Exhaustivity Verifier, the team roughly doubles annotation throughput versus a human-only pipeline; AI verifiers are shown to close ~half the gap between SAM 3 and human performance. SAM 3 sets new SOTA on PCS (≥2× the cgF1 of OWLv2 on SA-Co/Gold), surpasses prior detectors on LVIS/COCO zero-shot, and beats SAM 2 on VOS and interactive image segmentation while running at ~30 ms/image on an H200.
Key Ideas
-
Promptable Concept Segmentation (PCS) as a new task: detect + segment
- persistently identify all instances of an atomic noun-phrase concept
in an image or short video, with optional positive/negative image exemplars for refinement. Generalizes PVS (one object per prompt) of SAM 1/2.
-
Presence head / presence token: a single learned global token whose job is to predict
p(NP present in the image). Each per-query proposal only solves the conditional localization sub-problem, and the per-query score is the product. This +1.5 cgF1 ablation is the most important model change relative to standard DETR-style decoders for open-vocabulary detection. -
Shared backbone, decoupled detector + tracker: detector is identity- agnostic, tracker preserves identity across frames. Dual-supervised with DAC-DETR, Align loss, MaskFormer-style mask head, plus a semantic segmentation head.
-
Image exemplars as prompts: a (bbox, ±label) pair encoded by an exemplar encoder (position embedding + label embedding + ROI-pooled PE features), concatenated with text tokens to form a unified prompt representation. Works in isolation, as a refinement on top of text, or iteratively across frames.
-
Four-phase data engine: (1) human verification, (2) Llama-3.2-tuned AI verifiers for mask/exhaustivity checks, (3) ontology-driven domain expansion to 15 datasets via a 22.4M-node Wikidata-derived ontology, (4) video annotation. AI verifiers double pipeline throughput while preserving quality.
-
SA-Co benchmark: 207K unique phrases, 121K images/videos, 3M media-phrase pairs with hard negatives across 4 image splits (Gold/Silver/Bronze/Bio) and a video split (VEval). >50× more concepts than prior open-vocab segmentation benchmarks.
-
Ambiguity handling: 3 annotators per NP on SA-Co/Gold, oracle metric, and a small ambiguity module in the model.
-
Classification-gated F1 (cgF1): combines positive micro-F1 (pmF1, localization quality) with image-level Matthews Correlation Coefficient (IL_MCC, calibrated presence prediction at threshold 0.5), penalizing models that are accurate but uncalibrated.
-
SAM 3 Agent: combine SAM 3 with an arbitrary MLLM (Qwen2.5-VL, Llama 4 Maverick, Gemini 2.5 Pro) that proposes NP queries, calls SAM 3 as a tool, and iterates until the masks satisfy the user. Zero-shot SOTA on ReasonSeg and OmniLabel; outperforms specialist fine-tuned models like LISA-13B-LLaVA1.5.
-
Synthetic data scaling: SA-Co/SYN (synthetic, no humans) shows comparable scaling behavior to human-annotated SA-Co/HQ on a held-out domain, suggesting domain adaptation without any human annotation is feasible at scale (cf. Synthetic training data).
Comments
This is a textbook example of how task definition can do as much heavy lifting as architecture in a foundation model paper. The PCS task itself — “segment all instances of an atomic concept” — is the fundamental change from SAM 2; everything else (presence head, exemplar prompting, data engine, benchmark) is downstream of that choice. A useful comparison is OWLv2 / GroundingDINO, which already do open-vocabulary detection with text prompts; SAM 3 reframes the same underlying capability as segmentation with persistent identity in video and pairs it with a 50×-larger benchmark, which is a much stronger commercial moat than another +2 mAP on COCO.
The presence head is conceptually clean. DETR-style decoders force every object query to simultaneously decide what and where, which is a poor fit for open-vocabulary settings dominated by hard negatives: in such regimes most images contain no instance of a given query, so asking 1000 queries to each independently re-derive global absence is both wasteful and noisy. The presence head amortizes that decision and recovers calibration (IL_MCC +0.05 in ablation), which is disproportionately valuable when downstream metrics threshold at 0.5 confidence. Whether this idea generalizes beyond PCS — e.g. for generic open-vocabulary detection tasks — is worth watching.
The data engine deserves special attention. The cycle of SAM 3 proposes masks → AI verifier filters → humans correct only the rejects → retrain SAM 3 closely resembles the Foundation models data flywheel pattern that has been driving progress in LLM post-training (e.g. RLHF + reward-model-as-filter pipelines). Replacing humans with fine-tuned Llama 3.2 verifiers for the most repetitive verification steps is a concrete instance of the broader pattern of AI-assisted labeling. The Phase 4 video extension is conservative — they deliberately bias the sampler toward failure-prone clips so that human effort concentrates where SAM 3 is weakest, which is a sensible information-theoretic data acquisition strategy.
A few open questions and limitations:
- Concepts are restricted to atomic noun phrases. Compositional queries (“the cat on the left, but not the small one”) are punted to the SAM 3 Agent layer. This is a real limitation — many practical applications need referring-style segmentation — and it is half-addressed by the agent setup, but the agent does not benefit from end-to-end training over the compositional structure.
- Ambiguity is handled at evaluation time but only modestly at training time. The 3-annotator oracle metric inflates absolute numbers but does not directly train the model to represent uncertainty over interpretations.
- Compute cost is largely unreported. The four-stage training pipeline (PE pre-training, detector pre-training, detector fine- tuning, tracker training with frozen backbone) and the seven SAM 3 retraining cycles in the data engine are very expensive — the paper is not a recipe a small lab can replicate, even with the released checkpoints and benchmark.
- The AI verifier claim — that fine-tuned Llama 3.2 closes half of the gap between SAM 3 and human performance — is appealing but also self-referential: the AI verifier was distilled from human labels collected with the same conventions, so it inherits the annotation guidelines’ biases. The “AI verifier near human accuracy” framing should be read as “human-consistent at this task definition”, not as “objectively correct”.
The combination of (i) a new and clearly more useful task, (ii) a clean architectural decoupling that solves a real open-vocabulary recognition pathology, and (iii) an open-source benchmark + checkpoint release is exactly the playbook that made SAM 1 a default building block. SAM 3 is likely to become the default text-prompted segmentation backbone for downstream MLLM agents and annotation pipelines.
Connections
- Related to Computer vision because SAM 3 is an end-to-end vision system for detection, segmentation, and tracking.
- Related to Foundation models because SAM 3 is the third generation of a vision foundation model designed to be a reusable building block for downstream segmentation tasks.
- Related to Object recognition because the detector head is a DETR-derived open-vocabulary instance detector with a novel presence token that decouples recognition from localization.
- Related to Vision Language Models because (a) the model is conditioned on noun-phrase text via an aligned Perception Encoder backbone, and (b) the SAM 3 Agent setup uses a generic MLLM (Gemini, Qwen, Llama-4) to translate complex referring queries into atomic NP prompts.
- Related to Grounding because PCS is fundamentally a grounding task: binding a textual concept to all of its visual instances in a scene, with persistent identity in video.
- Related to Synthetic training data because the SA-Co/SYN study demonstrates that AI-generated annotations exhibit similar scaling behavior to human-annotated data on held-out domains, supporting the use of synthetic data for domain adaptation.
Bibliography
- Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, et al.. . "SAM 3: Segment Anything with Concepts". https://arxiv.org/abs/2511.16719.
Loading comments...