V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning by Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran, Koustuv Sinha, Mike Rabbat, Yann LeCun, Nicolas Ballas, Adrien Bardes (2026)

tags

Self-supervised learning, Vision transformer, Foundation models, Robotics

source

(Mur-Labadia et al. 2026)

Summary

V-JEPA 2.1 is a family of self-supervised video models (ViT-g/G, 1B/2B, plus distilled ViT-L/B variants) from FAIR at Meta that extends the Joint-Embedding Predictive Architecture (JEPA) line to produce representations that are simultaneously strong on dense spatio-temporal tasks (segmentation, tracking, depth, action anticipation) and global understanding tasks (action and image recognition). The paper’s central empirical observation is that when the masked-prediction loss is applied only to masked tokens (as in V-JEPA 2), the encoder has no incentive to encode fine-grained local structure in context tokens, so they collapse into register-like global aggregators and dense downstream performance suffers.

The fix is a dense predictive loss \(\mathcal{L}_\text{dense} = \mathcal{L}_\text{predict} + \mathcal{L}_\text{ctx}\) that adds a weighted L1 reconstruction term \(\mathcal{L}_\text{ctx}\) over the context (visible) tokens as well. The weighting \(\lambda_i = \lambda / \sqrt{d_\text{min}(i, M)}\) decays with distance from the nearest masked patch, which trades off local continuity against training collapse; a warm-up schedule prevents the context loss from degrading global action recognition early in training. Three further ingredients compound the gains: (i) deep self-supervision that applies prediction and context losses at multiple intermediate encoder layers via a multi-level predictor with MLP fusion; (ii) a multi-modal tokenizer using a 2D \(16\times16\) conv for images and a 3D \(16\times16\times2\) conv for videos with a learnable modality token, replacing V-JEPA 2’s 3D-only tokenizer; and (iii) VisionMix163M, a rebalanced pretraining mix that swaps ImageNet for LVD-142M and up-weights the motion-rich YT-1B source.

Empirically the approach sets or matches state-of-the-art with frozen backbones across a wide range of benchmarks: 7.71 mAP on Ego4D short-term object interaction anticipation, 40.8 Recall@5 on EPIC-KITCHENS-100, 0.307 RMSE on NYUv2 depth (linear probe), 72.7 J&F-Mean on DAVIS object tracking, 77.7 on Something-Something-V2 action recognition, 85.0 mIoU on Pascal VOC semantic segmentation, and a 20-point improvement in real-robot Franka grasping over V-JEPA-2 AC with a 10x speedup on TartanDrive navigation.

Key Ideas

• Context self-supervision: extending the masked-prediction loss to visible (context) tokens with a distance-weighted L1 penalty prevents context tokens from acting as global aggregators and yields spatially structured feature maps (PCA visualizations show coherent parts rather than noise).
• Warmup + weighted scheme for \(\mathcal{L}_\text{ctx}\): constant-weight context loss hurts action recognition; a warmup schedule and an inverse square-root distance weighting (emphasizing context patches near masked regions) restore global performance while keeping dense gains.
• Deep self-supervision: concatenating outputs of 3 intermediate encoder blocks with the final layer, fusing with an MLP, and applying both losses at each level gives consistent downstream improvements and removes the need for intermediate-layer probing at evaluation time.
• Multi-modal tokenizer: separate 2D/3D patch embedders with a learned modality token fix the representational bias of V-JEPA 2 which treated images as 16-frame static videos.
• VisionMix163M: a more diverse and motion-rich pretraining mix (LVD-142M curated images, up-weighted YT-1B, more SSv2) improves all downstream tasks jointly.
• Scaling: the recipe scales cleanly from ViT-L/80M to ViT-G/2B; a 512-px image / 384-px video cool-down phase further boosts dense performance.
• Robotics evidence: improved dense features translate to real-world benefits — +20% zero-shot Franka grasping and 10x faster planning on TartanDrive navigation — suggesting dense SSL features are a useful state-estimation substrate for world-model-driven control.

Comments

The paper is a careful empirical study that diagnoses a specific failure mode of Self-supervised learning masked-latent objectives — namely that “register-like” global aggregation in context tokens can dominate when the loss is applied only on masks — and fixes it with a minimal additive modification rather than a new architecture. This is reminiscent of the DINO/DINOv3 observation that dense features benefit from per-token supervision, but here achieved inside a predictive-latent (JEPA) framework rather than a contrastive one. The connection to Contrastive learning-style dense heads and to register tokens (Darcet et al. 2023) is explicit and worth tracking.

The frozen-backbone evaluation protocol is a notable strength: all downstream results use a single pretrained model with linear or attentive probes, which makes the ablations interpretable. The robotics results — in particular the real-robot grasping improvement — are the strongest evidence that dense SSL features matter for Robotics world models, and reinforce the broader JEPA/LeCun agenda of prediction-in-latent-space as a path toward embodied prediction and planning. Limitations: the paper does not compare against image-only DINOv3 at equal compute for global image tasks, and the context loss warmup adds two extra hyperparameters that practitioners will have to tune for new data regimes.

Connections

• Related to Self-supervised learning because V-JEPA 2.1 is a masked-latent SSL method and the paper’s central finding (dense predictive loss) is a general lesson about how masked-prediction objectives shape local feature structure.
• Related to Vision transformer because both encoder and predictor are ViTs and the multi-modal tokenizer is a ViT patch-embedding variant; the results add evidence that ViT-G (2B) scale is beneficial for video SSL.
• Related to Foundation models because V-JEPA 2.1 is positioned as a frozen visual foundation backbone evaluated across many downstream tasks with linear probing rather than fine-tuning.
• Related to Robotics because the paper reports real-world Franka grasping and TartanDrive navigation results, tying SSL feature quality to world modeling and embodied control.
• Related to Contrastive learning because the context-loss mechanism is a dense per-token signal in spirit similar to dense contrastive/self-distillation objectives (DINOv3), though instantiated in a predictive-latent framework.
• Related to Scaling laws because the paper shows systematic downstream gains from scaling model size (300M → 2B) and data (VisionMix163M).

Bibliography

1. Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran, Koustuv Sinha, Mike Rabbat, Yann LeCun, Nicolas Ballas, Adrien Bardes. March 15, 2026. "V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning". https://arxiv.org/abs/2603.14482.