ScanNet++ Novel View Synthesis and 3D Semantic Understanding Challenge

While radiance field methods have transformed novel view synthesis, many of their components still rely on implicit or neural ingredients that limit speed and expressivity. In this talk, I will present recent work revisiting explicit representations as a foundation for reconstruction and generation: Radiant Foam, a foam-based partition of space enabling real-time differentiable ray tracing; Power Foam, which unifies ray tracing and rasterization in a single framework; and Spherical Voronoi, a new explicit basis for directional appearance that surpasses strong neural baselines such as Zip-NeRF on standard benchmarks.

Physically-grounded world models — for digital twinning, robotics, and simulation — all rely on a fast, feed-forward geometry estimator, whether geometry sits explicitly inside the architecture or is learned implicitly from data.

The talk traces VGGT models (VGGT, VGGT-Ω) that established this foundation, then introduces Echo-2 — SpAItial's 3D Gaussian Splatting world generator, producing real-time, editable, simulatable scenes from a single prompt.

Generative models have transformed how we synthesize images, video, and 3D content, but their relationship to understanding scenes is less obvious. This talk follows a single question through a series of projects: can we get better 3D scene understanding by getting really good at generating views? I'll start with 3DiffTection, where adding view-synthesis control to a 2D diffusion model turns its features 3D-aware enough to drive detection — a first hint that generation and understanding are two sides of one coin. That motivates a search for excellent novel view synthesis. A recurring obstacle however, exists: pretrained generators often deviate from what we ask of them. I'll show how to fix this. First without retraining, by steering attention maps when models built for NVS won't follow their control signal (Zero-to-Hero), and repurposing video diffusion models that were never trained for NVS at all (Time-to-Move). When training-free tricks run out, the question becomes how to train well: FlowBender shows that closing the loop with feedback beats simply adding data, and Realiz3D tackles the synthetic-to-real gap by separating control from visual domain. I'll close with what I still don't know: whether all this progress on generation actually bought us better understanding?

Real-time world models now generate interactive environments at 720p on datacenter GPUs. However, AR/VR headsets demand stereo rendering at 72+ Hz on mobile chips drawing a few watts — a gap of roughly 100x in compute efficiency. You cannot wear a world model. This talk argues that explicit 3D representations, particularly 3D Gaussian Splatting, remain essential for wearable displays. I survey three frontiers: making 3DGS fast enough for mobile VR, generating Gaussians from sparse captures via camera-controlled diffusion models, and directly producing 3D Gaussians through flow matching — blurring the line between reconstruction and generation.

Three-dimensional understanding is a central challenge in computer vision, with direct applications to robotics and augmented reality. While classically framed as optimization-based SfM, recent methods have shown promising results via feedforward 3D prediction. In the first part of the talk, I will discuss our efforts in this space, and point out particular wins for the feedforward 3D approach, including metric understanding, time-varying dynamics, and hidden surface reconstruction. In the second part of the talk, I will discuss our efforts on implicit 3D understanding with pixel-level generation via diffusion/flow. Here, crucial issues include disentangling camera and scene motion. Finally, I'll conclude with some opinions on how these schools of thought might coexist. In particular, I will argue that explicit 3D reconstruction will remain useful for controllable and memory-efficient pixel generation.