ParticleSplat: Self-supervised Object-centric Latent Particle Splatting

Abstract

We present ParticleSplat, a self-supervised object-centric representation learning method that decomposes scenes into latent particles through feedforward 3D Gaussian Splatting. Building on Deep Latent Particles (DLP), we extend the particle representation beyond 2D image space to capture the geometric structure needed for robotic manipulation. Our model jointly encodes posed observations into a 3D object-centric latent space, then transforms each particle into an aligned field of 3D Gaussians whose composition reconstructs the scene.

Training with a novel view synthesis objective yields object masks without supervision and supports controllable scene editing through particle attributes. Evaluations on simulated and real-world data demonstrate the representation’s reconstruction and decomposition capabilities. On downstream imitation learning, ParticleSplat improves policy success on both RLBench and MimicGen, showing the value of combining object-centric structure with explicit 3D information.

A scene you can edit

Pick a point. Make the scene your own.

RLBench Close JarInteractive 3D

Particle-aligned Gaussian splats

Interactive scene preview: two jars and a separate lid beneath a robot arm.

A little room to experiment.

Loads ~10 MB once. Everything runs in your browser.

Click a ball to edit · Drag it to move Drag background to orbit · Right-drag to pan · Scroll / pinch to zoom

Ready to explore.

Translucent markers select editable regions. Move and scale the reconstruction, then explore a camera range extending slightly beyond the two input views. Rotation and tilt are limited because unseen surfaces may be incomplete; edits do not simulate robot kinematics or physics.

An interactive illustration based on a held-out RLBench scene and a two-view, 50k-step checkpoint. The initial camera is an unseen intermediate view; native model resolution is 128 × 128. This prepared demo is separate from the benchmark results below.

Architecture

From posed observations to object-centric particles and compositional 3D Gaussians.

Scene decomposition

In the clips, Ctx denotes a context input and GT the reference target view. Use the play controls to explore each silent video.

RLBench · single-view

A front-camera RGB-D observation conditions a sweep of novel viewpoints. The clips show reconstruction and particle decomposition of the same scene.

Novel-view reconstruction

RGB reconstruction MP4

Depth reconstruction MP4

Object decomposition

Segmentation MP4

Foreground MP4

Background MP4

Particle geometry

3D keypoints MP4

2D boxes MP4

3D boxes MP4

15.5 s per clip · Same scene and camera sweep across all eight views.

Real-world · multi-view

Two RGB context views from a handheld camera, without depth input. These clips show scene reconstruction and decomposition, not real-world policy execution.

Novel-view reconstruction

RGB reconstruction MP4

Depth reconstruction MP4

Object decomposition

Segmentation MP4

Foreground MP4

Background MP4

Particle geometry

3D keypoints MP4

2D boxes MP4

3D boxes MP4

42.8 s per clip · Same scene and camera sweep across all eight views. Overlapping particles are filtered with non-maximum suppression for visual clarity, as described in the supplement.

Results

Evaluating the representation through 3D reconstruction and downstream policy learning.

3D reconstruction

Novel-view synthesis in simulation and real scenes.

Quantitative results

Novel-view synthesis at 128 × 128, averaged over evaluation views and scenes. Higher PSNR and SSIM are better; lower LPIPS is better.

Reported reconstruction metrics from the paper.
Method	PSNR (dB)	SSIM	LPIPS
RLBench · single-viewOne front RGB-D context view
ParticleSplat (ours)	25.35	0.7293	0.3563
VAE-GS (no object-centric structure)	21.85	0.6858	0.4838
GNFactor	16.20	0.5273	0.6819
ManiGaussian	15.87	0.5154	0.6793
RLBench · multi-viewTwo RGB-D context views
ParticleSplat (ours)	28.77	0.8144	0.2565
VAE-GS (no object-centric structure)	25.93	0.7326	0.3404
Real-world · multi-viewTwo RGB context views; no depth input
ParticleSplat (ours)	24.24	0.8455	0.1153
VAE-GS (no object-centric structure)	20.33	0.4659	0.6587

PSNR: peak signal-to-noise ratio; SSIM: structural similarity; LPIPS: learned perceptual image patch similarity. RLBench uses four held-out target views; real-world evaluation uses every fourth view of the capture trajectory.

Qualitative results

Drag the divider to compare ground truth with ParticleSplat at the same novel camera view. You can also focus a slider and use the arrow keys.

RLBench · single-view RGB-D

ParticleSplat RGB reconstruction of RLBench scene 1 at the target camera view. — Scene 1

Ground-truth RGB image of RLBench scene 1 at the same target camera view. — Scene 1

ParticleSplat RGB reconstruction of RLBench scene 2 at the target camera view. — Scene 2

Ground-truth RGB image of RLBench scene 2 at the same target camera view. — Scene 2

ParticleSplat RGB reconstruction of RLBench scene 3 at the target camera view. — Scene 3

Ground-truth RGB image of RLBench scene 3 at the same target camera view. — Scene 3

ParticleSplat RGB reconstruction of RLBench scene 4 at the target camera view. — Scene 4

Ground-truth RGB image of RLBench scene 4 at the same target camera view. — Scene 4

Real-world · multi-view RGB

ParticleSplat RGB reconstruction of Real-world scene 1 at the target camera view. — Scene 1

Ground-truth RGB image of Real-world scene 1 at the same target camera view. — Scene 1

ParticleSplat RGB reconstruction of Real-world scene 2 at the target camera view. — Scene 2

Ground-truth RGB image of Real-world scene 2 at the same target camera view. — Scene 2

ParticleSplat RGB reconstruction of Real-world scene 3 at the target camera view. — Scene 3

Ground-truth RGB image of Real-world scene 3 at the same target camera view. — Scene 3

ParticleSplat RGB reconstruction of Real-world scene 4 at the target camera view. — Scene 4

Ground-truth RGB image of Real-world scene 4 at the same target camera view. — Scene 4

RGB panels extracted from the paper’s extended qualitative figures, in the original row order. No retouching or new evaluation; the real-world examples use two RGB inputs without depth.

Policy learning

Manipulation success with a frozen ParticleSplat encoder and the EC-Diffuser policy.

Quantitative results

Overall and per-task success rates on RLBench and MimicGen, averaged over three policy seeds.

Comparing representations. ParticleSplat, DLP, and VAE-GS share the EC-Diffuser policy on RLBench. On MimicGen, ParticleSplat, DLP, and 3D-DLP share EC-Diffuser. GNFactor, ManiGaussian, and EquiDiff use different policy architectures and are included as external references.

ParticleSplat (ours) Ablations (same EC-Diffuser policy network) Baselines (published methods)

Representation ablations appear above baselines, separated by a horizontal line. DLP retains object-centric particles but has no explicit 3D geometry; VAE-GS retains 3D Gaussian splatting but has no object-centric structure. All bars show reported overall success on the same 0–100% scale.