CoDimRecon: Agentic Reconstruction of Sim-Ready 3D Scenes with Deformable Geometry

Original paper: ​CoDimRecon: Agentic Reconstruction of Sim-Ready 3D Scenes with Deformable Curves, Surfaces, and Volumes

Highlights and Results

CoDimRecon reconstructs an editable indoor environment from multi-view RGB images, including frames captured while moving a camera through a room. Its output contains separate rigid objects, articulated parts with joints, and deformable assets prepared for physical simulation. For readers building interactive 3D environments, the central contribution is connecting visual reconstruction to the geometric representations and behavioral checks that simulation actually requires.

The official demonstrations include a stretching telephone cord, a deforming plastic bag, a compressible chair cushion, and a sliding drawer. On three Replica and three ScanNet++ scenes, the authors report Chamfer distances of 6.63 cm and 21.66 cm, respectively, versus HoloScene’s 6.79 cm and 27.15 cm. However, HoloScene retains higher input-view PSNR and SSIM. These results show a tradeoff between editable geometry, physical usability, and photometric fidelity; they do not establish universal reconstruction superiority. Deformable interactions are capability demonstrations, not a cross-method benchmark. Experiments.

How It Works

The workflow has three stages: construct an editable scene using geometric and generated references; refine appearance, joints, and rigid-body placement; then reconstruct deformables in solver-compatible forms and test their behavior. It orchestrates pretrained perception, generation, an agent, Blender, and simulators without task-specific training of a new scene generator. Official method overview.

First, VGGT-Omega supplies camera parameters, depth, and point maps to anchor layout and dimensions. CropFormer extracts object masks, which are back-projected and clustered across views into 3D instance tracks. SAM3D generates a reference mesh for each track; REST3D helps recover small omitted objects. The agent uses these meshes as shape references, rebuilding furniture from Blender primitives and modifiers. This produces compact, editable geometry and exposes movable components, instead of simply accepting every generated mesh as the final asset. Reconstruction.

Next, a render–evaluate–refine loop compares the authored scene with its input views. Per-channel quadratic color alignment, c′ = ac² + bc + d, helps diagnose errors: substantial improvement suggests lighting or material mismatch; little improvement points toward geometry or pose. The agent adds explicit joints and settles objects under gravity in MuJoCo, writing their stabilized positions back into the scene. Scene refinement.

Deformables then enter separate agent sessions. A cable becomes a continuous centerline plus radius for rod simulation; cloth or paper becomes a manifold midsurface plus thickness for shell simulation; a cushion becomes a closed solid for tetrahedral meshing. High-resolution solids can bind to a simpler simulation proxy. Reusable simulator skills initialize constitutive models and contact settings on an IPC-family backend. Geometry determines discretization, while behavior determines whether the chosen material model is adequate. Deformable reconstruction.

Finally, the agent plans diagnostic robot interactions and checks both numerical validity and task behavior. Failed trials trigger revisions in a fixed order: motion, geometry, numerical settings, then material model. In the paper-folding test, elastic bending remains numerically valid but springs back. Under the same gripper trajectory, plastic hinges retain 92.5% of the peak fold angle one second after release, versus 3.7% with elastic bending. The test therefore checks more than whether a simulation runs without crashing. Controlled folding experiment.

Applications, Availability, and Future Directions

The demonstrated outputs are relevant to robot-interaction experiments and editable simulation environments; the paper also motivates gaming and immersive content creation. Their physical parameters are effective settings under assumed models, not measurements recovered from real dynamics. As checked on November 6, 2026, the official project page still says “Code Soon”; no implementation linked there was available for code-level verification. The paper provides no explicit future-work roadmap. The concrete announced next step is code release, with no stated timetable. This article is based on the paper and official materials, without rerunning the system. Project and demonstrations.