SeMo3D: Unifying 3D Generation, Editing, and Completion via Selective Modeling

Highlights and Results: Change What Matters

Original paper: ​SeMo3D: Unifying 3D Generation, Editing, and Completion via Selective Modeling

SeMo3D, introduced by researchers at Nanjing University, brings ​3D generation, local editing, and completion into one framework​. For teams exploring AI asset creation, its central promise is continuity: create an object, revise a chosen region, or reconstruct a missing section while retaining the existing context. The name stands for ​Selective Modeling​, a strategy that makes the region to synthesize the common interface between these tasks. Official repository.

On Toys4K, the authors report geometric Chamfer Distance of 0.0118 versus TRELLIS.2’s 0.0330 (lower is better). Editing preservation reaches 37.39 dB ​PSNR​​​ versus VoxHammer’s 31.55 dB​. Completion F-score reaches 0.541 versus HY3D-Omni’s 0.412 with additional point guidance. Supported inputs differ across methods, and point hints come from ground truth; these comparisons do not isolate architecture gains. Paper, Tables 1–3.

How It Works: Select, Synthesize, Preserve

The key abstraction divides an asset into a selected region S and preserved context ​C​. Generation selects everything; editing selects the target; completion selects what is missing. For a chair, the same interface could create the whole chair, replace its backrest, or reconstruct an absent leg. These are illustrative uses of the task definition. The model learns a shared regional synthesis problem rather than requiring a separate task-specific training recipe. Repository overview.

SeMo3D fine-tunes TRELLIS.2’s three flow models on Trellis-500K and reuses its SC-VAE. Sparse structure predicts occupancy on a ​16³ latent ​grid​; shape and texture operate at occupied positions in a ​32³ grid​, producing geometry and PBR materials. The mask transfers across stages. Thus, task unification still uses three stage-specific models. Paper, method and training details.

Region-selective denoising creates training examples from ordinary assets. A random region receives noise while its surroundings stay clean; only predictions in that region contribute to the loss. Training spans local boxes through complete assets, avoiding paired before-and-after editing or completion datasets. At inference, selected tokens start from noise and evolve under guidance, while original context tokens are reused at every step. The model can therefore consult the whole object without resynthesizing all of it. Official method explanation.

Omni3DiT makes that mixed input understandable. Role embeddings identify selected and context tokens. Parallel cross-attention branches inject DINOv3 image features and CLIP text features; point tokens enter self-attention in the sparse-structure stage. Images guide appearance, text adds semantic intent, and points constrain spatial layout. Condition dropout varies the available guidance during training, encouraging the model to use the complementary signals. Official architecture diagram.

An image remains mandatory. Generation uses 12 Euler steps per stage; editing/completion use 50 for structure and 12 for shape and texture. Strict preservation applies to latent tokens, not necessarily every decoded vertex. Paper and inference appendix.

Applications and What Comes Next

The practical opportunity is iterative work: exploring prop variants, repairing incomplete references, and refining assets through successive instructions. The official project also demonstrates agent workflows that prepare reference images, identify edit regions, and invoke SeMo3D. This suggests a useful division of labor: an agent translates intent into a modeling request, while the 3D model handles regional synthesis. Interactive project demonstrations.

Readiness remains limited. As checked on November 6, 2026, the official ​repository​​ contains documentation and demonstration assets; weights and training, inference, and evaluation code remain unreleased. Its MIT license does not establish licensing terms for future weights. This article reviews the paper and available repository, without claiming an independent reproduction. Release checklist.

The authors identify two research directions. Higher-fidelity 3D representations could better capture thin structures such as palm fronds, which the current structured latents struggle to resolve. Stronger vision-language encoders could improve comprehension of complex compositional prompts; the paper shows orange fruits becoming orange texture rather than separate objects. Limitations and Future Work. The project's remaining release milestones are ​model weights on Hugging Face and complete inference, training, and evaluation code​. Official TODO list.