Frontier3DBench: Benchmarking Coding Agents for 3D Creation Across Objects, Scenes & Open Worlds

Original paper: Frontier3DBench: Benchmarking Coding Agents for 3D Creation across Objects, Scenes, and Interactive Open Worlds (November 6, 2026; v1).

Highlights and Results

Frontier3DBench asks whether coding agents can deliver editable, executable 3D projects across objects, scenes, and interactive worlds. For teams choosing an AI authoring workflow, its contribution is a common evaluation framework: 145 tasks across nine families, including 65 Blender object tasks, 65 Blender scene tasks, and 15 Unreal Engine world tasks. Inputs include instructions, reference images, photographs, and existing scenes.

The authors evaluate 11 model–harness configurations. GPT-6.1-sol with Codex leads overall at 66.02/100; GPT-6-astra with Codex leads interactive worlds at 61.28. Every configuration performs worse on reference reconstruction than on text-driven object and scene generation. Across 165 world submissions, local mechanisms average 83.08, moving-part clearance 20.82, and stateful workflows 30.26. These are component scores, not percentages of successfully completed worlds.

A human study provides limited calibration: eight raters rank four candidates on each of 45 reference-conditioned object/scene items. Automatic scores achieve mean Kendall tau-b of 0.727 and 79.2% agreement on the winner, but only 44.7% agreement on the complete ranking. This does not validate every task family or world behavior. See the experiments.

How the Evaluation Works

The agent receives a task contract: required geometry and appearance, reference images when applicable, and initial assets for editing. World tasks additionally define entities, starting states, allowed operations, and prerequisites. The agent writes and executes construction code, then submits a native Blender or Unreal project plus required exports. Evaluation files remain outside its workspace.

In the released orchestrator, public execution checks can trigger repairs, but the agent receives errors rather than private scores. Accepted source is frozen and hashed before evaluation. Separate evaluators rerun the program, inspect native geometry, render views, and judge task fulfillment. The paper uses Blender 5.2.1 and Unreal Engine 5.8.3; L1/L2 generation allows up to six attempts within 32 hours. These are budget ceilings, not measured completion times.

Scoring changes with the task. Reference reconstruction combines 50% image similarity, 20% projected object localization, 20% visual judgment, and 10% detail. Image similarity measures structure, edges, color, and tone; localization matches object boxes and penalizes missing or extra instances. GPT-6-astra at medium effort judges fixed visual criteria with generator identity hidden. Detail checks use coverage multiplied by one minus deviation, then take the weakest required execution/export variant. Ordinary L1 tasks inspect four executions in native and GLB form; L2 uses four native variants.

Scene editing separately measures the requested modification and preservation of protected surroundings. World evaluation rebuilds the submitted project with an independent observer and tests visible action effects, intersections of moving parts, actual character movement, and persistent state. Continuous workflows prohibit teleportation and resets. A door opening in isolation therefore cannot substitute for completing a route that requires reaching, unlocking, and passing through it.

World scores weight visual quality, task fidelity, layout, and interaction at 60/20/5/15. The final benchmark total weights L1/L2/L3 at 30/30/40. The scoring contract distinguishes confirmed candidate defects from missing technical evidence: an API or measurement failure must not silently become a zero.

Applications and Open Challenges

The benchmark supports selecting coding-agent workflows and diagnosing whether failures concern appearance, geometry, editing, or behavior. Its open repository includes task inputs, generation/evaluation code, and scoring contracts; benchmark software uses Apache-2.0, while task assets retain their own licenses. Blender, Unreal Engine, and model access require separate setup.

The paper identifies two central open challenges: matching specified geometry and layouts, and coordinating moving parts, actions, and persistent state. Those are the authors’ research directions, not announced product features. Comparisons describe complete model–harness setups, and visual judging retains calibration limits; photographic tasks also lack universal native 3D ground truth. This review reads the paper and implementation but does not rerun the benchmark.