A benchmark for 3D coding agents
19 models · 25 reference photos · 475 runs
One photo. One prompt. In a browser three.js playground, agents code, render, and refine a 3D scene—testing spatial understanding and coding.
CompareAny two models, side by side →(1)Results(2)Models(3)Method
ResultsHow well did each agent model what it saw? A vision model compares two runs of the same photo, blind to names and costs; the ranking below is the Elo fit of those votes.
Across 25 photos · anchored at Claude Sonnet 5.5 = 1000 · 2237 pairs compared
1
Gemini 4 Carbon
$22.61 · 5h 10m total
Elo 1521±105
2
GPT-6 Astra
$12.44 · 29m 40s total
Elo 1496±86
3
Gemini 4 Argon
$11.12 · 2h 30m total
Elo 1462±105
4
GPT-6.1 Sol
$2.06 · 26m 31s total
Elo 1412±91
5
Gemini 4 Hydrogen
$15.27 · 1h 25m total
Elo 1407±99
6
Claude Fable 5.2
$32.20 · 1h 59m total
Elo 1387±100
7
Gemini 3.8 Flash
$15.53 · 2h 54m total
Elo 1276±78
8
GPT-6.1 Terra
$0.84 · 20m 42s total
Elo 1251±93
9
Kimi K3.1 Pro
$1.50 · 30m 05s total
Elo 1171±103
10
Qwen3.8 Max
$9.59 · 1h 35m total
Elo 1146±94
11
Claude Opus 5.5
$5.67 · 29m 34s total
Elo 1139±72
12
Inkling 2 Small
$6.06 · 1h 50m total
Elo 1085±90
13
Claude Sonnet 5.5
$2.51 · 22m 46s total
Elo 1000anchor
14
MiniMax M3.1 Flash
$1.40 · 1h 38m total
Elo 919±150
15
DeepSeek V4.1 Flash (1017)
$1.33 · 2h 47m total
Elo 888±77
16
Step 5 Preview
$7.49 · 2h 4m total
Elo 862±100
17
GLM 5.3 Flash
$1.13 · 1h 10m total
Elo 821±98
18
Qwen3.8 27B
$0.78 · 1h 42m total
Elo 800±167
19
Nemotron 3.5 Super VL
$0.84 · 19m 28s total
Elo 472±132
GPT-6 Astra has the highest floor: it never placed below #6 on any photo.
GPT-6 Astra is the steadiest of the top six: its rank moves by 5 places across the photos it was ranked on.
Anchored at Claude Sonnet 5.5 = 1000; only gaps between models matter (100 points ≈ 64% head-to-head). Equal Elo shares a rank; ties are ordered by worst photo, then recorded cost.
Elo vs. costBlue rings highlight the best Elo–cost trade-offs. Notes compare neighboring models on the frontier. Totals include only recorded data. Models with more than 3 failed or missing runs are left out.Frontier modelOther model: matched or beaten using no more moneyCurve and shading: visual guides, not predictions
40080012001600
Elo
$1.00$2.00$5.00
+1612.4× the price+505.4× the price
Gemini 4 ArgonGPT-6 AstraGPT-6.1 SolKimi K3.1 ProClaude Sonnet 5.5GLM 5.3 FlashClaude Opus 5.5DeepSeek V4.1 Flash (1017)Nemotron 3.5 Super VLClaude Fable 5.2Gemini 3.8 FlashGemini 4 CarbonGemini 4 HydrogenGPT-6.1 TerraInkling 2 SmallQwen3.8 Max
Elo, cost, time and tokens for each model, cheapest first
ModelEloCostTimeTokensElo per $
Nemotron 3.5 Super VL472$0.8419m 28s5.4M562GPT-6.1 Terra1251$0.8420m 42s1.7M1489GLM 5.3 Flash821$1.131h 10m16.5M728DeepSeek V4.1 Flash (1017)888$1.332h 47m51.1M670Kimi K3.1 Pro1171$1.5030m 05s1.7M778GPT-6.1 Sol1412$2.0626m 31s2.6M686Claude Sonnet 5.51000$2.5122m 46s1.9M398Claude Opus 5.51139$5.6729m 34s3.3M201Inkling 2 Small1085$6.061h 50m63.1M179Qwen3.8 Max1146$9.591h 35m9.3M119Gemini 4 Argon1462$11.122h 30m9.3M131GPT-6 Astra1496$12.4429m 40s2.9M120Gemini 4 Hydrogen1407$15.271h 25m16.1M92Gemini 3.8 Flash1276$15.532h 54m35.7M82Gemini 4 Carbon1521$22.615h 10m16.8M67Claude Fable 5.21387$32.201h 59m9.0M43
Gemini 4 Carbon leads at Elo 1521; GPT-6 Astra is 25 behind, inside the overlap of their 95% intervals.
Gemini 4 Carbon leads GPT-6.1 Sol by 109 Elo and costs 11.0× as much.
Best run for each photoWhat agents built from each of the 25 reference photos. These are the highest-scoring final scenes; ties favor the lower recorded cost.
Every model. Every photo.All 19 models tackling the same 25 spatial modeling tasks. Each cell is the model's rank on that photo, where 1 is best. ✕ marks a run that failed and produced no model to rank. Open a model to explore what it built.
Choose a reference photo
How the benchmark worksSame photos, same prompts, same step budget.
By the numbers$150.36recorded API cost29h 48mtotal recorded runtime317.1Mrecorded tokens7,684recorded model steps440/475successful runs
Run
Build, inspect, refine. Agents work in an in-browser three.js and manifold.js playground that emulates Blender's core modeling loop—writing geometry code, checking rendered views, and refining 3D structure—to test spatial understanding and coding. Every run can be replayed and edited live in the playground.
Vote
Two runs of the same photo are shown to a vision model together, with model names and costs hidden. It compares six rendered views of each scene and picks A, B, or tie—4474 votes over 2237 pairs.
Rank
Those votes become one Elo per model, anchored at Claude Sonnet 5.5 = 1000. Only the gap between two models matters; the sign just says who is ahead. A 100-point gap is a 64% win chance for the model ahead, 36% for the one behind; 200 points, 76% and 24%; 400 points, 91% and 9%.
One run per model–photo pair. Stalled models are retried with less or no reasoning (51 runs), keeping the photo, prompt, and step budget fixed. Reported costs and times exclude discarded attempts.
Questions & answersWhat the benchmark measures.
01What do agents do in this benchmark?Reconstruct a reference photo as a 3D scene using code. Agents write geometry, inspect rendered views, and revise the scene to better match the photo.02What capabilities does this reveal?It probes spatial reasoning: inferring shape, scale, depth, and object placement from an image. It also probes whether an agent understands how a 3D scene projects into a 2D view, including perspective and occlusion.03Why does this matter beyond reconstruction?These skills also matter when composing scenes from existing 3D assets: choosing their scale, orientation, and placement, and modeling simple missing objects directly. The benchmark probes foundations of that workflow, rather than asset-library use itself.04Why a browser three.js playground instead of Blender?Heavy 3D suites like Blender conflate spatial reasoning with software-specific workflows, and agents easily get sidetracked tweaking shader nodes and textures—work that diffusion models handle far better than code.
By running a minimal manifold.js geometry runtime and three.js renderer in the browser, the benchmark keeps materials simple and focuses strictly on spatial understanding, 3D structure, and coding. With zero local setup, you can replay any run or test your own agent right in the playground.05Is this image-to-3D generation?Not in the diffusion or neural-reconstruction sense. Rather than generating a textured mesh in one pass, the agent writes explicit, editable code to construct clean geometry and part hierarchy through visual feedback.
ResultsModelsMethodFAQ
Texto de sketchagent.dev tal como lo publicó, leído de su página y mostrado aquí con el formato de El Autómata. El original, en sketchagent.dev · Volver a la noticia de El Autómata