-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
469 lines (437 loc) · 22.7 KB
/
Copy pathindex.html
File metadata and controls
469 lines (437 loc) · 22.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>PaintBench: Deterministic Evaluation of Precise Visual Editing</title>
<meta name="description" content="PaintBench: a pixel-level benchmark for evaluating precise, deterministic visual editing across 20 tasks and 1,920 problems. The best model today scores 17.1%." />
<link rel="icon" type="image/svg+xml" href="assets/img/favicon.svg" />
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Playfair+Display:wght@700&family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
<link rel="stylesheet" href="assets/css/style.css" />
</head>
<body>
<!-- ── Nav ────────────────────────────────────────────────────── -->
<nav>
<canvas id="nav-canvas"></canvas>
<div class="nav-inner">
<img src="assets/img/favicon.svg" class="nav-logo" alt="PaintBench" />
<div class="nav-links">
<a href="#example">How it works</a>
<a href="#benchmark">Tasks</a>
<a href="#results">Results</a>
<a href="#tinygrafixbench">TinyGrafixBench</a>
<a href="#citation">Cite</a>
</div>
</div>
</nav>
<!-- ── Hero ────────────────────────────────────────────────────── -->
<section id="hero">
<div class="wrap">
<h1 class="hero-title">PaintBench</h1>
<p class="hero-tagline">Can multimodal AI models make <strong><em style="color:#22c55e">exact</em></strong> visual edits?</p>
<p class="hero-sub">
A foundational benchmark for precise single-answer visual editing.<br>
Deterministically evaluated without bias-prone judge models.<br>
Infinite problem variety via procedural generation.<br>
The highest-performing model today reaches only <em>17.1%</em>.
</p>
<div class="hero-authors">
<a href="https://kaixubuilds.github.io/" target="_blank" rel="noopener" class="author-name">Kai Xu</a><sup>*</sup>
<span class="author-sep"> · </span>
<a href="https://ellisbrown.github.io/" target="_blank" rel="noopener" class="author-name">Ellis Brown</a><sup>*</sup>
<span class="author-sep"> · </span>
<a href="https://shrikar17.github.io/" target="_blank" rel="noopener" class="author-name">Shrikar Madhu</a>
<span class="author-sep"> · </span>
<a href="https://cs.nyu.edu/~fergus/pmwiki/pmwiki.php" target="_blank" rel="noopener" class="author-name">Rob Fergus</a>
<span class="author-sep"> · </span>
<a href="https://hhexiy.ai/" target="_blank" rel="noopener" class="author-name">He He</a>
<span class="author-sep"> · </span>
<a href="https://www.sainingxie.com/" target="_blank" rel="noopener" class="author-name">Saining Xie</a>
</div>
<div class="hero-affil">
<span class="affil-eq">* Equal contribution</span>
<span class="affil-sep"> · </span>
<span>New York University</span>
</div>
<div class="btn-row">
<a href="https://arxiv.org/abs/2606.00188" target="_blank" rel="noopener" class="btn cta-arxiv">
<img src="assets/img/arxiv_logo.png" alt="arXiv" class="btn-logo btn-logo-arxiv" /> Paper
</a>
<a href="https://huggingface.co/datasets/PaintBench/PaintBench" target="_blank" rel="noopener" class="btn cta-hf">
<img src="assets/img/hf_logo.png" alt="HuggingFace" class="btn-logo btn-logo-hf" /> Dataset
</a>
<a href="https://github.com/PaintBench/PaintBench" target="_blank" rel="noopener" class="btn cta-secondary">
<svg width="15" height="15" viewBox="0 0 24 24" fill="currentColor"><path d="M12 0C5.37 0 0 5.37 0 12c0 5.31 3.435 9.795 8.205 11.385.6.105.825-.255.825-.57 0-.285-.015-1.23-.015-2.235-3.015.555-3.795-.735-4.035-1.41-.135-.345-.72-1.41-1.23-1.695-.42-.225-1.02-.78-.015-.795.945-.015 1.62.87 1.845 1.23 1.08 1.815 2.805 1.305 3.495.99.105-.78.42-1.305.765-1.605-2.67-.3-5.46-1.335-5.46-5.925 0-1.305.465-2.385 1.23-3.225-.12-.3-.54-1.53.12-3.18 0 0 1.005-.315 3.3 1.23.96-.27 1.98-.405 3-.405s2.04.135 3 .405c2.295-1.56 3.3-1.23 3.3-1.23.66 1.65.24 2.88.12 3.18.765.84 1.23 1.905 1.23 3.225 0 4.605-2.805 5.625-5.475 5.925.435.375.81 1.095.81 2.22 0 1.605-.015 2.895-.015 3.3 0 .315.225.69.825.57A12.02 12.02 0 0 0 24 12c0-6.63-5.37-12-12-12z"/></svg> Code
</a>
</div>
</div>
<!-- Hero image — swap src to assets/img/hero.png once saved -->
<div class="wrap-wide">
<div class="hero-image-wrap">
<img id="hero-img" src="assets/img/hero.png"
onerror="this.parentElement.innerHTML='<div class=hero-image-placeholder>hero.png — drop the generated image here</div>'"
alt="PaintBench benchmark illustration: geometric shapes with measurement annotations and pixel-level evaluation grid" />
</div>
<div class="stats-bar">
<div class="stat-cell">
<div class="stat-val" style="color:var(--geo)">20</div>
<div class="stat-label">Foundational editing operations</div>
</div>
<div class="stat-cell">
<div class="stat-val" style="color:var(--str)">∞</div>
<div class="stat-label">Seed-generated problem variety</div>
</div>
<div class="stat-cell">
<div class="stat-val" style="color:var(--sym)">17.1%</div>
<div class="stat-label">Highest-performing model score (mIoU)</div>
</div>
</div>
</div>
</section>
<!-- ── §01 How a problem works ────────────────────────────────── -->
<section id="example">
<div class="wrap">
<div class="sec-num">01 / How a Problem Works</div>
<h2 class="sec-title">One correct answer per problem.<br>Evaluated at the pixel.</h2>
<p class="sec-body">
Each PaintBench problem gives the model an input image and precise instruction.
There is exactly one ground-truth answer.
Evaluation compares each pixel of the model output to the ground-truth answer and input image using
CIEΔE<sub>76</sub> perceptual color distance—no human rating, no judge model.
</p>
<div class="instruction-block">
<div class="label">Instruction</div>
Draw a filled olive-colored (#717A1E) circle centered at (66.1%, 56.7%) with a radius of 21.5% image width. Place it underneath any existing shapes. Clip any parts that may extend beyond the image boundary.
</div>
<div class="img-strip">
<div class="img-cell">
<img src="assets/img/eval_input.png" alt="Input image" />
<div class="cap">Input</div>
</div>
<div class="img-cell">
<img src="assets/img/eval_answer.png" alt="Ground truth answer" />
<div class="cap">Ground truth</div>
</div>
<div class="img-cell">
<img src="assets/img/eval_normalized_output.png" alt="Model output (Nano Banana 2)" />
<div class="cap">Model output <span style="color:var(--muted)">(Nano Banana 2)</span></div>
</div>
<div class="img-cell img-cell--error">
<img id="diff-img" src="assets/img/eval_diff_cie76_0.png" alt="Pixel error map" />
<div class="cap">Error map <span style="color:var(--muted)">(drag slider)</span></div>
</div>
</div>
<div class="slider-wrap">
<span class="slider-label">Tolerance ΔE ≤</span>
<input type="range" id="de-slider" min="0" max="10" value="0" step="1" />
<span class="slider-label" style="text-align:right">strict → lenient</span>
</div>
<div class="slider-stats">
<div class="stat-item">
<span class="skey">Tolerance </span>
<span class="sval" id="stat-t">ΔE ≤ 0</span>
</div>
<div class="stat-item">
<span class="skey">Edit accuracy </span>
<span class="sval" id="stat-edit">—</span>
</div>
<div class="stat-item">
<span class="skey">Preservation accuracy </span>
<span class="sval" id="stat-pres">—</span>
</div>
<div class="stat-item">
<span class="skey">IoU@tolerance </span>
<span class="sval" id="stat-iou">—</span>
</div>
</div>
<p style="margin-top:20px; font-size:.82rem; color:var(--muted); max-width:none;">
<strong style="color:var(--ink-2)">Green/blue pixels</strong> are correctly edited/preserved within the tolerance.
<strong style="color:var(--ink-2)">Red/orange pixels</strong> are incorrectly edited/preserved.
</p>
</div>
</section>
<!-- ── §02 Benchmark taxonomy ─────────────────────────────────── -->
<section id="benchmark">
<div class="wrap">
<div class="sec-num">02 / The Benchmark</div>
<h2 class="sec-title">20 tasks across 4 categories.<br>Infinitely scalable.</h2>
<div class="design-pillars">
<div class="pillar">
<div class="pillar-icon">⟳</div>
<div class="pillar-name">Procedurally Generated</div>
<div class="pillar-body">Each problem is generated from a unique seed and configurable scene/difficulty parameters.</div>
</div>
<div class="pillar">
<div class="pillar-icon">✓</div>
<div class="pillar-name">One Correct Answer</div>
<div class="pillar-body">Every output is evaluated pixel-by-pixel via CIE ΔE₇₆ color distance. No human ratings, no bias-prone judge models, no perceptual proxies.</div>
</div>
<div class="pillar">
<div class="pillar-icon">◈</div>
<div class="pillar-name">Atomic Operations</div>
<div class="pillar-body">Tasks target fundamental building-block operations found in many precise real-world visual editing tasks.</div>
</div>
</div>
<!-- Task switcher: all 20 tasks -->
<div class="taxonomy-layout">
<div class="cat-list">
<div class="cat-group">
<button class="cat-btn geo active" data-cat="geo">
<div class="cat-btn-name">Geometric Transformation</div>
</button>
<div class="task-list" id="tasks-geo">
<button class="task-btn active" data-task="translation" data-cat="geo">Translation</button>
<button class="task-btn" data-task="rotation" data-cat="geo">Rotation</button>
<button class="task-btn" data-task="reflection" data-cat="geo">Reflection</button>
<button class="task-btn" data-task="scaling" data-cat="geo">Scaling</button>
<button class="task-btn" data-task="shearing" data-cat="geo">Shearing</button>
</div>
</div>
<div class="cat-group">
<button class="cat-btn str" data-cat="str">
<div class="cat-btn-name">Structural Manipulation</div>
</button>
<div class="task-list" id="tasks-str" style="display:none">
<button class="task-btn" data-task="construction" data-cat="str">Construction</button>
<button class="task-btn" data-task="removal" data-cat="str">Removal</button>
<button class="task-btn" data-task="copying" data-cat="str">Copying</button>
<button class="task-btn" data-task="border" data-cat="str">Border</button>
<button class="task-btn" data-task="cropping" data-cat="str">Cropping</button>
</div>
</div>
<div class="cat-group">
<button class="cat-btn col" data-cat="col">
<div class="cat-btn-name">Color Change</div>
</button>
<div class="task-list" id="tasks-col" style="display:none">
<button class="task-btn" data-task="recolor" data-cat="col">Recolor</button>
<button class="task-btn" data-task="flood_fill" data-cat="col">Flood Fill</button>
<button class="task-btn" data-task="blending" data-cat="col">Blending</button>
<button class="task-btn" data-task="gradient" data-cat="col">Gradient</button>
<button class="task-btn" data-task="point_operations" data-cat="col">Point Operations</button>
</div>
</div>
<div class="cat-group">
<button class="cat-btn sym" data-cat="sym">
<div class="cat-btn-name">Symbolic Reasoning</div>
</button>
<div class="task-list" id="tasks-sym" style="display:none">
<button class="task-btn" data-task="comparison" data-cat="sym">Comparison</button>
<button class="task-btn" data-task="ordering" data-cat="sym">Ordering</button>
<button class="task-btn" data-task="pattern" data-cat="sym">Pattern</button>
<button class="task-btn" data-task="counting" data-cat="sym">Counting</button>
<button class="task-btn" data-task="legend" data-cat="sym">Legend</button>
</div>
</div>
</div>
<!-- Example panel -->
<div class="ex-panel-wrap">
<div class="ex-labels">
<span>Input</span>
<span>Ground Truth</span>
<span>Model Output <span class="ex-model-tag" id="ex-model-name">GPT-I2</span></span>
</div>
<div class="ex-triple">
<img id="ex-input" src="assets/img/ex_translation_input.png" alt="Input" />
<img id="ex-answer" src="assets/img/ex_translation_answer.png" alt="Ground truth" />
<img id="ex-output" src="assets/img/ex_translation_output.png" alt="Model output" />
</div>
<div class="instruction-block" style="margin-top:14px">
<div class="label">Instruction</div>
<div id="ex-instr"></div>
</div>
</div>
</div>
</div>
</section>
<!-- ── §03 Results heatmap ────────────────────────────────────── -->
<section id="results">
<div class="wrap-wide">
<div class="wrap" style="padding:0">
<div class="sec-num">03 / Results</div>
<h2 class="sec-title">The benchmark is far from solved.</h2>
<p class="sec-body">
The highest-performing model reaches only <strong>17.1%</strong> (mean IoU) overall.
Geometric transformation, most structural manipulation, and formula-based color changes are consistently hard.
</p>
</div>
<div class="heatmap-scroll">
<table class="heatmap-table">
<thead>
<tr>
<th rowspan="2" class="th-model" style="vertical-align:bottom">Model</th>
<th rowspan="2" class="th-overall">Overall</th>
<th colspan="5" class="cat-geo">Geometric Transformation</th>
<th colspan="5" class="cat-str">Structural Manipulation</th>
<th colspan="5" class="cat-col">Color Change</th>
<th colspan="5" class="cat-sym">Symbolic Reasoning</th>
</tr>
<tr>
<th>Translation</th><th>Rotation</th><th>Reflection</th><th>Scaling</th><th>Shearing</th>
<th>Construction</th><th>Removal</th><th>Copying</th><th>Border</th><th>Cropping</th>
<th>Recolor</th><th>Flood Fill</th><th>Blending</th><th>Gradient</th><th>Point Ops</th>
<th>Comparison</th><th>Ordering</th><th>Pattern</th><th>Counting</th><th>Legend</th>
</tr>
</thead>
<tbody id="hm-body"></tbody>
</table>
</div>
</div>
</section>
<!-- ── §04 Failure modes ───────────────────────────────────────── -->
<section id="failures">
<div class="wrap">
<div class="sec-num">04 / Room for Model Improvement</div>
<h2 class="sec-title">Model failure patterns revealed.</h2>
<p class="sec-body">
Detailed diagnostics reveal both universal failure patterns and model-specific behaviors. We find brittleness in performance to several scene variations: object count, background complexity, color scheme, and edit-region size.
</p>
<div class="failure-grid">
<div class="failure-card">
<div class="failure-cap-labels">
<span>Input</span><span>Ground Truth</span><span>Model Output</span>
</div>
<div class="failure-imgs">
<img src="assets/img/fm_omission_input.png" alt="Input" />
<img src="assets/img/fm_omission_answer.png" alt="Ground truth" />
<img src="assets/img/fm_omission_output.png" alt="Model output" />
</div>
<div class="failure-body">
<div class="failure-mode no-edit">Execution Omission</div>
<div class="failure-model">Qwen-Image-Edit · Cropping</div>
<div class="failure-instr">
<mark class="tgb-hl">Crop to the interior</mark> of the outlined region.
Scale to fill the canvas using nearest-neighbor interpolation.
</div>
</div>
</div>
<div class="failure-card">
<div class="failure-cap-labels">
<span>Input</span><span>Ground Truth</span><span>Model Output</span>
</div>
<div class="failure-imgs">
<img src="assets/img/fm_color_input.png" alt="Input" />
<img src="assets/img/fm_color_answer.png" alt="Ground truth" />
<img src="assets/img/fm_color_output.png" alt="Model output" />
</div>
<div class="failure-body">
<div class="failure-mode color-imp">Color Imprecision</div>
<div class="failure-model">LongCat-Edit · Removal</div>
<div class="failure-instr">
<mark class="tgb-hl">Remove all shapes</mark> except those that are a yellow heart.
</div>
</div>
</div>
<div class="failure-card">
<div class="failure-cap-labels">
<span>Input</span><span>Ground Truth</span><span>Model Output</span>
</div>
<div class="failure-imgs">
<img src="assets/img/fm_misalignment_input.png" alt="Input" />
<img src="assets/img/fm_misalignment_answer.png" alt="Ground truth" />
<img src="assets/img/fm_misalignment_output.png" alt="Model output" />
</div>
<div class="failure-body">
<div class="failure-mode struct-imp">Structural Imprecision</div>
<div class="failure-model">FLUX.1-Kontext · Shearing</div>
<div class="failure-instr">
<mark class="tgb-hl">Shear the hexagon</mark> so its top bounding box edge shifts left by 66% of its bounding box width, keeping the bottom edge fixed.
</div>
</div>
</div>
<div class="failure-card">
<div class="failure-cap-labels">
<span>Input</span><span>Ground Truth</span><span>Model Output</span>
</div>
<div class="failure-imgs">
<img src="assets/img/fm_catastrophe_input.png" alt="Input" />
<img src="assets/img/fm_catastrophe_answer.png" alt="Ground truth" />
<img src="assets/img/fm_catastrophe_output.png" alt="Model output" />
</div>
<div class="failure-body">
<div class="failure-mode catastroph">Structural Catastrophe</div>
<div class="failure-model">LongCat-Edit · Reflection</div>
<div class="failure-instr">
<mark class="tgb-hl">Reflect the gray shape</mark> across the top-left to bottom-right diagonal of its bounding box.
Place the transformed shape underneath any possible overlapping shapes.
</div>
</div>
</div>
</div>
</div>
</section>
<!-- ── §05 TinyGrafixBench ────────────────────────────────────── -->
<section id="tinygrafixbench">
<div class="wrap">
<div class="sec-num">05 / Generalization to Applied Tasks</div>
<h2 class="sec-title">TinyGrafixBench</h2>
<p class="sec-body">
Do PaintBench scores generalize to applied tasks? To find out, we create TinyGrafixBench, a data visualization editing eval adopting the same seed-gen, pixel-eval philosophy. Model scores strongly correlate with those on PaintBench,
suggesting generalization to applied precise editing tasks.
</p>
<!-- TGB auto-scroll marquee -->
<div class="tgb-marquee-wrap">
<div class="tgb-marquee-fade-top"></div>
<div class="tgb-marquee-track" id="tgb-marquee-track"></div>
<div class="tgb-marquee-fade-bot"></div>
</div>
<p class="tgb-hm-label">Mean IoU scores displayed. Scores strongly correlate with those on PaintBench (linear regression R² = 0.91, p < 0.001).</p>
<div class="heatmap-scroll">
<table class="heatmap-table tgb-hm">
<thead>
<tr>
<th class="th-model" style="vertical-align:bottom">Model</th>
<th class="th-overall">Overall</th>
<th>Bar Chart</th>
<th>Scatter</th>
<th>Line Chart</th>
<th>Heatmap</th>
<th>Network</th>
</tr>
</thead>
<tbody id="tgb-hm-body"></tbody>
</table>
</div>
</div>
</section>
<!-- ── Citation ────────────────────────────────────────────────── -->
<section id="citation">
<div class="wrap">
<div class="sec-num">Authors</div>
<p class="authors">
Kai Xu<sup>*</sup> · Ellis Brown<sup>*</sup> · Shrikar Madhu ·
Rob Fergus · He He · Saining Xie
</p>
<p class="affil"><span style="opacity:.7">* Equal contribution</span> · New York University</p>
<div class="sec-num">Citation</div>
<div class="cite-block">
<button class="copy-btn" id="copy-btn">Copy</button>
<pre id="cite-text">@article{paintbench2026,
title={PaintBench: Deterministic Evaluation of Precise Visual Editing},
author={Kai Xu and Ellis Brown and Shrikar Madhu and Rob Fergus and He He and Saining Xie},
year={2026},
eprint={2606.00188},
archivePrefix={arXiv},
primaryClass={cs.GR},
url={https://arxiv.org/abs/2606.00188},
}</pre>
</div>
</div>
</section>
<!-- ── Footer ──────────────────────────────────────────────────── -->
<footer>
<div class="wrap">
PaintBench · New York University · 2026
|
<a href="https://arxiv.org/abs/2606.00188" target="_blank">Paper</a>
·
<a href="https://huggingface.co/datasets/PaintBench/PaintBench" target="_blank">Dataset</a>
·
<a href="https://github.com/PaintBench/PaintBench" target="_blank">Code</a>
</div>
</footer>
<script src="assets/js/main.js"></script>
</body>
</html>