ScreenProof · a benchmark for phone UI faults without a reference

ScreenProofWhere is this screen wrong? Finding UI faults without a reference

Huiming Sun¹ · Yue Li² · Kangning Yang¹ · Ruineng Li¹ · Jie Cai¹ · Lan Fu¹ · Yunwan Fu³ · Jinlong Li¹ · Jinglin Shen¹

¹OPPO US Research Center · ²Cleveland State University · ³Shanghai Jiao Tong University

Text off the screen: the faulty area of the iOS chat page, enlarged — **Figure 1.** ScreenProof at a glance: one test example per fault family. On top, the faulty area enlarged; below, the whole screen. The corner marks are the data engine's exact label: every fault is made in page code, so its box is exact. A clean page has nothing to find. The last card gives the size of each set.

The whole iOS chat page — **Figure 1.** ScreenProof at a glance: one test example per fault family. On top, the faulty area enlarged; below, the whole screen. The corner marks are the data engine's exact label: every fault is made in page code, so its box is exact. A clean page has nothing to find. The last card gives the size of each set.

Abstract

A phone screen can be wrong in ways that need no reference: a label that runs off the screen, letters that touch, a line that breaks one character before its end, one button with sharp corners among round ones. We ask whether a model can box every such fault on a screenshot, given no target design and no reference image. Such a critic could serve as a cheap first pass over generated pages before a designer reviews them, but only if it stays silent on clean pages.

ScreenProof poses this target-free critique over three families of faults: general faults, local style faults (one element breaks the style its page sets) and text layout faults (letter spacing, line height, line breaks). A data engine makes each fault in page code and records the changed element's box, and hard negatives change every sibling alike.

No frontier judge is reliable: frontier models flag 18–62% of clean real screenshots and 18–82% of clean generated pages, find 47–72% of text layout faults, and rarely find a foreign icon (5–19%). A small baseline critic trained on ScreenProof-Train stays silent on clean pages and has the highest F1 on the consistency test, but still misses many local style and text layout faults.

Benchmark

The task

The input is a screenshot and nothing else: no target design, no reference image, no specification. The output is a verdict (clean or faulty) and a set of findings, each with a box, the element type, the fault family and a short reason. For local style faults the page is its own reference: an element is wrong because it differs from its siblings, so no brand knowledge is needed.

General

Wrong in any design, visible without a target.

overflowoverlapordermissing itemtwo selectedcontrasttap sizetypefacebroken imagenull valueneedless ellipsis

Local style

One element unlike the rest of its own page.

corner radiusmaterialaccenticoncomponent from another design system

Text layout

One text element set badly.

characters touchlines touchshort label wrapslone last wordword brokenneedless ellipsisCJK punctuation at line start

Splits

No test or validation page requirement occurs in the training set. Three page requirements per design language are held out for testing.

Consistency test

Clean pages · 607
Real screenshots120
Generated pages487
General faults · 920
Display (cropped image, null value, ellipsis)347
Overlap115
Two selected in a single-choice group111
Another typeface77
Low text contrast66
Reordered sections65
Shrunk tap target48
Text overflow47
Dropped item44
Local style faults · 1,238
Transplanted component662
Icon188
Accent179
Material123
Corner radius86
All screenshots2,765

One fault per faulty page.

Text layout test

Text layout faults367
Clean pages75
Hard negatives71
All items513

English pages on held-out page requirements. In a hard negative every sibling changes alike, so a critic must stay silent.

ScreenProof-Train

Clean page7,193 rows · 2,605 img
Designed difference (faulted twin)630 · 591
General fault1,542 · 1,427
Local style fault1,609 · 1,505
Transplanted component2,110 · 1,484
Text layout fault972 · 847
Page structure518 · 444
Two to four faults2,711 · 2,521
All17,285 · 11,424

From 990 pages: 542 generated and 448 public web pages. A row is an image with its labels; an image can appear in several rows.

Data engine

Faults are made in HTML page code, so the edit is the label. The engine makes exactly one change, reads the changed element's rendered box from the DOM, and re-renders the page identically on any machine.

One generated ColorOS 17 page and seven copies, each with one engine-made fault boxed. — **Figure 2.** The data engine on one generated ColorOS 17 page. Each copy holds one fault, made in code; the box is the changed element's rendered box. Brand faults (left) need a named target and form the brand tracks; general faults (right) need none.

Sources

Real screenshots come from each vendor's own public pages: product and release pages, user guides, newsroom images and design-guideline sites. Every image is recorded in a manifest with its source page, date of collection, terms of use, size and hash. An image is kept only if it is a single portrait phone screen, and it is labelled with the system version its page names.

In the snapshot of 23 October 2026 the manifest held 8,755 images, of which 458 were accepted (352 with a named version). Screens from public UI datasets (3,327 files, mostly iOS-1K, Design2Code and Sketch2Code) are recorded in the same manifest with their licences.

One accepted public screenshot for each phone design language at the target version. — **Figure 3.** One accepted public screenshot per design language at the target version, with the version, the screen and its source.

Format and evaluation

One JSON answer per screenshot

A label has the same form as a model's answer. Boxes use 0–1000 screen coordinates.

{
  "verdict": "faulty",
  "findings": [{
    "box": [112, 408, 896, 452],
    "element": "list row",
    "family": "local style",
    "reason": "square corners; its sibling
               rows are rounded"
  }]
}

Illustrative example of the schema; the box values are not from a real item.

LocalizationFindings and labelled elements are matched one-to-one at box IoU ≥ 0.5, so a small box inside a large faulty element does not count. Scores at IoU ≥ 0.3 are reported beside them.
MetricsPrecision, recall per family and type, box F1, and the share of clean pages and of hard negatives a judge flags.
Silence on clean pagesA useful first pass flags at most 15% of real and of generated clean pages (the gate).
Same promptEvery page is scored without a target and with the same prompt. Unparsable answers count as finding nothing.

Leaderboard

Results from the paper

Five frontier models, nine open VLMs, the open base Qwen3.5-9B and our baseline ProofJudge, all with the same prompt and no target. Local models run greedy without thinking; API judges run at default reasoning. ProofJudge is trained on ScreenProof-Train and marked as our baseline.

Judge	Real clean flagged ↓	Generated clean flagged ↓	Recall general ↑	Recall local style ↑	Precision ↑	F1 ↑	F1 @.3 ↑	No answer ↓

2,765 screenshots, 607 clean (120 real, 487 generated); IoU ≥ 0.5. Red dot: over half of the clean pages flagged. Green bold: best in the column among judges that find faults. No answer: unparsable outputs, scored as finding nothing.

Judge	Faults found ↑	Clean flagged ↓	Hard negatives flagged ↓	Word broken ↑	Needless ellipsis ↑	Characters touch ↑	Lone last word ↑
Claude Opus 5.5 (medium)	72%	80%	96%	98%	82%	73%	55%
gpt-6.1-sol	54%	49%	68%	100%	81%	27%	25%
Gemini 4 Argon	47%	32%	56%	86%	81%	17%	22%
ProofJudgebaseline	43%	15%	13%	94%	96%	5%	14%

367 faults, 75 clean pages, 71 hard negatives; IoU ≥ 0.5. At IoU ≥ 0.3: Opus 81%, gpt-6.1-sol 56%, Argon 51%, ProofJudge 44%. Opus finds the most faults but boxes almost every page.

Judge	Recall ↑	Normal flagged ↓	Precision ↑	F1 ↑	Accuracy ↑
API judges
Gemini 4 Argon	96%	28%	77%	0.86	84%
Claude Opus 5.5 (medium)	99%	53%	65%	0.79	73%
gpt-6.1-sol	100%	63%	61%	0.76	68%
Gemini 3.8 Flash	99%	64%	61%	0.75	68%
Trained on ScreenProof-Train
ProofJudgebaseline	71%	25%	74%	0.72	73%
Qwen3.5-9B base (no training)	93%	97%	49%	0.64	48%
Open VLMs, zero-shot
GLM-4.6V	98%	66%	60%	0.74	66%
Qwen3-VL-8B	76%	33%	70%	0.73	72%
Qwen3.8-27B	99%	100%	50%	0.66	50%
Qwen3-VL-32B	89%	80%	53%	0.66	55%
Mistral-Small-3.2	98%	99%	50%	0.66	50%
InternVL3.5-38B	98%	100%	50%	0.66	49%
InternVL3.5-8B	93%	93%	50%	0.65	50%
Qwen3-VL-30B-A3B	52%	57%	48%	0.50	48%
UI-TARS-1.5-7B	3%	1%	67%	0.05	51%

OwlEye: 600 real Android screenshots, half with display issues; screen-level verdicts with the consistency prompt unchanged. Flagging every screen scores F1 0.67 and accuracy 50%, so read F1 together with normal screens flagged. Transfer: a judge's rank on the consistency test predicts its rank on OwlEye: Spearman ρ = 0.58 between the F1 scores and ρ = 0.91 between the shares of clean screens flagged (21 judges and variants).

Baseline

ProofJudge is a small baseline critic for ScreenProof: Qwen3.5-9B with a LoRA, trained by SFT on ScreenProof-Train, that answers in the label format above. It keeps a finding only when it is confident. Its scores are in the leaderboard; its weights are released with the dataset.

Release

Download and license

DatasetBoth test sets and ScreenProof-Train with labels, the source manifest, the validation split and the evaluation script, on Hugging Face Datasets.
ProofJudge weightsThe LoRA adapter for Qwen3.5-9B and the clean-probability cut, on the Hugging Face Hub. coming soon
CodeThe data engine, the evaluation script with the matching rule, and the prompts. coming soon
LicenseLabels, the source manifest and the generated pages: CC BY 4.0. Code (data engine, evaluation, prompts): MIT. ProofJudge weights: license to be announced with their release. Screenshots collected from the web keep the terms of their owners and are included for non-commercial research only; the manifest records the source page, date and terms of each image.

Dataset card

Intended useEvaluate and train critics that find UI faults on phone screenshots without a reference, and study when such a critic can serve as a first pass before human review.
Out of scopeJudging a designer's work or a brand's conformance without human review; use as a production gate on its own; screens with interaction or motion.
CompositionGenerated base pages in eight phone design languages and their engine-made fault copies, public screenshots from vendor pages, and screens from public UI datasets.
LabelsEngine-made faults carry the exact box of the changed element. Clean labels rest on base pages that two judges found on-spec. A hand check of 481 test items found 85 mislabelled; without them every F1 changes by at most 0.01.
LimitationsSingle phone screens only; the text layout test is English only; scores come from model judges.
ContactHuiming Sun · huiming.sun1@innopeaktech.com

Cite

@article{sun2026screenproof,
  title   = {ScreenProof: Where Is This Screen Wrong?
             Finding UI Faults Without a Reference},
  author  = {Sun, Huiming and Li, Yue and Yang, Kangning and
             Li, Ruineng and Cai, Jie and Fu, Lan and Fu, Yunwan and
             Li, Jinlong and Shen, Jinglin},
  journal = {arXiv preprint},
  year    = {2026}
}