How accurate our audits are
We measure our audit engine against public reference pages whose defects were documented by other people, and we publish the numbers here before we sell anything. Last updated 17 November 2026.
Background reading: What automated accessibility tools miss, with the cases that matter and the method to reproduce these numbers.
Two kinds of result
Findings come from rules and measurements: the axe-core rule set plus our own browser measurements and interaction probes (keyboard traces, focus tests, menus, pop-ups, forms). They are counted as failures. Likely issues to verify come from an AI review of the evidence (alt text that may not describe the image, a focus order that may not match the page, an error message that may not help). They carry their evidence and are never counted as failures. Advisory notes are good practice, not WCAG failures.
Results on public reference sets
| Reference set | What it is | Rules and measurements alone | With the AI review | False alarms on the fixed version |
|---|---|---|---|---|
| W3C WAI Before-and-After Demonstration | 5 pages of a 2012 demo site, 43 annotated defects | 19 of 37 criteria (51%) | 42 of 43 defects (98%) | none in scope (two placeholder links on a template page) |
| University of Washington "Accessible University" | 1 modern page: hover menu, carousel, modal dialog, CAPTCHA, validation, data table, video; 22 documented defects | 14 of 22 | 22 of 22 | none |
| UK Government Digital Service tool audit | 142 pages, one defect each; the standard yardstick for automated tools | 69 of 142 (49%); 14 of 19 form cases in full mode | 102 of 142 (72%): the AI review, scored blind on redacted evidence (method below), adds 33 of the 73 pages the rules missed | not applicable |
| Our own fixture pair | one page with 18 planted defects and its properly built twin | all 18 | not applicable | 0 findings, 0 advisories |
Automated tools alone typically find a fifth to a half of real defects (published comparisons report 22% to 57%). Our rule layer sits at the top of that range because it measures behaviour in a real browser instead of only reading the markup. The AI review adds much of the rest (all of it on the two demo sites, 33 of the 73 rule-missed pages on the GDS set), with every finding carrying evidence you can check.
How the GDS set was scored with the AI review
On 17 November 2026 the 73 GDS pages the rules had missed were reviewed from their evidence packs (scan data, outline, computed text styles and Tab trace, no screenshots) with the page title and heading redacted and the pages shuffled and numbered. The reviewer therefore did not know which defect each page documented until scoring. A page counts only when the review names the documented defect or its direct cause. Six pages whose title leaked through the redaction and two partial matches are left out of the 102. On these pages 21 of 21 high-confidence observations were real defects; 2 low-confidence observations were wrong. Twelve of the 33 are defects the report already lists as advisories (justified text, tight line height, very small text, skipped heading levels, identical labels and the like). The rules-only count leaves those out because advisories are not failures. Of the 40 pages not counted: 9 hold their defect on a linked example page that a single-page scan does not reach. Seventeen are markup-only cases with no computed trace in the evidence (several are now rule candidates). Fourteen are pages the method excludes (leaked titles, partial matches) or where the review observed a different, real defect than the documented one.
Our rule for going live
An AI-observed check type is reported as a finding only once its precision on the reference sets is 90% or better; below that it stays a likely issue to verify. We have not yet declared that bar met for any AI check type, so today every AI observation is reported as a likely issue, never as a failure.
What we do not test
- Screen-reader output. We test the DOM, computed styles, keyboard behaviour and screenshots, not what a screen reader speaks.
- Pages behind a login or a payment step.
- The meaning of content: whether alt text describes the image well, whether an error message helps. These are reported as likely issues, with the evidence.
- A free scan never submits your forms or changes your controls; those tests run only in an audit you order for your own site.
How the numbers are made
Each reference page is scanned with the same engine customers get. The engine's own findings are compared, criterion by criterion or note by note, with the defects documented by the people who built the reference page. A finding on the fixed version of a page counts against us as a false alarm unless the criterion is outside the reference set's scope. The method, every number and its limits are kept in our calibration log, and the table above is updated whenever the engine changes.
This page describes measurements on reference pages. It is not a promise that every defect on your site will be found, and an audit is not a certification or legal advice.