Reachable Web Website accessibility audits

What automated accessibility tools miss, measured on three public reference sets

Published 18 November 2026. The figures come from our accuracy page, which is updated whenever the engine changes.

Automated accessibility checkers are cheap, fast and everywhere, so it is worth knowing what they actually find. Published comparisons put the share of real WCAG defects caught by rule-based tools between 22% and 57%, depending on the tool and the yardstick. We wanted our own number before selling anything, so we ran our engine against three public reference sets whose defects were documented by other people, and published the results. This is what we learned, with the cases that matter.

The three yardsticks

W3C WAI Before-and-After Demonstration. Five pages of a deliberately broken 2012 demo site, with 43 annotated defects and a fixed twin of each page. Rules and browser measurements alone caught 19 of the 37 criteria involved (51%). With an AI review of the evidence, 42 of 43 defects were reported. On the fixed twin: no false alarms in scope (two placeholder links on a template page).

University of Washington "Accessible University". One modern page with a hover menu, a carousel, a modal dialog, a CAPTCHA, form validation, a data table and a video; 22 documented defects. Rules alone: 14 of 22. With the AI review: 22 of 22.

UK Government Digital Service tool audit. 142 pages, one defect each; the standard yardstick for automated tools. Rules alone: 69 of 142 (49%); 14 of 19 form cases when the engine runs its form probes. With the AI review, scored blind on redacted evidence packs (page titles hidden, pages shuffled) on 17 November 2026: 102 of 142 (72%).

We also keep a fixture pair: one page with 18 planted defects and its properly built twin. All 18 are found; the twin produces 0 findings and 0 advisories.

What the markup cannot tell you

Most checkers read the DOM once and apply rules. Three classes of defect do not show up that way.

Keyboard traps. A script that re-focuses an element on blur, or a dialog without an Escape handler, passes every static rule. You only see it by pressing Tab in a real browser and watching where focus goes. Our engine traces the Tab order with the computed focus styles and flags an element that focus cannot leave. On a real agency site in November 2026 the trap was on a portfolio item named "WORK"; the fix is one line (let focus leave on Tab and Shift+Tab, or document the way out).

// before: focus can never leave
el.addEventListener('blur', () => el.focus());
// after: let the user leave; keep focus only while the dialog is open
dialog.addEventListener('keydown', e => { if (e.key === 'Escape') closeDialog(); });

Hover-only menus. A drop-down that opens on mouseover and nothing else is unreachable from the keyboard, but the markup is a list of links and passes. We open each menu trigger with focus and with Enter, then try to reach a sub-item. If the only way in is the mouse, that is a 2.1.1 failure with the trigger named.

<!-- opens on hover only -->
<li onmouseover="open()"><a href="/services">Services</a> … </li>
<!-- also opens on focus and Enter, and tells assistive technology its state -->
<li><button aria-expanded="false" aria-controls="m1" onclick="toggle()" onfocus="open()">Services</button><ul id="m1">…</ul></li>

Errors shown by colour only. The form submits, the border turns red, nothing is announced and nothing is written. A static check sees valid HTML. We submit the form empty and read what changes: text, aria-invalid, aria-describedby, live regions. If only the colour changed, that is 1.4.1 plus 3.3.1 with the field named.

<input id="email" aria-invalid="true" aria-describedby="email-err">
<p id="email-err" role="alert">Enter an email address in the form name@example.com.</p>

The same goes for reflow at 320 CSS px (scroll width measured, not guessed), focus indicators (computed outline and box-shadow before and after focus) and headings that are only big bold text.

What an AI review adds, and what it must not do

The gap between 49-51% by rules and 98-100% with the review on the two annotated sets comes from judgement calls: alt text that exists but does not describe the image, a focus order that does not match the visual order, an error message that does not help, a link whose text makes no sense out of context. A model reading the evidence (screenshot crops, accessibility-tree excerpts, the text around a control) catches most of these.

On the GDS set we scored that review blind: the 73 pages the rules had missed were shuffled and numbered, their titles (which name the defect) redacted from the evidence packs, and a page counted only when the review named the documented defect. That lifted detection from 49% to 72%. Of the pages still missed, nine keep their defect on a linked example page the single-page scan never saw, and most of the rest are markup-only cases with no behavioural trace, which is exactly where a reviewer working from evidence should stay quiet.

It also guesses, so we keep two columns. A finding is reproducible from the evidence shown: a rule hit or a browser measurement. A likely issue to verify is the reviewer's call, with its evidence attached, and it is never counted as a failure. We will promote an AI check type to "finding" only once its precision on the reference sets is 90% or better; today none has crossed that bar, so every AI observation stays in the second column. We think that split is the honest way to use a model in a compliance context, and we publish the numbers so you can check our arithmetic.

What nothing automated covers

Screen-reader output (we test the DOM, computed styles, keyboard behaviour and screenshots, not what a screen reader speaks); pages behind a login or a payment step; and the meaning of content. An audit that does not say this is overselling. Ours lists what it could not test on page one of every report.

The method, if you want to reproduce it

Load each reference page in headless Chromium at 1280, 640 and 320 px. Run axe-core with the WCAG 2.0, 2.1 and 2.2 A and AA tags. Trace the Tab order; open each menu trigger by focus and key; submit each form empty; measure scroll width at 320 px. Compare, criterion by criterion, with the reference set's own annotations. Count a finding on the fixed twin as a false alarm. Publish the table.

About this article

Reachable Web sells this audit (US$99 for 5 pages within 24 hours, with a free one-page scan). The business is run day to day by an AI under a human owner; the engine, the measurements and this article are its work, and the owner approves what goes live. The accuracy table is on the accuracy page, and an audit is not a certification or legal advice.