What can't an automated tool check on your site? What UXMachine doesn't see, and why it tells you

In short: an automated tool only sees part of your site: the part that can be opened, loaded and measured the same way every time. UXMachine doesn't fill in forms, doesn't go where a password is needed, doesn't follow links and looks at each page as it stands after loading. When something stops it from checking a thing, it doesn't assume it's fine: it says so, with the reason. This note explains where those limits are and why we'd rather show them than hide them.

Silence isn't good news

Every review tool faces a temptation: show what it finds and stay quiet about what it couldn't look at. The report looks cleaner and the user walks away calmer. The trouble is that the calm is false. If a check couldn't be done, the result isn't «no problem»; it's «I don't know».

At UXMachine we made the opposite choice from the start. Every report says what was observed, but also what could not be checked and why. We call those abstentions, and to me they matter as much as the observations. A tool that never abstains isn't more thorough: it just isn't telling you where it can't reach.

What it looks at, and how

UXMachine opens your site in a real browser, at two widths: 1280 pixels, like a laptop, and 390, like a phone. It reads up to eleven pages, the home page and the ones reachable from it, and applies a fixed, public list of checks. If you repeat the measurement tomorrow on the same site, with no changes, the result has to be the same. That is the starting condition, and it explains almost all of its limits.

What it can't see

What happens after loading. UXMachine looks at each page as it stands once loaded. It doesn't press buttons or open menus. So when it checks whether a drop-down menu says if it's open or closed, it can only look at its initial state: it doesn't know whether that state changes when you tap it. And it says so, in those words, in the report itself.

What's behind a form or a password. It doesn't fill in forms or sign in. A checkout, a sign-up or your customers' dashboard are left out. It's not that they're fine or broken: it simply hasn't seen them.

Where links lead. It checks that every link has a declared destination and a name that says where it goes. But it doesn't follow them, so it can't tell you whether any of them leads to a page that no longer exists.

What a cookie banner covers. This is the limit we run into most. When we enter a site, the first thing that usually appears is the cookie banner. If it has a button to reject, we use it. If it only lets you accept, we accept, because that's what anyone would do to carry on. But some banners cover everything and won't let you through. In those cases, what's underneath can't be measured, and the report says so: there are no observations because the banner was covering the page, not because the page is fine.

Sites that don't want to be measured. If a site blocks us with an anti-bot service, or its robots.txt file asks us not to come in, we don't get around it. We don't pretend to be an ordinary browser. That site simply isn't measured, and that's what gets recorded.

When the problem is ours

There's a less obvious limit, and this week we lived it first-hand. UXMachine compares each measurement with the site's previous one to tell you what stays the same, what has gone and what is new. That comparison only makes sense if both measurements were made the same way.

Between the night of 27 October and the early hours of the 28th, a change in our own instrument altered the way pages were captured without the comparison knowing. As a result, some measurements from those hours could flag as «new» something that hadn't changed on the site. We spotted it, switched it off and then fixed it properly: now, when one of those measurements is compared with one made differently, the comparison says «not comparable», with its reason, instead of inventing a change.

I'm telling this because it's exactly the kind of limit a tool tends to hide. Saying «not comparable» is less eye-catching than saying «new», but it's the truth.

What no automated tool does

Beyond the technical limits, there are things a mechanical check can't judge. It doesn't know whether your copy makes sense. It doesn't know whether your offer is convincing, whether the design inspires trust or whether the path to buying makes sense to someone arriving for the first time. It can tell you a button has no name for a screen reader; it can't tell you whether that button is in the right place.

That still takes a person looking with judgement, and often the people who use your site. UXMachine doesn't try to replace that judgement. What it tries to do is take work off it: so you don't have to check by hand, after every change, what a machine can check the same way every time.

How to read a report with this in mind

When you read a UXMachine report, or the summary your agent gives you from it, look at three things. What was observed, where and at what width. What could not be checked, and why. And which pages were measured, because anything not on that list hasn't been looked at.

If a check didn't find anything, it means it didn't find it, not that that part is fine. And if part of your site couldn't be measured, the honest conclusion isn't that it's fine or that it's broken: it's that you don't know yet.

← All notes