How can you tell if your AI agent fixed something, or if it just stopped showing up?

By Albert Garcia Pujadas · Published on November 6, 2026.

In short: when you compare two measurements of your website, each problem comes out as "still observed", "no longer observed", "new" or "not comparable". None of those labels means "fixed". And "not comparable" isn't good news either: it only says there's no way to know. I found this out measuring my own website.

Zero errors, and nothing to compare

On November 3 I measured uxmachine.app twice, a little over an hour apart. In between, we shipped a change to how UXMachine handles cookie notices.

In the first measurement, UXMachine didn't recognise the cookie notice on its own website. The report said it couldn't tell whether there was a notice on any of the ten pages. A bit embarrassing, I'll admit. In the second one it did recognise it, rejected it and said so in the report.

The summary of the second measurement started like this: 0 rules with errors and 1 with warnings. If you stop there, the site looks better. But the next line said it couldn't be compared with the earlier measurement on any of the 17 rules, because the way we measure had changed.

So I don't know whether the site is better or worse than an hour before. All I know is that the two measurements weren't made the same way. If an agent had summed that up for me as "all good", it would have told me something the measurement doesn't say.

By the way, that rule with warnings is the contrast one: there are 210 warnings, and they come from the colour swatches in our case studies, which copy on purpose the colours of the websites we measured. They're staying.

What each label means

This is how I read it when I look at a comparison:

Still observed. We saw it both times, on the same page. Careful: it doesn't have to be exactly the same element.
No longer observed. We saw it the first time and not the second, on a page that was measured both times. Maybe you fixed it. But maybe the content changed, that part now displays differently, or the agent hid it instead of fixing it. The measurement doesn't know which of those happened.
New. It shows up in the second measurement and wasn't in the first. That doesn't prove your change brought it: sometimes it's content that changes on its own.
Not comparable. The two measurements can't be put side by side, and the comparison tells you why. For example, because we changed how we measure, or because one was made in another language.

And then there's what couldn't be checked, which is kept apart and said as such. It isn't the same as not having seen it.

What I do now

I always measure through the same address and in the same language.
I give each measurement the earlier one, so it compares them.
If something comes out as "no longer observed", I open the page and look before calling it fixed. If an agent made the change, I ask it exactly what it touched.
If it comes out as "not comparable", I don't draw conclusions. I read the reason and, if the way of measuring changed, I measure again and take that measurement as the starting point.
I don't ask the agent for a summary: I ask what came out for each rule, and why.

The data

First measurement of uxmachine.app, November 3, 2026, at 14:35 (Madrid time). Cookie notice: we couldn't tell whether there was one.
Second measurement of uxmachine.app, November 3, 2026, at 15:52. Cookie notice: rejected once, and that rejection was kept on the other pages.
In each measurement, ten pages, viewed on a computer (1280×800) and on a phone (390×844).
Comparison: not comparable on all 17 rules, because the way we measure changed.
How we measure.

← All notes