Your agent reads a page you don't see
By Albert Garcia Pujadas · Published on .
When you ask an AI agent to look at a website for you, you assume it sees what you see. We measured it on 400 software company sites. It can almost always get in, and it almost always has the text. The problem is different: it reads more than you see, and it doesn't know which part is yours.
What we measured
- 400 software product sites from a public list (Tranco crossed with Wikidata). This is not "the web": these are software companies.
- Each home page, on 14 November 2026, with our identified robot, at 1280 pixels wide.
- Three things: whether an agent can get in, whether the text is in the page without running JavaScript, and how much text in the page a person can't see.
Getting in is not the problem
- 9% block at least one robot that trains models.
- None specifically block agents acting for a person. The 4 that do block everyone.
- What does stop agents is the security layer in front of the site. 26 home pages answered us with a block; in 11 of them, robots.txt didn't forbid it.
- Almost half have an llms.txt file. It's a proposal, not a standard, and we don't know whether agents read it.
Reading the text usually isn't either
On most home pages, what you see is already in the HTML. Only 14 of 327 depend on JavaScript so much that an agent without a browser sees almost nothing.
The problem: it reads too much
- On 1 in 4 home pages there is more text that exists only hidden than visible text. We don't count text meant for screen readers or text that also appears visibly elsewhere on the page.
- It's ordinary text: dropdown menus, closed tabs, carousels, dialogs and the cookie notice. Nothing malicious.
- But the agent can't tell what's in front of you. One example: on 14 November, Asana's pricing page showed the annual-billing price when opened. The monthly-billing price was also in the page, hidden behind a toggle. An agent reading the text gets both.
- We looked for hidden instructions aimed at agents. We saw none on these home pages. Not seeing them doesn't prove there are none: our pattern is narrow.
What to do with this
- If you build sites: keep what matters (price, what's included, terms) visible without opening anything.
- If you use an agent to decide: ask it to tell you where on the page it found what it's telling you.
Limits
One date, one pass, home pages only, one width, cookie notice not closed, and only what our robot receives. The figures depend on how "hidden" is defined: a different definition gives different numbers. We're testing with real agents whether this hidden text makes them get things wrong. We'll report it, whatever the result.
Correction, 14 November 2026: the first version of this note said "6 in 10". That figure also counted text meant for screen readers and menus that appear twice on the page. Without them, it's 1 in 4.