AI User Testing: What It Catches, What It Misses, and What It Is Not

AI user testing means pointing an agent at your product with a task like “sign up and invite a teammate”, and letting it work out its own way there while you watch. It’s a young category with loud claims attached, so this page sticks to what it catches, what it misses, and what it gets confused with.

Disclosure before anything else: I build Usero, a feedback tool, and we’ve been experimenting with this. Every number below comes from runs we did ourselves, and I’ve said where each one came from.

What a run looks like

A run is a loop. The agent looks at the page, decides one action toward the goal, takes it, and looks again. Click, type, scroll, wait. It keeps a note of what it expected each time and what it got instead, and when those disagree, that is a candidate finding. At the end you get the trace, usually a recording, and a list of the friction it hit.

Nobody wrote the path down. That is the difference from a test suite, and a run can wander into a flow you forgot existed. Two runs of the same task on our product took different routes, one through the dashboard and one through the forms area. Neither route was in a spec.

What it gets confused with

“AI” and “user testing” are attached to at least three products that do unrelated work, and buying the wrong one is easy.

Synthetic participants (Synthetic Users, and the interview-simulation features inside broader research tools) have a language model play a persona and answer your interview questions. There is no browser and no product. You describe a concept, you get responses. Nielsen Norman Group published a hard look at this in June 2024 and concluded that synthetic users “cannot replace the depth and empathy gained from studying and speaking with real people”, and that they “often provide shallow or overly favorable feedback”. Worth reading before you spend anything in that category.

Agentic QA (Ranger, Momentic, QA Wolf and friends) runs regression against flows you specified, using a model to keep the selectors from rotting. The question it answers is “does this still work”.

AI user testing, the subject here, is an agent attempting a goal against your running product. The question it answers is “where did the path confuse it”, which is a different question from whether the flow passes. Loop11 shipped browser agents against live sites in March 2025, Testers.ai runs agents that return annotated evidence, and there is a body of academic work (UXAgent, uxCUA) that is honestly ahead of most of what is on sale.

What it catches

One category, structural friction: dead ends, broken flows, labels pointing at the wrong thing, missing affordances, errors you cannot back out of. Narrower than the marketing around it suggests, and more useful.

Two findings from our own runs are the clearest illustration I have. The first: a “Connect GitHub” button on our preview deploy pointed at a GitHub App slug that had been renamed, so it 404’d. The wrong value lived in a CI variable, outside the repository entirely. No grep, no typecheck, no code review and no test can see it. Only clicking the button on a real deploy can.

The second: a dialog whose close X was unclickable, because a sticky header in one component painted over an absolutely positioned close button in another. Both files are correct on their own. They are only wrong in the same render, and to find it you need the page drawn and a click that lands on the wrong element.

The surface matters a lot. We measured the same harness on public marketing pages and on authenticated in-product flows. On marketing pages, 1 of 8 findings was real (the other seven were confident inventions about deliberate design choices). Inside the product, 5 of 8 were real, and every one of them needed something done first: a promo card that appears mid-flow and covers the submit button, a workspace dropdown that swallows clicks, a team name that saves but leaves the sidebar showing the old value. None are visible on a page load. How did we decide which ones were real? By reading the source, which tells you what a page was meant to do rather than what was on screen when the agent looked. So 5 of 8 is our best count, and I wouldn’t call it settled.

What it can’t catch

Whether it sees pixels is a choice you make, and mostly a cost one. Our harness sends screenshots to the analyst pass that assigns severity, and runs the per-step decide loop on page structure alone, because adding a screenshot to that loop took a step from about 10 seconds to 112 on Sonnet 5. Most harnesses do the same for the same reason. So in practice contrast, hierarchy, clutter and a button nobody notices go unremarked, and a control that’s in the markup and visually buried reads to an agent as present.

Give it pixels and some of that comes back. Vision models can assess contrast and clutter to a degree, and there’s academic work (PerceptUI) predicting UX answers from screenshots alone. What no amount of pixels fixes is taste, or knowing what someone in a hurry doesn’t notice.

It’s also more patient and more web-literate than any of your users. It doesn’t get bored, it doesn’t get annoyed and close the tab, and it will find the aria-label a real person never would. Across our runs it never hesitated, and there is only one thing it ever gave up on: a login screen it had no account for. It said so and quit at step 1. So a clean run is weaker evidence than a person finishing the same task cleanly.

And the measurements aren’t kind to anyone claiming more than that. Agreement between agent-identified and human-identified usability problems comes out worse than chance in the studies I’ve read. The two methods are finding different problems. That’s fine while you’re buying coverage you don’t have. It’s a disaster if you cancel your research budget over it.

There is a practical wall too. Bot protection stops an agent at signup on most modern SaaS. We got through 4 of 15, and almost every failure was a captcha rather than a usability problem. On your own staging you control that. On anyone else’s product you don’t, so “we ran a test on your site” is a weaker offer than it sounds.

If you want to try it yourself

You can build a usable version in an afternoon.

Write the task as a goal, never as steps. “Sign up and create your first project” produces a run. “Click Sign up, then enter an email” produces a slow test script.

Then spend your time on the observation layer. Ours silently manufactured false findings in two consecutive rounds before we caught it. The first version handed the agent 6,000 characters of page text plus every link URL on the page, so it answered a navigation task by reading the answer out of the string instead of clicking. Nine steps of interaction produced nothing. A later version showed it 45 controls of which 19 were invisible to a human, sitting inside collapsed cards and a hidden mobile duplicate, and the findings that came out of that were pure fiction. Own-element style checks are not a visibility test. Hit-test the point.

Cap the steps, log every action, and make the agent classify findings into a fixed taxonomy rather than writing free-form vibes. Print what your agent can see before you read a single one of its findings.

On cost: it depends almost entirely on payload size. Handing a model a screenshot plus six thousand characters of page text each step took us 112 seconds per step on Sonnet 5. A lean structural snapshot in one long-lived session was 2.9 seconds. Same task, same infrastructure, a factor of nearly forty.

So is it worth running?

For a small team with no research function and no pool of users to recruit, the alternative is usually nothing, and a method that finds real structural bugs at half precision beats nothing. For a team with a researcher and a panel, this is a canary you run after a deploy, not a line item that replaces anything.

The label I’d put on it if I were buying is flow regression testing. Less exciting than user testing, and it’s what the runs did.

The full write-up of our spike, including the runs that found nothing and the one confidently wrong finding, is in AI user testing on our own product.

Frequently Asked Questions