Trustworthy QA for AI-built apps: what counts as evidence?

"It works now." That sentence, from an agent or from a person, is a claim. It might be true. It might be true for the one path someone happened to try. The only way to tell the difference is to look at what evidence actually backs it up — and that evidence comes in more than one kind, each with a different amount of weight.

Three kinds of evidence, not one

The full breakdown lives on evidence levels, but the short version is worth internalizing on its own:

• A captured screen proves an image exists for that exact screen at that exact release. It doesn't prove the underlying behavior works — only that this is what rendered.
• A reported check proves a test runner returned a verdict. It doesn't prove the check covered the right thing, and if it was skipped or blocked, that has to stay visible instead of being dropped from a score.
• A release decision is the only one of the three that's actually a decision — a person looked at the evidence and decided, for that specific version, with a decision the release-review checklist can record. That's a human judgment your team keeps track of — not a status UIProof stores or certifies on your behalf.

A captured screen, a reported assertion, and a release decision are different kinds of evidence. Treating any one of them as if it were the other two is where "looks done" quietly turns into "shipped a regression."

Why AI-built changes make this sharper, not different

None of this is unique to AI-generated code — the same gap exists between a human's "should be fine" and a verified result. What changes is the pace: an agent can produce a plausible, confident-sounding "done" far faster than a reviewer can form an independent opinion, which makes it easy to skip straight from claim to shipped without anything in between. The fix isn't distrust — it's the same evidence discipline, applied consistently instead of skipped under time pressure.

Declare the scope before you start

For an app with roles, routes, and multiple viewport sizes, an unqualified "all screens passed" is hard to interpret. Write the intended coverage before the run — environment, role, route, state, and viewport — and give each declared case a status: captured, assertion passed, assertion failed, blocked, skipped, or not tested. A captured case can still have a failed assertion, and a missing screenshot shouldn't quietly vanish from the denominator. Start with the states that matter to the next decision — an empty state, the primary action, invalid input, a failure response, a meaningful permission boundary — rather than generating a giant matrix just to advertise a large count.

A blank box is not evidence of success

The empty-projects-scope review pattern is a small, concrete version of this problem: a zero-data list rendered as a bare rectangle is visually identical whether there are genuinely no projects yet, the request failed silently, or the page hasn't finished loading. "It renders without an error" and "it correctly explains an empty state to the person looking at it" are different claims, and only a specific, checked difference in the DOM tells them apart — not a glance.

Treat browsing permissions as part of QA

"Read-only browsing" needs an enforced contract, not an assumption — pages can trigger actions, requests, analytics, or external navigation on their own. Name the allowed origins and actions, deny unapproved mutations and destinations, and don't infer that a URL is safe to fetch just because it's public; OWASP's SSRF guidance is a useful reference for explicit destination validation. And keep saved browser authentication state out of anything public — Playwright warns it can enable account impersonation on its own.

What this means for a release decision

Before calling a release ready, it's worth being able to answer, specifically: which screens were actually captured (and which weren't), which checks were reported and by what source, and who reviewed it and when. The release-review checklist turns that into a repeatable step-by-step pass. And to be clear about what UIProof itself automates today versus what's still proposed: see product status. There's no autonomous "safe to ship" score anywhere in this — that decision stays with a person, on purpose.

See the pattern in action on a working example: explore the synthetic demo, or create a private board for your own release.

Sources and limitations

Playwright, Assertions and Authentication. OWASP, Server-Side Request Forgery Prevention Cheat Sheet. See also evidence levels. Examples in this guide are hypothetical, not customer findings; it is not a security audit, compliance opinion, or exhaustive production-readiness standard.