Asserting Meaning, Not Values: Why Software Testing Must Become Agentic
Scripted tests assume a correct program has one correct output for each input. That assumption is failing.
Yunhao Jiao and Rui Li36 分で読めます

Abstract
Scripted end-to-end frameworks such as Playwright, Cypress and Selenium encode a test as a fixed path through an interface plus an assertion that compares what the program shows with a value the test author knew in advance. That contract assumes a correct program has one correct output for each input. The assumption is failing. Products now embed language-model agents whose correct answers form large sets defined by meaning rather than by form. Test-generation agents produce artifacts whose size and structure change from run to run on identical input. Interfaces written by coding agents change faster than hand-maintained locators can follow.
We show that under these conditions a scripted assertion can only sit at one of two ends. At one end, exact checks reject correct behavior and the suite becomes flaky. At the other, existence checks accept incorrect behavior and the suite becomes vacuous. We call the predictable slide toward the second end the existence collapse. We argue that escaping it requires three capabilities scripted tests lack by construction: oracles derived from intent, navigation by goal, and verdicts stated as measured rates. Together these define agentic testing. We describe the approach, show it on two running examples, and state where deterministic assertions should remain.
1 Introduction
A retailer ships an AI support agent. On launch day a customer types: “I bought a laptop 45 days ago. Can I still return it?” The store’s policy allows returns within 30 days. The agent replies:
“Absolutely! Our generous 60-day return window has you covered. Just print a prepaid label from your account page and drop the laptop at any carrier location.”
The reply is polite, well formatted, arrives in under two seconds, and is wrong in a way that costs real money. The agent’s regression suite passes. It is written in Playwright, and it checks that a reply bubble appears, that the bubble is not empty, and that the loading spinner goes away. All three are true.
Nobody was careless here. The engineer who wrote the suite could not have written a stronger assertion with the tools available, because the agent never phrases the same answer twice. A check for the exact text fails on the next run. A regular expression for “30 days” fails on “thirty days” and “one month”, and passes on “your 30-day window has closed, but we’ll make an exception this time”. The only assertion that survives a week of model updates is that something was said.
This paper argues that the problem is structural, not a matter of better selectors or smarter regular expressions. Scripted tests were designed for programs whose correctness is a value: a total, a status code, a row in a table. Three developments are moving correctness away from values:
- Model-backed features. Chat agents, summarizers, recommenders and planners return outputs whose correct form is a set, often an unbounded one, defined by what the output means.
- Agents in the toolchain. The tools that produce test plans, migrations and code are themselves generative. Their outputs differ in size and structure across runs on the same input.
- Agent-written applications. When coding agents rewrite interfaces on every pull request, the paths a script hard-codes go stale faster than people can repair them.
We make four contributions. Section 2 gives a minimal model of a scripted test and names the premise it rests on. Section 3 shows three concrete ways that premise breaks. Section 4 describes the existence collapse, the process by which scripted suites lose their ability to find faults while still passing. Sections 5 and 6 define agentic testing by three properties, work through the running examples, and discuss its limits.
2 Anatomy of a scripted test
A scripted end-to-end test is a triple T = (x, π, O). Here x is the input data, π is a fixed sequence of interface actions addressed by locators (click this button, fill that field), and O is an oracle that maps the observed state y to a verdict. In Playwright, the oracle is assembled from a small vocabulary: toHaveText, toContainText, toHaveValue, toHaveCount, toHaveURL, toBeVisible, toHaveScreenshot. Every one of these either compares an observation with a constant fixed when the test was written, or checks that something is present. Two oracle shapes cover nearly all of them:
Oeq(y) = 𝟙[ y = y* ]Oex(y) = 𝟙[ y ≠ ∅ ]
The equality oracle needs a known expected value y*. The existence oracle needs nothing, and for the same reason it checks almost nothing.
The contract rests on one premise: for each input there is a single correct output, known when the test is written, and the program produces it every time. Under that premise scripted tests are excellent instruments. They are cheap to run, fast, reproducible bit for bit, and a failure points to a line. Most of the last twenty years of web testing practice, from page objects to visual snapshots, is engineering on top of this premise.
The premise has always had exceptions. Weyuker called programs for which no practical y* exists “non-testable” [1], and Barr et al. survey four decades of work on what they name the oracle problem [2]. Interface testing largely avoided it, because a web page usually rendered values that a database already held: the cart total, the order status, the user’s name. The test could ask the database what the page should say. Model-backed features remove that shelter, because the value on the page was never stored anywhere before the model produced it.
3 Where determinism breaks
The premise fails in three distinct ways. Each one is already routine in production software, and each one defeats a different part of the triple (x, π, O).
3.1 Answers that are sets
The correct reply to the laptop question is not a string. It is any reply that does three things: states the 30-day window, declines the return for a purchase made 45 days ago, and points the customer to a next step such as warranty repair. Thousands of phrasings satisfy that description, and thousands of equally fluent phrasings violate it. Here is the strongest test that stays green across model updates:
test('support agent answers a return question', async ({ page }) => { await page.goto('/support'); await page.getByRole('textbox', { name: 'Ask us anything' }) .fill('I bought a laptop 45 days ago. Can I still return it?'); await page.getByRole('button', { name: 'Send' }).click(); const reply = page.getByTestId('agent-reply').last(); await expect(reply).toBeVisible({ timeout: 15_000 }); await expect(reply).not.toBeEmpty(); // The assertion we need, and cannot write: // await expect(reply).toBeCorrectAboutTheReturnPolicy();});The missing line is the whole test. The two that remain would pass if the agent answered in French, quoted the wrong policy, or recommended a competitor. In terms of Section 2, the answer set A(x) is large and defined by meaning, so no constant y* exists to compare against.
3.2 Outputs whose size is not fixed
The second case comes from inside the testing toolchain itself. A test-planning agent reads a product requirement document (PRD) and proposes test cases. Feed it the same PRD for a checkout flow with coupon codes on three consecutive days:
| Run | Cases | What changed | toHaveLength(22) | length > 0 | Every critical requirement covered? |
|---|---|---|---|---|---|
| Monday | 22 | Baseline | Pass | Pass | Yes |
| Tuesday | 31 | “Invalid coupon” split into expired, malformed and already redeemed | Fail | Pass | Yes |
| Wednesday | 30 | Extra cases on totals and rounding; no case for an expired coupon | Fail | Pass | No |
Tuesday’s plan is better than Monday’s and fails the count assertion. Wednesday’s plan is worse and passes everything a script can express. The property that matters, every critical requirement covered by at least one runnable case and no case asserting behavior the PRD never asked for, is a statement about the meaning of two sets: the requirements and the cases. A count says nothing about either. A snapshot of the plan fails on every run, because titles, ordering and granularity legitimately differ. Anyone who ships a generative testing tool, including the authors, faces this question directly: how do you regression-test a component whose correct output for one input might be 20 cases or 30?
3.3 Paths that move
The third case breaks the path π instead of the oracle. A coding agent refactors the checkout page in one pull request. The button labeled “Place order” becomes “Complete purchase” and moves into a sticky footer. In the same change, tax starts being computed on the subtotal before the coupon discount instead of after it.
await page.getByRole('button', { name: 'Place order' }).click(); // times out: the button was renamedawait expect(page.getByTestId('order-total')).toBeVisible(); // would pass: the total is wrong, but visibleThe suite fails for the wrong reason and would have passed for the wrong reason. The person who repairs the locator will most likely leave the visibility check alone, and the tax bug ships. Empirical studies of web test suites found locator changes to be the leading cause of breakage in record-and-replay tests [3], and repairing tests as the application evolves to be a large share of their total cost [4]. When agents write the application, that churn is no longer occasional; it happens on most pull requests.
4 The existence collapse
When an equality assertion starts rejecting correct behavior, teams do the reasonable thing and relax it. Exact text becomes a substring, the substring becomes a regular expression, and the expression becomes “not empty”. Each step removes a false alarm and adds a blind spot. The process stops at the only assertion that never fails on a correct output: something exists. We call this the existence collapse.
Two error rates make the trade explicit. For an oracle O and a system S:
α(O) = Pr[ O fails | S behaves correctly ]β(O) = Pr[ O passes | S has a fault ]
Here α is the false-alarm rate and 1 − β is the power to detect a class of faults. For a deterministic program, a well-chosen equality oracle keeps both near zero, which is why scripted testing worked so well. For a model-backed feature whose correct answers spread over a set A(x), an equality oracle against one expected answer y* fails every correct answer that is not y*:
α(Oeq) = 1 − Pr[ Y = y* | Y ∈ A(x) ]
This approaches 1 as A(x) grows. An existence oracle has α near 0, but for any fault that still produces fluent output, such as quoting the wrong return policy, its β is near 1.
Every assertion in the scripted vocabulary sits somewhere on the line between these two points, because the vocabulary can only vary how much of the observed form must match (Figure 1). No point on that line checks whether the output means the right thing. Moving along the line trades α for β. Getting off the line requires a different kind of oracle.
The collapse also changes what flakiness is. The flaky-test literature treats nondeterminism as an accident of the harness: asynchronous waits, concurrency, dependence on test order [5]. Practitioners operating at scale already advise assuming that every test is flaky [6], and the standard remedies are retries, quarantine and better waits. For a model-backed feature the nondeterminism is essential: it is the product working as designed. Retrying an equality assertion until it passes does not remove a timing artifact. It samples the model until it happens to produce the expected string, which hides exactly the variance a test should be measuring.
The collapse is quiet. A collapsed suite stays green, keeps its coverage numbers, and still runs on every commit. Its cost appears only when a semantic fault reaches users, and by then the dashboard has been green for as long as the fault existed.
5 Agentic testing
We define an agentic test as T = (g, I, J, n, τ): a goal g stated in natural language, a set of intent sources I (requirements, policy documents, the PRD, earlier decisions), a judge J that decides whether an observed trajectory satisfies the intent and returns the evidence for its decision, a sample budget n, and a pass threshold τ. The executor that pursues g, the judge, and the planner that writes the tests may each be a language-model agent. Three properties separate this from a scripted test, and each answers one of the breakdowns in Section 3.
P1. Oracles derived from intent. The expected behavior is not frozen as y* when the test is written. It is derived from I when the test runs, as a rubric of checkable claims. For the laptop question, the rubric comes from two sentences of the store’s policy:
| Criterion | Source in the policy | Verdict on the 60-day reply | Evidence quoted by the judge |
|---|---|---|---|
| R1. States the return window as 30 days | “Items may be returned within 30 days of delivery.” | Fail | “Our generous 60-day return window” |
| R2. Declines a return for a 45-day-old purchase | Same sentence | Fail | “has you covered” |
| R3. Offers a next step the policy allows | “Items outside the return window may be eligible for warranty repair.” | Fail | No mention of warranty or repair |
The rubric accepts every correct phrasing, including “thirty days” and “one month”, and rejects every fluent wrong one. Because the judge must quote its evidence, a person can audit any verdict in seconds.
P2. Navigation by goal. The goal “place an order for one laptop with coupon SAVE10” is reached by reading the page the way a user does, through labels, roles and layout, not by replaying a stored locator. A renamed button is not a failure; a checkout that no longer accepts coupons is. Freed from repairing locators, the test can spend its effort on outcomes. In the refactor of Section 3.3, the agent computes the expected total from the PRD’s tax rule and reports that tax was applied before the discount. Where a value is known exactly, as a total is, the agentic test asserts it exactly. The aim is not to discard equality assertions. It is to derive their expected values from intent, and to stop pretending that every output has one.
P3. Verdicts as measured rates. A stochastic system is run n times, and the verdict is a statement about a rate rather than about one sample. With k passing runs out of n, the test passes only if a lower confidence bound on the pass rate clears the threshold. We use the Wilson bound because it behaves well at small n; sequential tests in the style of Wald [7] can stop early once the evidence is decisive.
L(k, n) = ( p̂ + z2/2n − z √( p̂(1 − p̂)/n + z2/4n2 ) ) / ( 1 + z2/n )p̂ = k/npass ⇔ L(k, n) ≥ τ
Suppose the support agent answers the laptop question correctly in 7 of 10 runs. The test does not flicker between green and red. It reports a pass rate of 0.70 with a 95% interval of 0.40 to 0.89, well below a 0.95 bar, and attaches the three failing transcripts. A flaky failure becomes a measurement.
Set-valued outputs. The test-plan example of Section 3.2 combines P1 and P3. Let Rcrit be the critical requirements extracted from the PRD and P the generated plan:
pass(P) ⇔ ( ∀ r ∈ Rcrit ∃ c ∈ P : J(c, r) = covers ) ∧ ( ∀ c ∈ P : J(c, I) = grounded )
The size of P does not appear anywhere. Monday’s 22 cases and Tuesday’s 31 both pass. Wednesday’s 30 fail with a one-line reason: no case covers R7, an expired coupon is rejected at checkout. Precision can be checked the same way, by flagging near-duplicate cases or cases that assert behavior absent from I. The oracle states what the set must contain, not what it must look like.
| Scripted test | Agentic test | |
|---|---|---|
| Input | Fixed data x | Goal g plus intent sources I |
| Path | Locator sequence π, replayed | Planned at run time from what the page shows |
| Oracle | Equality with y*, or existence | Rubric derived from I, judged with quoted evidence |
| Verdict | One run, one bit | n runs, a rate with an interval |
| What a failure says | “Locator not found” or “expected X, got Y” | Which requirement failed, how often, and the transcript |
| What forces maintenance | Any change to labels, layout or wording | A change to the intent itself |
6 Objections and limits
Who tests the tester? A judge built on a language model can be wrong, and its errors are not random. Studies of model-based evaluation report position bias, a preference for longer answers, and a preference for outputs resembling the judge’s own [8]. Four practices contain this. First, calibrate: hold out a set of human-labeled trajectories and report the judge’s agreement with people before trusting its verdicts. Second, keep rubric items atomic and require quoted evidence, so each verdict is a small claim a person can check rather than an overall impression. Third, judge with a different model from the one under test. Fourth, add checks that need no expected answer at all. Metamorphic relations [9, 10] supply them: asking the laptop question as “45 days ago”, “six weeks ago” and “a month and a half ago” must yield the same decision, so a system that declines one and approves another is wrong somewhere, whatever the policy says. Behavioral test suites for language models apply the same idea [11], and property-based testing has long checked invariants instead of values [12].
Cost and latency. Running an agent n times and judging each run with another model costs more than one scripted run. Two facts limit the bill. Deterministic checks remain the base layer: unit tests, API contract tests and exact assertions on known values still run on every commit, while the agentic layer runs where meaning matters, on pull requests and before releases. The sample budget can also scale with the stakes, a handful of runs to screen a change and many more to gate a release. Confidence has a price either way. With zero observed failures, showing a failure rate below 5% at 95% confidence takes about 60 runs (the rule of three, 3/n). A scripted suite that reports green after one run is claiming that guarantee without paying for it.
Reproducibility. A scripted failure reproduces bit for bit; an agentic verdict reproduces as a distribution. That is a weaker property only if we pretend the system under test is deterministic. Agentic runs should record what is needed to replay and audit them: model versions, prompts, seeds where available, the full trajectory with screenshots, and the judge’s evidence. Two runs of the same suite should agree on rates within their intervals. When they do not, the disagreement is itself a finding.
The tester is stochastic too. The executor and the judge are models, so the instrument has its own variance. Measure it as one would any instrument: re-judge fixed trajectories, report judge consistency, and take majority votes where single judgments disagree.
Where scripted tests still win. Pure functions, protocol contracts, numeric pipelines and any interface whose correct output is a value should keep exact, deterministic tests. They are cheaper and their failures are sharper. Agentic testing does not replace them; it supplies the oracle and the path where they cannot. Dijkstra’s remark that testing can show the presence of bugs but never their absence [13] applies to both kinds. Agentic testing does not claim to prove correctness. It claims to find the faults that matter in software whose correctness lives in meaning, which scripted assertions cannot see by construction.
7 Conclusion
Scripted testing answers the question does the program produce this value? For a growing share of software that question has no fixed answer. The support agent’s reply, the planner’s test cases and the coding agent’s latest interface are each correct in many forms and wrong in many fluent ones. The question that matters is does the program do the right thing, often enough, for reasons we can inspect? Answering it takes a tester that reads the intent, finds its own way through the interface, and reports a rate with evidence. That tester is an agent.
This draft argues the case from first principles and worked examples. The full paper should measure it. We propose a benchmark of model-backed features with seeded semantic faults, such as a wrong policy, a dropped requirement or a miscomputed total, and an experiment that estimates α and β on that benchmark for three oracles: scripted assertions as practitioners actually write them, agentic oracles as defined here, and human reviewers as the reference. Section 4 makes a specific, falsifiable prediction. On semantic faults, scripted suites will show high α or high β and never both low, while agentic oracles will approach the human reference at a measurable cost per verdict.
References
- E. J. Weyuker. On testing non-testable programs. The Computer Journal, 25(4):465–470, 1982.
- E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, S. Yoo. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5):507–525, 2015.
- M. Hammoudi, G. Rothermel, P. Tonella. Why do record/replay tests of web applications break? In Proc. IEEE International Conference on Software Testing, Verification and Validation (ICST), 2016.
- M. Leotta, D. Clerissi, F. Ricca, P. Tonella. Capture-replay vs. programmable web testing: An empirical assessment during test case evolution. In Proc. Working Conference on Reverse Engineering (WCRE), 2013.
- Q. Luo, F. Hariri, L. Eloussi, D. Marinov. An empirical analysis of flaky tests. In Proc. ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), 2014.
- M. Harman, P. O’Hearn. From start-ups to scale-ups: Opportunities and open problems for static and dynamic program analysis. In Proc. IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM), 2018.
- A. Wald. Sequential tests of statistical hypotheses. Annals of Mathematical Statistics, 16(2):117–186, 1945.
- L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023.
- T. Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. H. Tse, Z. Q. Zhou. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys, 51(1), 2018.
- S. Segura, G. Fraser, A. B. Sanchez, A. Ruiz-Cortés. A survey on metamorphic testing. IEEE Transactions on Software Engineering, 42(9):805–824, 2016.
- M. T. Ribeiro, T. Wu, C. Guestrin, S. Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
- K. Claessen, J. Hughes. QuickCheck: A lightweight tool for random testing of Haskell programs. In Proc. ACM SIGPLAN International Conference on Functional Programming (ICFP), 2000.
- E. W. Dijkstra. Notes on structured programming. EWD249, Technological University Eindhoven, 1970.