AI Agent Testing: Assert Behavior, Not Wording

A red test does not mean the agent got it wrong

An agent renames ticket PROJ-142 to billing-export. One run reports this:

PROJ-142 → billing-export

Another says Renamed PROJ-142 to billing-export. A third prints a tool log and no summary at all. All three leave the same correct row in the database, and an exact string check fails two of them. The database is right. The test is red.

So the test goes red on work that was done correctly, which reads exactly like an unstable prompt and sends you off to tune wording that was never the problem. The check is what broke. It was measuring the sentence instead of the rename.

Why does checking the wording fail in both directions?

Because the words and the work come apart. It happens in both directions.

A function has one expected return value. An agent can finish the same task several valid ways, and its text drifts even when the task does not. Sampling changes a summary. A tool response changes the next command. A small prompt edit rewrites the whole transcript without touching the result. When a golden-text check fails, two things could have happened. The agent made the wrong change, or it described the right change in different words. The failure cannot tell you which.

That is the annoying direction. The expensive one is the reverse.

Some agents report success after an operation that failed or never finished. That message is clear and stable, which makes it look like the easiest thing in the run to assert against. It is also written by the system under test. Checking it is like asking a plumber whether the pipe is fixed instead of turning on the tap. The claim can stay stable while the work stays wrong.

The scale of that is measurable. Advani and colleagues checked 9,876 tau2-bench trajectories and 1,879 from AppWorld (arXiv, June 2026). In the two single-control domains, false success accounted for 45% and 48% of failures. In the dual-control domain it was 3%. On AppWorld, where ground truth comes from database state rather than from anything the agent says, the same analysis found false success in 75.8% of failures among runs that produce explicit completion signals.

Those are not wording failures. They are task failures hidden by good wording. Keep the transcript and use it to work out why something failed, but leave its prose out of the pass condition.

What can you assert instead?

Assert the process, the stored state, and the chain. Each is a boundary the agent does not write.

Each agent in AuricIDE runs as a real PTY child process, so its exit code is the actual process status rather than a generated claim about one. A test can spot a crash or a rejected command. No summary needed. That is one signal, and it is a weak one on its own, because a process can exit 0 having done nothing at all.

Project state is the second boundary. It is the stronger one. AuricIDE keeps it in a SQLite file inside the repository and exposes tickets, dependencies, requirements, test cases, goals and history over MCP. A test can query that directly, which means it never has to ask the agent whether the agent's work passed. AuricIDE's MCP server for AI workflows is the same idea one level down: the agent can claim it changed a ticket, and the ticket record is independent evidence.

The rename contract becomes four facts:

• The child process exits with code 0.
• Ticket PROJ-142 now has the title billing-export.
• The history contains the expected update.
• Where it matters, the MCP tool received the intended ticket ID and title.

None reads the final summary.

Be careful with that last one. Check tool arguments when the operation genuinely belongs in the contract, not because one successful run happened to pick that tool. Another supported operation may reach the same valid result, and pinning the strategy turns a passing alternative into a failure.

Chains are the third boundary. Each AuricIDE step receives a cleaned tail of the previous step's terminal output: ANSI codes stripped, interface chrome filtered, repeats removed, and AuricIDE caps what is left at 2,000 characters. That makes a handoff inspectable, but the tail is still prose that can change, so it belongs in debugging rather than in a fixture. The stop is the thing to assert. When step two fails, AuricIDE stops the chain, and step three never runs on stale context. A test should confirm both halves: that the failure was reported, and that the later changes did not happen anyway.

The AI agent integration guide covers connecting agent workflows to shared project operations.

What still goes wrong once you assert state?

Observable data can be checked badly, and a badly scoped check on real state passes for the wrong reason.

An exit code of 0 says the process met its own rules, not that the ticket changed. A changed database row says something changed, and it may be the wrong row or the wrong project entirely. Configuration is a third way to be fooled. AuricIDE loads each external agent CLI from a JSON dynamic provider config, imported under Settings → Agent, and a config pointing at the wrong command makes a test pass on nothing. A harmless stub returns 0. The wrong working context updates a different repository. A query written too broadly finds an old row that happened to match. (The industry term for the wrapper around an agent CLI is a harness; the dynamic provider is what AuricIDE calls its own mechanism for loading one.)

The fix is scope. Name the thing each check is about:

Then break things on purpose, one at a time. Sever an MCP connection, remove a required input, make a provider command return a failure code, and check that the chain stops and no success state appears. Watch for partial success while you are there: a failed rename must not leave a completion marker, close a dependency, or record a transition that never happened.

AuricIDE's Fleet view helps when it does go wrong. It shows one count per repository and ranks errors above agents blocked on input, and those above stalled agents, which points at the failed process rather than at the effects it left downstream.

What carries over to any other agent tool?

All of it. None of the boundaries is specific to one product.

Start from the result you expect. A file exists at a known path. A row has the values it should. A process returns the status it should. A required tool received valid arguments. Work downstream of a failure stayed stopped.

Then separate required behaviour from the agent's strategy. Command order matters when the domain says it does, like creating a resource before updating it. Everywhere else, let the agent pick its route and check where it arrived. The guide to building an MCP server covers the integration boundary that exposes project state for exactly this.

A small acceptance contract carries the whole thing:

Given ticket PROJ-142 exists
When the agent renames it to billing-export
Then the process exits successfully
And PROJ-142 has the new title
And no other ticket changes

The transcript sits beside those checks. Not inside them. It can show which row the agent picked, which input it asked for, why a tool call failed. It cannot decide whether the task passed.

So when an agent test flakes, read the check before you touch the prompt. Two transcripts describing the same correct result do not expose an unstable prompt. They expose a matcher pointed at text that can move. AuricIDE gives you three places to point it instead, and the smallest useful change is to take one chain and replace its first exact-output check with an exit code or a stored record.