Agent-Driven Tests (ADT) via SCENARIOS.md files

I’ve realized that not many people are using this kind of techniques for “it’s 2027 soon, agentic codebases need a different kind of testing”, and I’ve been told I should document it somewhere at some point, so here it is.

A lot of tests in my codebases are not written as code, they’re written as scenarios in a SCENARIOS.md file.

Agents know that they should maintain a manageable list of such scenarios that describe end-to-end tasks and expected behavior from different point of views.

Agents can decide to run through all of them, or some of them, after code changes in order to verify that some expected high-level specification is still respected.

To do that, agents have access to two things:

1. A codebase CLI that has commands to speed up running through these scenarios. Things like adding the right data in the database, running well-known db queries with the right settings, spinning up redis, etc. No end-to-end tests, just atomic things.
2. A way to orchestrate any LLM responses, so that I pay sub cost instead of actually calling the token API. Usually I do this by having my own LLM provider in pi, and a codebase CLI command that creates the server side of things and let the LLM interact with it to see requests and produce responses.

That’s it. I call it ADT for Agent-Driven Testing.