I generate API tests from OpenAPI instead of writing them one request at a time. The public libraries do the plain half. open_api_import turns a spec into request hashes. create_tests turns those hashes into tests. At NetApp the same idea ran on the product spec, so the suite tracked Swagger rather than a pile of hand-written calls.
A generator tells the truth on the day it runs. You import the spec, you commit the requests, the tests exist. The next change adds a required field, removes an operation, or rewrites an error body. The files already in git still pass. They describe the contract you had, not the contract you have. Nothing in that test run opens the spec.
The frozen suite
When a test fails or the contract changes, the modern reflex is to ask the agent to patch the generated files. Agents make a particular mess of that freeze. Ask one to add the missing tests and it will write new cases next to the old ones, from the spec text in the prompt. It will not delete the cases for an operation you removed. It will not add a required field that never appeared in the chat. The folder now has two authors: the last generator run, and the model’s guesses. There is no single diff that says what the spec did.
The loop I trust is dull, and that is the point. The spec change is the input. The generator rewrites the request hashes. Git shows the result. A person reviews that diff. The agent can help read it. The agent is not the author that keeps the tests updated from memory of an older spec.
Read the diff by operation
The review is not the line count. Generated diffs are large and look like noise, and a summary that says the suite was regenerated will walk past the only line that matters: an operation that disappeared, or a field that became required. I want that review grouped by operation. Added. Removed. Shape changed. A removed operation is a small delete inside a large diff. That is the one I need a person to see.
If the agent hand-edits a generated request to make a failure go away, it has forked the suite from the spec. The next regeneration will either overwrite the edit or leave a conflict. Both are worse than refusing the edit. The fix belongs in the spec, or in the assertion that runs against the service, not in a generated hash someone smoothed over.
The spec is not the service
There is a quieter lie in tests generated from the spec that only check the spec. This field is a string. This call is declared to return 200. Those cases prove the document can produce a request. They do not prove the service matches the document. I still generate them. A spec that cannot produce a request is already broken, and I would rather fail there than in a review three weeks later. They are not the test that tells you the implementation drifted.
The test that matters sends the generated request to the service and fails when the service disagrees with the spec you just imported. Generation without that run is a document check with extra files. Hand-written tests can lie as well. A generated file is more dangerous because it looks like it came from the contract. People trust it. Other people run these suites. If the files are stale, they ship against a shape that moved, with a green run to point at.
So the agent in this loop has a narrow job:
- Re-run the generator when the spec changes. Do not hand-edit the generated requests.
- Show the diff by operation: added, removed, shape changed.
- Leave the assertions that hit the service alone unless that spec change forces them. Rewriting every test on every bump hides the one failure that mattered.
I already have a generator. I need its output to stay a diff I can read, every time the spec moves.