How do you know an AI agent's check can actually fail?
Run the check against something you know is broken. If it passes for the real result and for the broken one, it measures nothing. Every check an agent relies on should carry two inputs: one that must pass and one that must fail.
Agents report "verified" all the time. The question to ask is not whether the check passed. It is whether the check could have failed.
We run agent.ceo on AI agents, and this is one of the operating rules each of them loads before work:
Carry a known-negative. Compare the real output against something you
know should fail. If your check passes for both, it measures nothing.
Several endpoints here return 200 with an empty list for inputs that
do not exist.
Why checks pass in both worlds
| The check reads | Why it goes green on broken work |
|---|---|
HTTP status 200 | The endpoint returns 200 with an empty list for an ID that does not exist |
| A count of results | An error response parsed as an empty result reads exactly like zero |
| A shared health endpoint | It answers for the whole service, not for the page you changed |
A status field saying done | The field was written; the work behind it was not |
| The same result from five systems | A uniform result is almost always your tool, not the world |
Each of these is a real reading, taken correctly, that says nothing about the work.
Build the pair before you trust the check
For every check, write down the known-negative next to it. The pattern:
check() { # $1 = slug; passes only if the page is served
test "$(curl -s -o /dev/null -w '%{http_code}' "https://agent.ceo/blog/$1")" = 200
}
check the-fovea-loop && echo "real: PASS" || echo "real: FAIL"
check zz-fabricated-slug && echo "negative: PASS (check is blind)" \
|| echo "negative: FAIL (good, the check can fail)"
If the second line ever prints PASS, stop. Fix the check before you read its result on real work.
Rendering diagram…
Good known-negatives, by kind of check
| Kind of check | Known-negative |
|---|---|
| A page is live | A fabricated slug on the same host must be 404 |
| An API key works | An invented key on the same route must be refused |
| A revoke worked | The same call with the revoked key must be refused |
| A loop stopped | One full cadence after stop, there is no new output |
| Access is scoped | The same caller on another organization must be refused, not handed an empty list |
| A filter works | A value you know is present must appear (a known-positive) |
The last row matters too. A filter that returns nothing for every input also "fails the negative". Pair it with a value you know is there.
Make the pair re-runnable by someone else
A check that only its author can run is not evidence. Keep the inputs in the check itself, point it at the served surface rather than a local file, and let a second party run both halves. That is what turns "the agent says it works" into something you can verify. Why the author should not be the one to certify it is covered in should an AI agent verify its own work?
Three questions before you accept a green result
- What input would make this check fail, and did anyone run it?
- Does the check read the exact surface a user sees, or one next to it?
- Could a different person run it from a different machine and get the same verdict?
If any answer is "no" or "not sure", the green result is not yet information.
FAQ
How do you know an AI agent's check can actually fail?
Run it against a known-negative: an input you know is wrong, such as a fabricated ID or a revoked key. If the check passes for the real input and for the known-negative, it measures nothing. Keep both inputs next to the check so anyone can re-run the pair.
What is a known-negative test?
A known-negative is an input that must make a check fail. A fabricated slug must return 404, an invented API key must be refused, and a stopped loop must produce no new output. It proves the check discriminates between the world where the work is done and the world where it is not.
Why do agent checks pass when the work is broken?
Because many surfaces answer success for inputs that do not exist. An endpoint can return 200 with an empty list, and an error parsed as an empty result reads exactly like zero. A check that only looks at the status code, or only counts results, goes green in both cases.
Related: The FOVEA loop · Verification as code · Should an AI agent verify its own work?