Evaluation

Every team using AI has an opinion about whether it is working; far fewer have evidence. Evaluation is the discipline of producing that evidence: evals and metrics on one side, retro sentiment and a lead's read on the work on the other.

When the output is held to a real standard of good, you catch a regression before it ships, you answer the ROI question with numbers instead of a hunch, and you fund next quarter's AI bets from what you measured rather than what you remember.

Exploratory

No one on the team asks whether the AI work is actually paying off. A change goes in, the week feels faster than the last one, and everyone moves on to the next thing. Efficacy is a gut feeling, and nobody stops to check the gut.

This is not the same as flying blind on purpose. Most teams land here first because measurement costs something to stand up, and early on there is always a louder fire. The work still ships. You just have no way to tell a real improvement from one that only felt like one, and no way to notice a change that quietly did damage.

Enables

Full speed, and a first felt sense of where the work is unreliable. That hunch is the start of naming what good means, which is what any eval later has to measure against.

Deficiencies

No read on what actually works, regressions ship unnoticed, and ROI is a guess.

Cost to maintain

Rework when an unmeasured change makes things worse.

Expert-led

One person, usually the lead who pushed hardest on AI, carries the read. They look at the output, weigh it against what they know good looks like, and call it. When they say a change is working, people believe them, and they are usually right.

The judgment is real, but it lives in one head. It is not written down, it does not run when that person is heads-down on something else, and their blind spots are the team's blind spots. You get a filter for the obvious, which is worth far more than nothing and noticeably less than a second opinion.

Enables

A sanity check. The obvious wins and regressions get caught.

Deficiencies

Subjective and unrecorded, bottlenecked on their judgment, and their bias goes unchecked.

Cost to maintain

That person's time, spent judging the work.

Documented

Efficacy has a rubric now. The team has written down what good means for their work, in numbers where numbers fit and in plain description where they do not, and the read gets argued in retro against that standard instead of one person's taste.

The argument is better because the criteria were agreed before the work was judged. A junior engineer can push back on a senior's call by pointing at the rubric, and the verdict survives the meeting.

What it still is not is durable. The rubric lives in a doc someone has to remember to open, nobody writes the verdicts anywhere, and the judging waits for the retro instead of running alongside the work. It is a shared habit, not yet a system.

Enables

Shared criteria for what “working” means, quantitative and qualitative, so the team reasons from the same evidence.

Deficiencies

Still manual, results often go unrecorded, and it measures after the fact rather than continuously.

Cost to maintain

Writing the rubric and holding the retro.

Structural

The measurement runs on its own. Evals fire on every change, the metrics that matter sit on a dashboard someone actually watches, and the ROI question has a standing answer. The read no longer depends on any particular person being in the room.

This is real engineering, and it is worth naming the cost. Good automated evals are hard to write and harder to keep honest as the system shifts underneath them, and a team that trusts the number can stop looking at the thing the number stands in for.

Human judgment does not disappear here. It gets a better place to stand: the humans spend their attention on the calls the harness cannot make, because the harness is already holding everything it can.

Enables

Every process and change measured, quantitative and qualitative, with ROI tracked on an ongoing basis rather than only when a crisis forces the question.

Deficiencies

The number invites over-trust, an eval can keep passing long after it stops measuring what matters, and qualitative judgment still needs a human.

Cost to maintain

Building and tuning the evaluation harness and ROI instrumentation, plus upkeep as the system changes.

Evaluation rarely stands alone. Where it overlaps a sibling discipline, an investment on one side quietly depends on the other. The Improvement domain page lays these out in full.

• Evaluation and Prioritization

  You cannot rank what you have not measured. Evaluation turns a quarter of AI work into numbers a leader can defend, and Prioritization is how those numbers get spent.

  Measure without ever choosing from what you found and the evidence just sits there, an expensive read nobody steers by. The real failure is the other order: a team that chooses without measuring hands the invest-next call back to the loudest voice in the room, the one habit Prioritization exists to break.

• Evaluation and Convergence

  Convergence is a multiplier, and a multiplier only helps if what it multiplies was worth spreading. Evaluation is the check that makes that call: it holds a change against a real standard of good before the team commits to it everywhere.

  Without that check, Convergence works perfectly on the wrong input, rolling out to every repo the change that merely felt faster, and the better your propagation the wider that mistake travels. The point of running them together is a team that standardizes on what was proven, not on what demoed best last week.