# Working with Robots > A framework for engineering leaders deciding where to invest in working with AI. > Source of truth: https://gobetweenlab.com/working-with-robots ## How to use this This is a map, not a scorecard. Four **domains** are the conditions that must hold for AI collaboration to work. Each domain has three **disciplines**, the units of practice a leader funds, and **techniques** are the concrete moves. Cutting across all of them are four **investment tiers**: Exploratory → Expert-led → Documented → Structural. No tier is a failure state: each names both what it buys and what it costs, and the right tier is the one that fits the org, not the deepest one. Your job when you reason with this framework is to **diagnose where an org sits, discipline by discipline, and name the trade-off of moving.** Velocity isn't value: choose where you invest and say what it costs. You have succeeded when the leader can locate their own team. For example: “we're Structural on Collection but Expert-led on Management, and nothing in Observability is past Exploratory, so the next investment is lifting Management to Documented.” So when you are asked about an org's AI practice: locate the relevant discipline, read its four tiers, place the org, then name the specific trade-off of the next tier up. Diagnose, do not rank. ## Contents - The five principles - Domain: Context - Domain: Control - Domain: Observability - Domain: Improvement - Context › Collection - Context › Management - Context › Disclosure - Control › Capability - Control › Delegation - Control › Environment - Observability › Capture - Observability › Availability - Observability › Trace - Improvement › Evaluation - Improvement › Prioritization - Improvement › Convergence - The technique catalog - Domain overlaps - Field guide: Building an evaluation practice - Field guide: Building a harness ## The five principles The posture behind the framework: the why beneath the techniques. ### 1. Humans make decisions _Robots contribute. Humans decide._ Sometimes the robot executes, sometimes it thinks alongside you, but humans hold judgment where it matters: approving plans, reviewing output, deciding what ships. **In the framework:** Control > Delegation is this principle made a discipline: which calls an agent makes alone, and which wait for a person. Evaluation puts the standard of good in a person's hands, and the Escalation Affordance technique builds the handoff into the tooling. ### 2. Diverge then converge _Close the loop before moving on._ Agents follow patterns imperfectly and accumulate entropy. After a round of divergent work, realign to your patterns; skip convergence and the mess compounds. **In the framework:** Improvement > Convergence is the converge half at team scale: the discipline of folding what one engineer proved out back into shared practice. Context > Management fights the same entropy in what agents read, keeping it current as conventions change. ### 3. Fluency isn't value _The sensation of work is not the work._ AI manufactures the sensation of progress, convincingly enough to stop you checking whether the output is useful. Measure against a standard the system can't generate. **In the framework:** The discipline built on it is Improvement > Evaluation, where efficacy is measured against a written standard instead of by how the week felt. Verification Scripts and the Evaluation Harness turn that standard into running checks. ### 4. Input is the lever _You don't own the output. You own everything around it._ What you control is the intent and information you feed in, the tools the agent can reach, and the checks its work has to clear, all set before the run starts. Build those into the system and they pay off on every run; fix each output by hand and you are only doing cleanup. **In the framework:** Its three levers map onto the framework directly: the information you feed in is Context, what the agent can reach is Control > Capability, and the checks its work must clear are Control > Environment. The investment tiers measure the same move: from fixing each output by hand to building the fix into the system, where it pays off on every run. ### 5. Map, not a scorecard _A map tells you where you are, not how you rank._ The four domains and twelve disciplines are a useful fiction: the real work overlaps, and investing on both sides of an overlap beats either alone. The tiers are real but not grades: each costs more than the last to run and to change, so the right tier is the one that fits your org, not the deepest one. And the map is of your terrain, not an ideal to conform to, so the right shape is something you discover by working up from the bottom, not a phase you skip to the top. **In the framework:** The map itself practices it. Every domain page's Domain Overlap section treats both sides of an overlap as one investment, and every discipline page opens on “the right tier, not the top.” The Prioritization Rubric applies the same stance to spending. ## Context _The agent has what it needs to act._ Context is made of three disciplines: Collection, Management, and Disclosure. Collection gathers what the agent needs to orient, like pulling the ticket, the related code, and the last design discussion into reach before work starts. Management keeps that material true over time, so the agent is not following a convention your team dropped two quarters ago. Disclosure feeds it in efficiently, giving the payments task the payments runbook instead of the whole wiki. **The business outcome.** Faster delivery with less rework. When agents start from the right information, work ships sooner and comes back for correction less often, and your AI spend stops growing with every re-explained session. **Before investment.** Every session starts from zero. Engineers paste in whatever background they remember, results depend on who wrote the prompt, and agents build confidently on facts that stopped being true a sprint ago. Senior people spend their afternoons repeating themselves and correcting what comes back. **After investment.** The background is simply there. An agent picks up a task the way a well-onboarded hire would, already knowing your conventions and history, and the questions it asks are about the genuinely new parts of the problem. Quality stops depending on who typed the prompt. ### The three disciplines #### Collection > When a new agent picks up a task, does it have the transcripts, tickets, and code it needs, or is someone hand-feeding it every time? Getting context in: gathering the artifacts an agent needs to orient. Full depth: the Collection section below. #### Management > Is the context an agent reads current, or three sprints stale? Keeping the context your agents read current as decisions and conventions change. Full depth: the Management section below. #### Disclosure > Do we feed it the right slice at the right moment, or dump everything and hope? Feeding context in efficiently: routing the agent to the slice it needs, not a data dump. Full depth: the Disclosure section below. ## Control _The agent can only do what it should._ Control is made of three disciplines: Capability, Delegation, and Environment. Capability sets what the agent can do at all: the tools you connect and the ones you withhold, like a database it can read but never write. Delegation sets what it decides alone versus what waits for a person, like merging a dependency bump on its own but never running a production migration. Environment is the machinery that catches mistakes, the tests and checks a change must pass before it ships. **The business outcome.** Moving fast without betting the company on an agent's judgment. A bounded agent can be trusted with more work, and several can run at once, because a mistake stays small and every action has an answer to who allowed it. **Before investment.** Safety depends on each engineer's personal caution. Agents hold credentials nobody remembers granting, a confident mistake can reach production before a person sees it, and after an incident nobody can say what the agent was allowed to do. So everything gets re-reviewed by hand, and the speed you were buying evaporates. **After investment.** The boundaries are set once and hold for everyone. Agents get the access the work needs and nothing more, high-stakes calls are reserved for people by design, and most mistakes die in a failing check instead of a postmortem. That is what makes it safe to hand agents bigger work. ### The three disciplines #### Capability > Can our agents reach the tools the work needs, and are they walled off from the ones that could do real damage? What the agent can do: the tools you connect, and the ones you deliberately withhold. Full depth: the Capability section below. #### Delegation > Which decisions do we let an agent make on its own, and which still wait for a human? What the agent should do: the responsibilities you hand over, and the ones you reserve. Full depth: the Delegation section below. #### Environment > Does the system around the agent catch its mistakes before they land, or do we find out in review? The system the agent works within: the tests and gates its work must clear, and the points where a human takes over. Full depth: the Environment section below. ## Observability _The human can see what happened._ Observability is made of three disciplines: Capture, Availability, and Trace. Capture records each run as it happens: everything the agent was told, everything it did, and what it cost. Availability puts that record where your team can query it, not locked in a vendor console nobody opens. Trace reconstructs a single run end to end, like walking back through an agent's whole bug fix, reasoning included. **The business outcome.** Trust backed by evidence. You can reconstruct an incident, answer an auditor from the record instead of from memory, and see exactly where the AI spend goes. **Before investment.** Agent work is a black box you pay for monthly. When output is wrong, nobody can say where the run went sideways, so debugging is guesswork and re-runs. The bill is one number with no breakdown, and when someone asks what the agent did, the honest answer is that you do not know. **After investment.** A record exists for every run, and someone can open it. An incident becomes an afternoon reading a trace instead of a week of speculation, spend maps to teams and workflows, and questions about agent behavior are answered from the log. ### The three disciplines #### Capture > Are we recording what our agents do: the prompts, the responses, the tool calls, the cost? Making sure every run leaves a raw record. Full depth: the Capture section below. #### Availability > When we want to look, is the data actually queryable, or buried where no one can reach it? Putting the record of agent work where your team can query and review it. Full depth: the Availability section below. #### Trace > When something goes wrong, can we reconstruct exactly what the agent did and why? Walking back through a single run, end to end, with the reasoning alongside the decisions. Full depth: the Trace section below. ## Improvement _The team gets better over time._ Improvement is made of three disciplines: Evaluation, Prioritization, and Convergence. Evaluation measures whether a practice works, like comparing rework rates before and after a new review workflow. Prioritization decides what to change next, and what to leave alone. Convergence spreads a win, so the prompt one person perfected becomes the team's starting point. **The business outcome.** An AI investment that compounds instead of plateauing. You know what the spend returns, the next dollar goes where the evidence points, and gains stay with the team through turnover. **Before investment.** Progress is anecdotal. Adoption is up and the demos look good, but nobody can say what any of it is worth. The best workflow in the company lives in one engineer's setup and leaves when they do, while the tools move quarterly and team practice quietly falls behind. **After investment.** Improvement runs as a loop. Results get measured, the next change is chosen from what they show, and each win becomes team practice, so capability rises every quarter instead of resetting whenever someone leaves. ### The three disciplines #### Evaluation > Do we actually know whether our AI investments are paying off, or is it a feeling? Measuring whether your processes work, by the numbers and by gut-check. Full depth: the Evaluation section below. #### Prioritization > When we decide what to improve next, are we reading real signal or chasing the loudest voice? Choosing what to change and where to invest next. Full depth: the Prioritization section below. #### Convergence > When something works for one engineer, does it reach the whole team, or die in their private setup? Making a change stick across the team. Full depth: the Convergence section below. ## Collection _Context · When a new agent picks up a task, does it have the transcripts, tickets, and code it needs, or is someone hand-feeding it every time?_ An agent can only act on what it can see. Collection is the discipline of putting the right material in front of it: the meeting transcripts where a decision actually got made, the code and its git history, the wiki or knowledge base, the epics and tickets that say what the work is for. Give an agent all of this from day one and it orients itself. Your team stops re-explaining and stops chasing down what the agent missed, and the work starts from what is true rather than what the agent guessed: fewer wrong turns, less rework, decisions that land right the first time. ### The right tier, not the top You do not have to automate Collection to do it well. A small team where one careful engineer assembles context by hand may be exactly where it should be, and building connectors before you know which artifacts are worth collecting just makes machinery you will tear out later. There is a second reason not to jump straight to the top. You cannot know in advance what shape a deep tier should take: which artifacts a Structural setup should pull in is specific to your team, and doing the gathering by hand first is the cheapest way to learn it. Wire up the connectors before you know, and you have baked a guess into software that is expensive to change, not just to maintain. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is an honest look at the tradeoff. Each tier up buys less manual work and more consistency, and each costs more to build and maintain. Read down the four, find where your team actually sits, then decide whether the next tier is worth buying. ### The four investment tiers #### Exploratory People grab whatever artifacts they think the agent needs, with no shared standard. It is where every team starts. - **Enables:** The cheapest look at what your agents actually need. Every session someone hand-feeds shows which artifacts an agent reached for and which it ignored, the knowledge a durable setup gets built from. - **Deficiencies:** Critical artifacts get missed and stale ones get pulled in, and no one can say what is actually available. - **Cost to maintain:** The quiet tax of re-gathering the same context, session after session. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One engineer knows which artifacts matter and curates them for everyone else. - **Enables:** Consistent gathering and a foundation the team can rely on. - **Deficiencies:** A single point of failure. Their judgment is not written down, and nothing works well while they are out. - **Cost to maintain:** That person's time, spent curating and teaching. - **Techniques:** - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. - **Orientation Docs.** Auto-loaded files (CLAUDE.md, README) the agent reads first, so it opens every session already knowing your stack and conventions. #### Documented The expert's list is written down. The team knows what to gather and where to find it. - **Enables:** Consistency across projects, new hires who follow the playbook, and a practice that survives the expert leaving. - **Deficiencies:** Still manual. It depends on people following it, and gaps open whenever the artifacts change. - **Cost to maintain:** Writing the playbook once, then the ongoing work of keeping it current. - **Techniques:** - **Orientation Docs.** Auto-loaded files (CLAUDE.md, README) the agent reads first, so it opens every session already knowing your stack and conventions. - **Domain Knowledge.** One file per service or entity, each with the same shape (purpose, boundaries, key paths), so the structure itself is the navigation. #### Structural Artifacts are pulled in automatically. The agent always opens with what it needs. - **Enables:** No manual gathering, consistency across every repo, and a corpus that grows the moment something new appears. - **Deficiencies:** Bad rules pull in noise or stale data, and the connectors are real software that has to be built and owned. - **Cost to maintain:** Building the connectors and capture hooks, then maintaining them as APIs, tools, and repos change. - **Techniques:** - **Source Connectors.** Live integrations that pull artifacts straight from the systems of record: the repo, the PM tool, the wiki, the transcript store. - **Automatic Capture.** Hooks that record new artifacts as they land (a merged PR, a closed ticket, a fresh transcript) so context stays complete without anyone maintaining it. ## Management _Context · Is the context an agent reads current, or three sprints stale?_ What your agent collected on day one quietly goes out of date. Conventions get revised, specs get superseded, architectural decisions get reversed, and the documents that record all of it drift the moment they stop being anyone's job. Management is the discipline of catching that drift: the CLAUDE.md that still names last quarter's stack, the decision the team walked back but never wrote down, the spec that says one thing while the code does another. When what the agent reads matches what is true now, it stops heading down paths the team has already abandoned. Your team stops paying for stale context in rework, and the agent's decisions land right because they rest on the system as it is, not as it was. ### The right tier, not the top Keeping context current is not a thing you buy once. The deepest investment wires freshness into the commit itself, but a smaller team whose one engineer keeps the handful of core docs honest may be spending exactly the right amount. Automating currency across docs that rarely change is machinery that earns nothing. There is a second reason not to jump straight to the top. You cannot know in advance which docs a freshness system should watch: that depends on which parts of your context actually rot fast enough to matter, and keeping a handful current by hand is the cheapest way to find out. Wire currency into the build before you know, and you have baked a guess into machinery that is expensive to change, not just to maintain. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is a clear read on that tradeoff. Each tier up catches drift sooner and leans less on someone remembering, and each costs more to build and tune. Read down the four, find where your team sits, then decide whether the next tier is worth the freshness it buys. ### The four investment tiers #### Exploratory Docs exist, but no one owns keeping them current, so they rot. - **Enables:** A first read on which docs actually go stale fast enough to matter. Watching context rot in real time tells you where freshness is worth paying for and where it never will be. - **Deficiencies:** Stale artifacts steer the agent wrong, outdated conventions get reapplied, and trust in the docs erodes until people stop reading them. - **Cost to maintain:** Wasted agent effort on dead ends and rework, paid quietly and continuously. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One engineer stays on top of the key docs and fixes drift whenever they happen to notice it. - **Enables:** The core artifacts stay reasonably current, and the agent can trust the parts that person watches. - **Deficiencies:** Only their corner stays fresh while the rest rots, and what changed lives in their head rather than on the record. - **Cost to maintain:** That person's attention, a background tax that scales with the number of docs. - **Techniques:** - **Orientation Docs.** Auto-loaded files (CLAUDE.md, README) the agent reads first, so it opens every session already knowing your stack and conventions. - **Prompt Templates.** Template files with placeholders for runtime context, so the same workflow encodes the current state and adapts to a new issue without a rewrite. #### Documented A written policy says what gets updated and when, with a clear owner for each area. - **Enables:** Systematic currency on a schedule the team can see, no longer dependent on one person happening to notice. - **Deficiencies:** Still manual and reliant on follow-through, so gaps open between a planned update and the change that already shipped. - **Cost to maintain:** Writing the policy once, then the ongoing work of the updates themselves. - **Techniques:** - **Orientation Docs.** Auto-loaded files (CLAUDE.md, README) stamped with refresh dates, so anyone can see when a doc was last held against the code. - **Index Files.** When-to-read tables maintained as part of sprint work, so the routing layer refreshes on the same cadence as the code it points at. - **Domain Knowledge.** One file per service or entity, each with a named owner, so every area of the corpus has someone accountable for its freshness. #### Structural Docs update themselves as part of the work, and currency is enforced rather than remembered. - **Enables:** Near-zero manual maintenance, docs that stay in sync with the code automatically, and drift caught as soon as it appears. - **Deficiencies:** It needs tight integration, mis-tuned rules raise false positives, and an over-strict check can slow the work it guards. - **Cost to maintain:** Upfront engineering to hook docs into the build and tooling, plus ongoing tuning as the rules meet real changes. - **Techniques:** - **Doc Currency Hooks.** Hooks in the commit and CI path that update or validate docs against the code as the work lands, so orientation files and indexes cannot silently drift. - **Verification Scripts.** Standardized host scripts (lint, test, e2e) the agent runs and reads the exit code from, so checks fire before a human sees the output. - **Config-Driven Deployment.** Config plus install scripts that propagate a skill or doc from one place to every project, so an update in one repo lands in all of them. ## Disclosure _Context · Do we feed it the right slice at the right moment, or dump everything and hope?_ Your agent can have everything it needs gathered and current and still drown in it. Disclosure is the discipline of matching what you serve to the task at hand: a focused set of requirements rather than every doc in the repo, an index that routes the agent to the one file it needs rather than pasting all of them in, an orientation note that points, not dumps. It covers picking up where the last session stopped, too: a fresh session starts empty, and re-priming it with the state of the work lets the agent resume instead of starting over. When a session carries only what the task calls for, the agent reaches an answer faster, and you spend less per run because the context window stays lean: fewer tokens, fewer wrong turns, decisions that land the first time. ### The right tier, not the top You do not need automated routing to serve context well. A team whose one engineer has a feel for what to load, and loads it, may be exactly where it should be, and building an auto-routing layer before you know which slices a task actually needs just adds a system to tune. There is a second reason not to jump straight to the top. You cannot know in advance which slices a routing layer should serve: that is specific to your tasks, and loading context by hand is the cheapest way to learn which ones a job actually needs. Stand up auto-routing before you know, and you have baked a guess into a system you now have to tune, one expensive to change and not just to maintain. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is an honest look at the tradeoff. Each tier up makes lean sessions the standard rather than one person's habit, and each costs more to build and keep on target. Read down the four, find where your team sits, then decide whether the next tier is worth buying. ### The four investment tiers #### Exploratory People paste in whatever they have, or start the agent cold with almost nothing. - **Enables:** Zero setup, and a fast read on what a task actually needs in front of it. Overstuffing and starving sessions by hand is how you learn which slices matter, before you build anything to route them. - **Deficiencies:** The agent drowns in an overstuffed window or guesses from a starved one, and either way it misreads the task. Every new session starts cold, re-deriving what the last one already knew. - **Cost to maintain:** Tokens burned on irrelevant context, plus the rework when the agent acts on what it skimmed past. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One engineer is good at feeding the agent just enough in the right order, and the knack lives with them. - **Enables:** The sessions that engineer touches run lean and well-aimed. - **Deficiencies:** A single point of failure whose judgment is unwritten, while everyone else still dumps or starves. - **Cost to maintain:** Their time hand-curating each session's context and correcting the sessions that went wide. - **Techniques:** - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. - **Index Files.** Lightweight when-to-read tables that route the agent from a task to the one file it needs, instead of loading everything. - **Session Prompts.** A standalone protocol document for picking up work in a fresh session: read the state, take the next step, repeat in a fixed loop. #### Documented “Link the index, don't paste the file” is written down, and the team knows to route rather than dump. - **Enables:** Consistent disclosure across the team, a habit new hires inherit, and a practice that survives the expert leaving. - **Deficiencies:** Still applied unevenly and dependent on people following it, and an out-of-date index sends the agent to the wrong file. - **Cost to maintain:** Writing the routing layer and the convention, plus keeping the indexes pointed at the right files. - **Techniques:** - **Index Files.** Lightweight when-to-read tables that route the agent from a task to the one file it needs, instead of loading everything. - **Session Prompts.** A standalone protocol document for picking up work in a fresh session: read the state, take the next step, repeat in a fixed loop. - **Progress Trackers.** A JSON state file that is decomposition, controller, and handoff at once, so a new session knows what is done, what is next, and what the last one learned. #### Structural Progressive disclosure is built into the system: the agent loads orientation first and pulls deeper context only when a task calls for it. - **Enables:** Efficient context by default rather than by talent, the same disclosure path everywhere, and windows that stay small without losing reach. - **Deficiencies:** A mis-tuned index or auto-load can route the agent wrong or withhold what it needs, and it takes real engineering to stand up. - **Cost to maintain:** Building and tuning the indexes and auto-loaded files, plus propagating them as the repo changes. - **Techniques:** - **Index Files.** Lightweight when-to-read tables that route the agent from a task to the one file it needs, instead of loading everything. - **Session Prompts.** A standalone protocol document for picking up work in a fresh session: read the state, take the next step, repeat in a fixed loop. - **Config-Driven Deployment.** Config plus install scripts that propagate a skill or doc from one place to every project, so an update in one repo lands in all of them. ## Capability _Control · Can our agents reach the tools the work needs, and are they walled off from the ones that could do real damage?_ An agent is only as useful as what you let it touch, and only as safe as what you keep out of its reach. Capability is the discipline of connecting your agent to the systems the task calls for: the test runner that proves its work, the PR tooling that carries a change toward review, the staging environment where it can try things without consequence. The other half of the discipline is deliberate withholding: production, the credentials that open it, and the destructive commands are kept where no prompt can reach them. Get that surface right and your team moves fast on the work that matters, while the actions that could wreck something are simply not reachable: fewer incidents from a confident agent reaching too far, and a much smaller mess to clean up when something does slip. What Capability owns is the reach itself, the call about whether a tool or command is connected at all and whether a dangerous one is taken back off the table. The judgment of which actions an agent may take on its own is Delegation, and the harness of checks it has to clear is Environment. The question it leaves you with is how much of your safe surface lives in one careful engineer's local setup, and how much is written down where every agent inherits it by default. ### The right tier, not the top You do not have to build your allow-list into the repo to hold the line well. A small team whose one cautious engineer keeps production and the destructive commands locked away may be exactly where it should be, and hardening a surface into version control before you know which tools the work leans on just builds controls you will loosen later. There is a second reason not to jump straight to the top. You cannot know in advance which tools and commands a checked-in allow-list should scope: that depends on what your work actually leans on, and holding the line by hand is the cheapest way to learn it. Harden the surface into version control before you know, and you have baked a guess into controls that are expensive to change, not just to maintain. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is an honest look at the tradeoff. Each tier up makes the safe surface something every agent starts from rather than one person's habit, and each costs more to build and keep tuned. Read down the four, find where your team actually sits, then decide whether the next tier is worth buying. ### The four investment tiers #### Exploratory The agent has whatever access the tool shipped with, and no one chose it. - **Enables:** A fast start, and a live read on which tools the work actually leans on. What you reach for and what you never touch is what a scoped surface later gets built around. - **Deficiencies:** Nothing scopes the surface, so a destructive command or a wrong-account push is one prompt away. - **Cost to maintain:** Incidents and cleanup when the agent does something it should never have been able to do. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One security-minded engineer hardens their own setup, and only theirs. - **Enables:** A safe surface for the people who copy that engineer's config. - **Deficiencies:** A single point of failure. Their posture is not written down, and everyone else still runs the defaults. - **Cost to maintain:** That engineer's time tuning the allow-list and fielding “why can't the agent do X” questions. - **Techniques:** - **Permission Allow-Lists.** A settings file that enumerates exactly which commands an agent may run, started from zero and widened only as the work needs it, so its reach is a reviewed decision rather than a default. - **Tool Wrappers.** Scripts that wrap standard CLI tools with guardrails (credential routing, identity checks, confirmation on destructive operations) so the safety lives in the tool rather than in the prompt. #### Documented “Don't give it prod” is a written rule the team follows by convention and review. - **Enables:** A shared posture new people inherit from the doc rather than from one person's laptop. - **Deficiencies:** Still enforced by humans. A missed review or an ignored line and the rule does nothing. - **Cost to maintain:** Writing the policy, plus the human attention to catch breaches before they ship. - **Techniques:** - **Contracts and Prohibitions.** Explicit do-not lines written into skills and orientation docs, ruling out whole classes of action without asking the agent to weigh the alternatives. - **Permission Allow-Lists.** A settings file that enumerates exactly which commands an agent may run, started from zero and widened only as the work needs it, so its reach is a reviewed decision rather than a default. - **Tool Wrappers.** Scripts that wrap standard CLI tools with guardrails (credential routing, identity checks, confirmation on destructive operations) so the safety lives in the tool rather than in the prompt. #### Structural Allow-lists and scoped connections are checked into the repo. The agent cannot exceed the posture without a tracked change. - **Enables:** The posture survives turnover and applies to every agent automatically, and breaches need a reviewed commit. - **Deficiencies:** An over-tight list blocks real work, loosening it has friction, and the rules need upkeep as tools change. - **Cost to maintain:** Upfront engineering to build the controls in, plus tuning the surface as the work evolves. - **Techniques:** - **Permission Allow-Lists.** A settings file that enumerates exactly which commands an agent may run, started from zero and widened only as the work needs it, so its reach is a reviewed decision rather than a default. - **Tool Wrappers.** Scripts that wrap standard CLI tools with guardrails (credential routing, identity checks, confirmation on destructive operations) so the safety lives in the tool rather than in the prompt. - **Config-Driven Deployment.** Config plus install scripts that propagate a skill or doc from one place to every project, so an update in one repo lands in all of them. - **Verification Scripts.** Standardized host scripts (lint, test, e2e) the agent runs and reads the exit code from, so checks fire before a human sees the output. ## Delegation _Control · Which decisions do we let an agent make on its own, and which still wait for a human?_ Control buys your team speed inside safe boundaries, and Delegation is where you draw the line that makes the speed safe: what an agent handles without asking, what it brings to a person first, and how much work you hand it at once. You let the agent draft freely, explore an approach, and propose the change, then have it pause at the moves that are expensive to walk back: before it merges a PR, ships to production, or touches a database. When that line is deliberate, you get the throughput of an agent that rarely blocks and the calm of knowing the riskiest actions still pass a human gate, so a confident wrong call costs a review comment instead of an incident. What Delegation owns is the judgment about who decides, not the mechanism that enforces it: the allow-list and the destructive-command gate are Capability, and breaking work into session-sized units the system around it can checkpoint is Environment. The line itself runs from soft rules a human can waive in a low-risk case to hard policy (“never delete a database, any agent, any environment”) that nothing is allowed to cross. The question it leaves you with is how much of that line lives in one person's head on a given run, and how much the tooling itself enforces, so it holds when no one is watching. ### The right tier, not the top You do not have to enforce the line in the harness to draw it well. A team whose one lead reserves the risky moves and signs off on plans may be spending exactly the right amount, and building stop-points into every skill before you know which calls need a gate just slows the safe work down. There is a second reason not to jump straight to the top. You cannot know in advance which calls a hard gate should stop: that depends on where your agents' judgment actually needs a human, and reserving the risky moves by hand is the cheapest way to learn it. Build stop-points into every skill before you know, and you have baked a guess into a workflow that is expensive to change, not just to maintain. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is an honest read on that tradeoff. Each tier up leans less on someone remembering the line and more on the system holding it, and each costs more to build and tune. Read down the four, find where your team sits, then decide whether the next tier is worth buying. ### The four investment tiers #### Exploratory No line between agent and human work. Whatever the agent does, it does. - **Enables:** Full speed, and a cheap way to learn where the agent's judgment actually needs a human. The calls that go fine and the ones that scare you are what a real gate later gets drawn from. - **Deficiencies:** High-stakes actions get taken with no human at the points that matter, one bad run reaches production, and oversized tasks degrade or stall mid-way. - **Cost to maintain:** Cleanup after the agent oversteps, plus the latent risk of the run that does real damage. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One lead applies their own rules to their own work: never merge, always review the plan first, and break work into a session-sized chunk before handing it over. - **Enables:** A sane line for the work that lead touches, with the riskiest moves kept with a human. - **Deficiencies:** A single point of failure. The rule lives in their head, not the workflow, and others delegate differently or not at all. - **Cost to maintain:** Their attention holding the line on every run, plus re-teaching it whenever someone new picks up the agent. - **Techniques:** - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. - **Contracts and Prohibitions.** Explicit do-not lines written into skills and orientation docs, ruling out whole classes of action without asking the agent to weigh the alternatives. - **Progress Trackers.** A JSON state file that is decomposition, controller, and handoff at once, so a new session knows what is done, what is next, and what the last one learned. #### Documented “Agents draft, humans approve” is written into team norms, with the high-stakes actions named and a convention for how big a task to hand an agent at once. - **Enables:** A shared line across the team. New people inherit it, and it survives the lead being out. - **Deficiencies:** It still relies on people honoring it, nothing in the workflow stops an agent from crossing the line, and gaps open where a norm is silent. - **Cost to maintain:** Writing the policy and keeping the list of reserved actions current as stakes change. - **Techniques:** - **Contracts and Prohibitions.** Explicit do-not lines written into skills and orientation docs, ruling out whole classes of action without asking the agent to weigh the alternatives. - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. - **Progress Trackers.** A JSON state file that is decomposition, controller, and handoff at once, so a new session knows what is done, what is next, and what the last one learned. #### Structural Stop-points sit in skills, rules that take no exceptions live in contracts, and the workflow itself decomposes work into session-sized units, so the line survives whoever drew it. - **Enables:** The agent cannot take a reserved action on its own. The human gate is enforced rather than remembered, and it holds consistently across every run and project. - **Deficiencies:** A too-strict line blocks low-risk work and trains people to wave it through, and an agent that hits the line mid-task still has no good way to hand the problem back to a human. - **Cost to maintain:** Upfront engineering of stop-points and allow-lists, plus tuning the line as trust and stakes shift. - **Techniques:** - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. - **Progress Trackers.** A JSON state file that is decomposition, controller, and handoff at once, so a new session knows what is done, what is next, and what the last one learned. - **Contracts and Prohibitions.** Explicit do-not lines written into skills and orientation docs, ruling out whole classes of action without asking the agent to weigh the alternatives. - **Permission Allow-Lists.** A settings file that enumerates exactly which commands an agent may run, started from zero and widened only as the work needs it, so its reach is a reviewed decision rather than a default. - **Tool Wrappers.** Scripts that wrap standard CLI tools with guardrails (credential routing, identity checks, confirmation on destructive operations) so the safety lives in the tool rather than in the prompt. - **Escalation Affordance.** A built-in path for the agent to stop and ask a human when it is stuck, instead of looping or silently giving up, so asking for help is built into the harness rather than left to the agent's judgment. ## Environment _Control · Does the system around the agent catch its mistakes before they land, or do we find out in review?_ The work an agent produces is only as trustworthy as the harness it runs inside. Environment is that harness: the lint and test gates a change has to clear before it counts as done, the branch or worktree isolation that keeps two parallel runs from overwriting each other, the handoff points that force a checkpoint before work moves on. When those checks are built into the environment itself rather than left to whoever is paying attention, mistakes get caught where they happen instead of in a reviewer's lap. Your team stops finding out about a broken build or a colliding branch after the fact, and quality stops being something you pay for downstream: fewer surprises, less cleanup, parallel agents that stay out of each other's way. What Environment owns is everything around the work, not the work itself: the gates, the isolation, the enforced checkpoints. What an agent is allowed to reach is Capability, and which calls it makes on its own is Delegation. Environment is where the lines those two draw get backed by the system instead of taken on trust. The question it leaves you with is how much of this harness lives in one engineer's local setup, and how much is set up once and shared, so the same checks meet every run. ### The right tier, not the top You do not have to put the whole team inside one shared harness to shape the work well. A team whose one engineer has a homegrown setup that stops the worst mistakes may be exactly where it should be, and building an enforced environment before you know which gates the work needs just makes machinery that boxes people in. There is a second reason not to jump straight to the top. You cannot know in advance which gates a shared harness should enforce: that depends on where your runs actually go wrong, and shaping the work with your own scripts is the cheapest way to learn it. Build the enforced environment before you know, and you have baked a guess into machinery that is expensive to change, not just to maintain. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is an honest look at the tradeoff. Each tier up makes a shaped run the default rather than one person's setup, and each costs more to build, version, and coordinate. Read down the four, find where your team sits, then decide whether the next tier is worth buying. ### The four investment tiers #### Exploratory Engineers run the agent bare: no wrapper, no checks on what comes out. - **Enables:** A fast start, and a clear look at where the bare run actually goes wrong. The failures you catch by hand are the gates a shared harness later gets built to enforce. - **Deficiencies:** No guardrails, bad output ships unchecked, behavior varies turn to turn, and parallel runs collide on shared files and branches. - **Cost to maintain:** Humans catching every failure downstream. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One engineer has built their own scripts, wrappers, and skills into a working setup. - **Enables:** A shaped space for that person, with repeatable runs and checks that catch the obvious failures. - **Deficiencies:** It lives on their machine and is not portable, everyone else runs raw, and their config drifts undocumented. - **Cost to maintain:** Their time building and tending it. - **Techniques:** - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. - **Verification Scripts.** Standardized host scripts (lint, test, e2e) the agent runs and reads the exit code from, so checks fire before a human sees the output. - **Tool Wrappers.** Scripts that wrap standard CLI tools with guardrails (credential routing, identity checks, confirmation on destructive operations) so the safety lives in the tool rather than in the prompt. #### Documented The useful scripts and config are written up and shared, but each person assembles their own harness from them. - **Enables:** A common toolkit. New people can stand up a working environment from the docs. - **Deficiencies:** Still hand-assembled. Versions drift between people, and guardrails depend on who set them up. - **Cost to maintain:** Documenting the setup, plus everyone keeping their own copy current. - **Techniques:** - **Verification Scripts.** Standardized host scripts (lint, test, e2e) the agent runs and reads the exit code from, so checks fire before a human sees the output. - **Tool Wrappers.** Scripts that wrap standard CLI tools with guardrails (credential routing, identity checks, confirmation on destructive operations) so the safety lives in the tool rather than in the prompt. - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. - **Structured Comments.** Typed comments with parseable markers (plan, progress, fix, review, complete) posted as the work runs, so parallel work signals ownership and a downstream reader pulls what happened without reading everything. #### Structural One shared, versioned environment everyone operates inside, with validation and guardrails enforced and parallel agents isolated into their own worktrees with clear ownership. - **Enables:** Every run shaped the same way, gates that cannot be skipped, and a harness that survives turnover. - **Deficiencies:** Upfront engineering, rigidity if the harness is wrong, and changes that need coordination across everyone inside it. - **Cost to maintain:** Building and versioning the shared environment, plus tuning guardrails as the work changes. - **Techniques:** - **Config-Driven Deployment.** Config plus install scripts that propagate a skill or doc from one place to every project, so an update in one repo lands in all of them. - **Tool Wrappers.** Scripts that wrap standard CLI tools with guardrails (credential routing, identity checks, confirmation on destructive operations) so the safety lives in the tool rather than in the prompt. - **Verification Scripts.** Standardized host scripts (lint, test, e2e) the agent runs and reads the exit code from, so checks fire before a human sees the output. - **Structured Comments.** Typed comments with parseable markers (plan, progress, fix, review, complete) posted as the work runs, so parallel work signals ownership and a downstream reader pulls what happened without reading everything. - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. ## Capture _Observability · Are we recording what our agents do: the prompts, the responses, the tool calls, the cost?_ When an agent runs unsupervised, the only proof of what it did is whatever you thought to keep. Capture is the discipline of instrumenting the run so that proof exists: the prompts it received, the responses it gave, the tool calls it made, and the metadata around them, from cost and token spend to the model and platform behind each session. When every session lands in a durable record, a failure from weeks ago is still reconstructable and your AI spend is legible instead of a line item that surprises you on the bill. Without it, the hardest incidents are the ones you can never reopen, and the budget conversation runs on guesses. ### The right tier, not the top You do not have to capture everything on every run to do Capture well. A small team whose one engineer saves the sessions that matter may be spending exactly right, and standing up a full telemetry pipeline before you know which runs you will ever revisit is instrumentation you will pay to carry and rarely read. There is a second reason not to jump straight to the top. You cannot know in advance which runs a telemetry pipeline should record: that depends on what you actually end up needing to revisit, and saving the sessions that matter by hand is the cheapest way to learn it. Stand up the pipeline before you know, and you have baked a guess into instrumentation that is expensive to change, not just to carry. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is an honest look at the tradeoff. Each tier up leaves less of the record to chance and less to individual habit, and each costs more to build and maintain. Read down the four, find where your team actually sits, then decide whether the next tier is worth buying. ### The four investment tiers #### Exploratory Nothing is captured. When something breaks, you have the chat window, if that. It is where everyone starts. - **Enables:** You can still act, and the first unreconstructable incident tells you exactly which runs you needed a record of. That is what a capture pipeline later gets aimed at. - **Deficiencies:** No record of the prompts, calls, cost, or model, so failures are unreconstructable and there is nothing to learn from later. - **Cost to maintain:** Nothing, until an incident is paid for in lost context. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One engineer logs their own runs into a personal drawer. - **Enables:** That person can reconstruct their own sessions and spot their own patterns. - **Deficiencies:** It covers one person's work in their own format, and nothing is captured when anyone else runs. - **Cost to maintain:** That person's diligence to keep logging by hand. - **Techniques:** - **Structured Comments.** Typed comments with parseable markers (plan, progress, fix, review, complete) posted as the work runs, so parallel work signals ownership and a downstream reader pulls what happened without reading everything. - **Progress Trackers.** A JSON state file that is decomposition, controller, and handoff at once, so a new session knows what is done, what is next, and what the last one learned. #### Documented A few people keep records to a shared standard of what to capture and where. - **Enables:** Comparable records across people and projects: a team trail, not a private one. - **Deficiencies:** Still opt-in, so gaps open whenever someone forgets, and metadata like cost and model gets captured unevenly. - **Cost to maintain:** Writing the standard once, then the ongoing manual logging. - **Techniques:** - **Structured Comments.** Typed comments with parseable markers (plan, progress, fix, review, complete) posted as the work runs, so parallel work signals ownership and a downstream reader pulls what happened without reading everything. - **Progress Trackers.** A JSON state file that is decomposition, controller, and handoff at once, so a new session knows what is done, what is next, and what the last one learned. - **Database Audit Trail.** Append-only, queryable logging at multiple granularities (tool-call events, orchestrator events, session summaries), so the record can be watched live and interrogated long after the run, self-serve dashboards included. #### Structural Every session's prompts, calls, cost, and model land in a store as a side effect of the work. - **Enables:** Complete capture with zero per-session effort, and nothing that depends on someone remembering. - **Deficiencies:** It captures volume indiscriminately, and the record is only as useful as its sibling discipline, Availability, makes it. - **Cost to maintain:** Upfront engineering on the logging pipeline, plus its upkeep as tools and platforms change. - **Techniques:** - **Database Audit Trail.** Append-only, queryable logging at multiple granularities (tool-call events, orchestrator events, session summaries), so the record can be watched live and interrogated long after the run, self-serve dashboards included. - **Session Telemetry Hooks.** Hooks that record each session's operational metadata (cost, token spend, model, and platform) automatically as it runs, so the record stays complete without anyone choosing to log. ## Availability _Observability · When we want to look, is the data actually queryable, or buried where no one can reach it?_ Capture gets the work recorded, but a record no one can question is just storage. Availability is the discipline of organizing that record so anyone can put a question to it and get an answer: how often a tool call fails for a given reason, which sessions touched a service, where the spend is concentrated. When the trail is queryable, your team answers its own questions instead of routing each one through whoever knows where the logs live. The payoff is a record the org can interrogate directly: finding out how your agents are behaving costs a query, not a wait. ### The right tier, not the top You do not need a dashboard for every question to do Availability well. A team whose one engineer can pull the answer when asked may be exactly where it should be, and formalizing the record before you know which questions the org actually asks just adds a schema you have to version and keep aligned with what capture writes. There is a second reason not to jump straight to the top. You cannot know in advance which questions a schema should answer: that depends on what your org actually asks of the record, and pulling answers by hand is the cheapest way to learn it. Formalize the record before you know, and you have baked a guess into a schema you now have to version, one expensive to change and not just to maintain. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is an honest look at the tradeoff. Each tier up turns answers from a favor into self-serve and leans less on any one person, and each costs more to build and keep current. Read down the four, find where your team sits, then decide whether the next tier is worth buying. ### The four investment tiers #### Exploratory Whatever was captured is a pile, and “how often does this fail?” has no answer yet. - **Enables:** A raw record exists, and reading it by hand surfaces the questions your org actually puts to it. Those questions are what a queryable schema later gets shaped around. - **Deficiencies:** Nothing is queryable, no one knows what is in there or where to look, and questions go unanswered. - **Cost to maintain:** Time spent digging through logs for an answer that may not be there. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One person is the query layer, and the record is only as reachable as they are. - **Enables:** Real answers to “how often” and “which sessions,” as long as that person is around to pull them. - **Deficiencies:** A single point of failure whose knowledge of how to query lives in their head, so it does not scale past their calendar. - **Cost to maintain:** Their time fielding and running queries. - **Techniques:** - **Structured Comments.** Typed comments with parseable markers (plan, progress, fix, review, complete) posted as the work runs, so parallel work signals ownership and a downstream reader pulls what happened without reading everything. - **Progress Trackers.** A JSON state file that is decomposition, controller, and handoff at once, so a new session knows what is done, what is next, and what the last one learned. #### Documented The team knows where the record lives and how to ask it questions, and the conventions are written down. - **Enables:** Anyone on the team can answer common questions from the record, and it survives the expert leaving. - **Deficiencies:** Still manual querying that depends on people keeping the format parseable, with gaps whenever capture changes shape. - **Cost to maintain:** Writing down the schema and query conventions, plus keeping comments and trackers in the agreed format. - **Techniques:** - **Structured Comments.** Typed comments with parseable markers (plan, progress, fix, review, complete) posted as the work runs, so parallel work signals ownership and a downstream reader pulls what happened without reading everything. - **Progress Trackers.** A JSON state file that is decomposition, controller, and handoff at once, so a new session knows what is done, what is next, and what the last one learned. #### Structural The record is append-only with a schema, and anyone can run their own query long after the author is gone. - **Enables:** Questions answered without a person in the loop, consistent and repeatable queries, and a system of record with a memory the org can draw on. - **Deficiencies:** Schema drift breaks queries when capture changes, and a dashboard can imply an answer the data does not support. - **Cost to maintain:** Building and versioning the schema and dashboards upfront, plus tuning as capture evolves. - **Techniques:** - **Database Audit Trail.** Append-only, queryable logging at multiple granularities (tool-call events, orchestrator events, session summaries), so the record can be watched live and interrogated long after the run, self-serve dashboards included. ## Trace _Observability · When something goes wrong, can we reconstruct exactly what the agent did and why?_ An agent's finished work does not explain how it got there. Trace is the discipline of answering that for a single run: every decision the agent made, and the rationale behind it, preserved from the opening prompt to the change that shipped. When that trail holds, a failure from last quarter is something you reconstruct from the record left as the work ran, not something you reverse-engineer from the output and a few people's memories. Incident reviews start from evidence, and when an auditor asks how a decision got made, you have an account to put in front of them. ### The right tier, not the top You do not have to build full reconstruction into every run to do Trace well. A team whose one engineer can walk back the runs they had a hand in may be exactly where it should be, and building a decision trail across every session before you know which ones you will have to defend is infrastructure you will maintain far more than you consult. There is a second reason not to jump straight to the top. You cannot know in advance which runs a decision trail should reconstruct: that depends on which ones you actually have to defend, and walking back your own runs is the cheapest way to learn it. Build the trail across every session before you know, and you have baked a guess into infrastructure that is expensive to change, not just to maintain. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is an honest look at the tradeoff. Each tier up leaves more of the why in the record and leans less on who happened to be watching, and each costs more to build, store, and govern. Read down the four, find where your team sits, then decide whether the next tier is worth buying. ### The four investment tiers #### Exploratory A run goes wrong and the why is gone: you stare at the output with no path back to the decisions behind it. Every team starts here. - **Enables:** The final output is still there, and the runs you find yourself wishing you could walk back are the ones a decision trail later gets built to preserve. - **Deficiencies:** No record of what the agent did or why, so postmortems are guesswork and there is nothing to show an auditor. - **Cost to maintain:** Reruns to try to reproduce a failure, often without reproducing it. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One engineer can reconstruct a run from memory and their private notes. - **Enables:** The cases they were close to can be traced, and their reconstruction is usually right. - **Deficiencies:** A single point of failure whose rationale lives in their head rather than the record, so cold and old runs are unrecoverable. - **Cost to maintain:** Their time reconstructing, plus the risk that walks out the door with them. - **Techniques:** - **Structured Comments.** Typed comments with parseable markers (plan, progress, fix, review, complete) posted as the work runs, so parallel work signals ownership and a downstream reader pulls what happened without reading everything. - **Progress Trackers.** A JSON state file that is decomposition, controller, and handoff at once, so a new session knows what is done, what is next, and what the last one learned. #### Documented The team pieces a run together from scattered comments and trackers, and events are on the record even where rationale is thin. - **Enables:** Anyone can follow what happened, and postmortems draw on a shared trail rather than one memory. - **Deficiencies:** Rationale is still partial, provenance like sources read and queries run is not consistently cited, and gaps open where no one commented. - **Cost to maintain:** The habit of posting comments and updating trackers as the work happens. - **Techniques:** - **Structured Comments.** Typed comments with parseable markers (plan, progress, fix, review, complete) posted as the work runs, so parallel work signals ownership and a downstream reader pulls what happened without reading everything. - **Progress Trackers.** A JSON state file that is decomposition, controller, and handoff at once, so a new session knows what is done, what is next, and what the last one learned. #### Structural Decisions are recorded with rationale and provenance as the work runs, so anyone can reconstruct a run months later. - **Enables:** Full postmortems and an audit-ready record, cold cases that stay recoverable, and a trace that does not depend on who was present. - **Deficiencies:** Logging volume can bury the signal, and capture must be designed not to leak secrets into the trail. - **Cost to maintain:** The logging pipeline upfront, plus storage, retention policy, and query tooling over time. - **Techniques:** - **Database Audit Trail.** Append-only, queryable logging at multiple granularities (tool-call events, orchestrator events, session summaries), so the record can be watched live and interrogated long after the run, self-serve dashboards included. - **Structured Comments.** Typed comments with parseable markers (plan, progress, fix, review, complete) posted as the work runs, so parallel work signals ownership and a downstream reader pulls what happened without reading everything. - **Progress Trackers.** A JSON state file that is decomposition, controller, and handoff at once, so a new session knows what is done, what is next, and what the last one learned. ## Evaluation _Improvement · Do we actually know whether our AI investments are paying off, or is it a feeling?_ Every team using AI has an opinion about whether it is working; far fewer have evidence. Evaluation is the discipline of producing that evidence: evals and metrics on one side, retro sentiment and a lead's read on the work on the other. When the output is held to a real standard of good, you catch a regression before it ships, you answer the ROI question with numbers instead of a hunch, and you fund next quarter's AI bets from what you measured rather than what you remember. ### The right tier, not the top You do not have to measure efficacy automatically to measure it well. A team whose one lead judges the output honestly may be exactly where it should be, and building an evaluation harness before anyone on the team can name what good is just instruments a target you have not defined yet. There is a second reason not to jump straight to the top. You cannot know in advance what a harness should even measure: that depends on what good means for your work, and judging the output by hand is the cheapest way to find that out. Build the evaluation harness before you can name the target, and you have baked a guess into instruments that are expensive to change, not just to maintain. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is an honest look at the tradeoff. Each tier up makes the verdict less subjective and less tied to who happens to be looking, and each costs more to build and tune. Read down the four, find where your team actually sits, then decide whether the next tier is worth buying. ### The four investment tiers #### Exploratory No one on the team asks whether the AI work is actually paying off. A change goes in, the week feels faster than the last one, and everyone moves on to the next thing. Efficacy is a gut feeling, and nobody stops to check the gut. This is not the same as flying blind on purpose. Most teams land here first because measurement costs something to stand up, and early on there is always a louder fire. The work still ships. You just have no way to tell a real improvement from one that only felt like one, and no way to notice a change that quietly did damage. - **Enables:** Full speed, and a first felt sense of where the work is unreliable. That hunch is the start of naming what good means, which is what any eval later has to measure against. - **Deficiencies:** No read on what actually works, regressions ship unnoticed, and ROI is a guess. - **Cost to maintain:** Rework when an unmeasured change makes things worse. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One person, usually the lead who pushed hardest on AI, carries the read. They look at the output, weigh it against what they know good looks like, and call it. When they say a change is working, people believe them, and they are usually right. The judgment is real, but it lives in one head. It is not written down, it does not run when that person is heads-down on something else, and their blind spots are the team's blind spots. You get a filter for the obvious, which is worth far more than nothing and noticeably less than a second opinion. - **Enables:** A sanity check. The obvious wins and regressions get caught. - **Deficiencies:** Subjective and unrecorded, bottlenecked on their judgment, and their bias goes unchecked. - **Cost to maintain:** That person's time, spent judging the work. - **Techniques:** - **Verification Scripts.** The same lint, test, and e2e checks run before and after a change, so a claimed improvement has a baseline to argue from instead of a feeling. - **Agent Devil's-Advocate Pass.** A separate agent pass that argues against the work, hunting regressions, weak spots, and unstated assumptions, so confirmation bias gets caught before a human signs off. #### Documented Efficacy has a rubric now. The team has written down what good means for their work, in numbers where numbers fit and in plain description where they do not, and the read gets argued in retro against that standard instead of one person's taste. The argument is better because the criteria were agreed before the work was judged. A junior engineer can push back on a senior's call by pointing at the rubric, and the verdict survives the meeting. What it still is not is durable. The rubric lives in a doc someone has to remember to open, nobody writes the verdicts anywhere, and the judging waits for the retro instead of running alongside the work. It is a shared habit, not yet a system. - **Enables:** Shared criteria for what “working” means, quantitative and qualitative, so the team reasons from the same evidence. - **Deficiencies:** Still manual, results often go unrecorded, and it measures after the fact rather than continuously. - **Cost to maintain:** Writing the rubric and holding the retro. - **Techniques:** - **Verification Scripts.** The same lint, test, and e2e checks run before and after a change, so a claimed improvement has a baseline to argue from instead of a feeling. - **Database Audit Trail.** The append-only session log read as an evidence base: before-and-after metrics that answer the ROI question with numbers. - **Agent Devil's-Advocate Pass.** A separate agent pass that argues against the work, hunting regressions, weak spots, and unstated assumptions, so confirmation bias gets caught before a human signs off. #### Structural The measurement runs on its own. Evals fire on every change, the metrics that matter sit on a dashboard someone actually watches, and the ROI question has a standing answer. The read no longer depends on any particular person being in the room. This is real engineering, and it is worth naming the cost. Good automated evals are hard to write and harder to keep honest as the system shifts underneath them, and a team that trusts the number can stop looking at the thing the number stands in for. Human judgment does not disappear here. It gets a better place to stand: the humans spend their attention on the calls the harness cannot make, because the harness is already holding everything it can. - **Enables:** Every process and change measured, quantitative and qualitative, with ROI tracked on an ongoing basis rather than only when a crisis forces the question. - **Deficiencies:** The number invites over-trust, an eval can keep passing long after it stops measuring what matters, and qualitative judgment still needs a human. - **Cost to maintain:** Building and tuning the evaluation harness and ROI instrumentation, plus upkeep as the system changes. - **Techniques:** - **Verification Scripts.** The same lint, test, and e2e checks run before and after a change, so a claimed improvement has a baseline to argue from instead of a feeling. - **Database Audit Trail.** The append-only session log read as an evidence base: before-and-after metrics that answer the ROI question with numbers. - **Agent Devil's-Advocate Pass.** A separate agent pass that argues against the work, hunting regressions, weak spots, and unstated assumptions, so confirmation bias gets caught before a human signs off. - **Evaluation Harness.** Automated evals that score output against a fixed standard of good on every run, so efficacy is measured continuously instead of judged after the fact. ## Prioritization _Improvement · When we decide what to improve next, are we reading real signal or chasing the loudest voice?_ The field does not hold still: new tools, new techniques, a new thread every week claiming to change how the work gets done. Prioritization is the discipline of steering your team's attention through that churn: which sources of signal you let in, how you weigh them against the work already in front of you, and which changes you take up versus deliberately let pass. When that steering runs on evidence, your team spends its effort on the changes that compound, and keeps pace instead of waking up a year behind. ### The right tier, not the top You do not need a standing process to prioritize well. A small team whose one engineer reads the field and picks the next bet may be spending exactly the right amount, and building an intake pipeline before you know which signal is worth acting on just adds a ritual to feed. There is a second reason not to jump straight to the top. You cannot know in advance which signal an intake process should let in: that depends on what actually turns out worth acting on, and reading the field by hand is the cheapest way to learn it. Stand up the pipeline before you know, and you have baked a guess into a ritual you now have to feed, one expensive to change and not just to run. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is an honest look at the tradeoff. Each tier up makes the choosing something the team does together rather than one person's instinct, and each costs more to run and maintain. Read down the four, find where your team actually sits, then decide whether the next tier is worth buying. ### The four investment tiers #### Exploratory No plan. Changes happen reactively, driven by whoever cares most or whatever thread landed last. - **Enables:** Movement without a prioritization tax, and a cheap read on which signals are worth acting on. Chasing a few loud ideas by hand teaches you what an intake process later gets built to filter for. - **Deficiencies:** Effort scatters with nothing to rank it, no outside signal comes in, and the field moves while the team does not. - **Cost to maintain:** Compounding obsolescence, plus the opportunity cost of investing in the wrong things. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One engineer's reading and judgment are the whole team's intake and plan. - **Enables:** Some signal reaches the team, and the occasional well-chosen bet gets made. - **Deficiencies:** A single point of failure whose feed and priorities are unwritten, so the awareness leaves when they do. - **Cost to maintain:** Their time scanning, triaging, and choosing, unrecognized and invisible. - **Techniques:** - **Landscape Scan / Research Digest.** A written scan of the external AI landscape (new tools, techniques, threads) that ends in a recommendation rather than a pile of links. - **Pattern Libraries.** Catalog documents that encode the choices already made for a domain: which libraries to use, how to structure things, and the anti-patterns to avoid. #### Documented Signal gets collected and discussed, and priorities are written down, though intake and prioritization stay uneven. - **Enables:** Shared awareness and an agreed set of priorities, with debate happening in the open rather than in one head. - **Deficiencies:** Discussion without a decision, signal and noise mixed in one feed, and priorities that drift from what actually gets done. - **Cost to maintain:** Curating the intake and holding the prioritization, plus the time spent on threads that go nowhere. - **Techniques:** - **Landscape Scan / Research Digest.** A written scan of the external AI landscape (new tools, techniques, threads) that ends in a recommendation rather than a pile of links. - **Prioritization Rubric.** A written rubric that ranks candidate changes against shared criteria, so what to invest in next is chosen legibly rather than by whoever pushes hardest. - **Pattern Libraries.** Catalog documents that encode the choices already made for a domain: which libraries to use, how to structure things, and the anti-patterns to avoid. #### Structural A standing cadence pulls signal in and produces prioritized investment decisions, independent of who spotted what. - **Enables:** The team stays current and invests deliberately rather than reactively, and priorities are legible and revisited. - **Deficiencies:** A cadence can chase novelty over value if the prioritization bar is loose, and its upkeep competes with shipping. - **Cost to maintain:** The recurring ritual, the intake pipeline, and the follow-through to hold the priorities. - **Techniques:** - **Landscape Scan / Research Digest.** A written scan of the external AI landscape (new tools, techniques, threads) that ends in a recommendation rather than a pile of links. - **Prioritization Rubric.** A written rubric that ranks candidate changes against shared criteria, so what to invest in next is chosen legibly rather than by whoever pushes hardest. - **Pattern Libraries.** Catalog documents that encode the choices already made for a domain: which libraries to use, how to structure things, and the anti-patterns to avoid. - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. ## Convergence _Improvement · When something works for one engineer, does it reach the whole team, or die in their private setup?_ Somewhere on your team, somebody has already solved a problem the rest are still living with: a skill that makes review faster, a pattern that keeps agents on the rails. Convergence is the discipline of turning that solo win into team practice: surfacing it where everyone can see it, packaging it so others can pull it in, and making it cheap enough to try that saying yes costs an afternoon. When a good change propagates, skills travel across your repos and the tooling starts compounding: tools that build tools. The improvement outlives the person who found it, and the next hire inherits it instead of rediscovering it. The discipline takes its name from the framework's diverge-then-converge principle. The principle names the posture: let people explore, then fold what works back in before everyone's practice drifts apart. This discipline is that posture at team scale: engineers experimenting on their own is the divergence, and Convergence is how the team closes the loop. ### The right tier, not the top You do not have to automate the spread to make a change stick. A team whose one engineer passes improvements along by hand and the rest borrow them may be exactly where it should be, and building that machinery before you know which wins are worth spreading just moves noise around faster. There is a second reason not to jump straight to the top. You cannot know in advance which wins propagation should spread: that depends on which improvements actually earn their way across the team, and passing them along by hand is the cheapest way to learn it. Build the propagation machinery before you know, and you have baked a guess into a system that is expensive to change, not just to maintain. So each tier earns the next, which is no license to camp at Exploratory: it costs a lot for little, and for most teams the question is not whether to climb but how far. What the four investment tiers give you is an honest look at the tradeoff. Each tier up moves adoption from luck toward default, and each costs more to build and police. Read down the four, find where your team actually sits, then decide whether the next tier is worth buying. ### The four investment tiers #### Exploratory Improvements stay in the head of whoever found them. Nothing is shared or tried widely. - **Enables:** The person who improves gets the benefit at no coordination cost, and which of their wins others reach for on their own tells you which are worth spreading. That is what propagation later gets pointed at. - **Deficiencies:** Wins stay siloed, the same lesson gets re-learned, and there is no path from one person's change to team practice. - **Cost to maintain:** Re-paying for improvements that never propagate. - **Techniques:** none yet (manual and ad hoc). #### Expert-led One person shares their wins informally and others copy them when they notice. - **Enables:** Some spread. The people close to that person pick up what works. - **Deficiencies:** A single point of propagation, sharing that is ad hoc, and opt-in that depends on being in the right channel. - **Cost to maintain:** Their time evangelizing, on top of the work itself. - **Techniques:** - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. - **Pattern Libraries.** Catalog documents that encode the choices already made for a domain: which libraries to use, how to structure things, and the anti-patterns to avoid. #### Documented A sharing ritual runs, a retro or a demo, and adopting what a teammate built is deliberately made easy. - **Enables:** Changes reach the team rather than one corner, what is available to try is visible to everyone, and convergence happens by agreement. - **Deficiencies:** It still relies on the ritual running and people choosing to opt in, so a win lags before the team adopts it. - **Cost to maintain:** Holding the ritual and curating the shareable artifacts. - **Techniques:** - **Share Ritual.** A recurring retro or demo where engineers surface what worked, so a win reaches the whole team instead of dying in one person's setup. - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. - **Pattern Libraries.** Catalog documents that encode the choices already made for a domain: which libraries to use, how to structure things, and the anti-patterns to avoid. - **Convergence Trial.** A low-friction, reversible way to try a new practice, so adopting a change is a small opt-in step rather than a project. #### Structural Sharing and propagation are built into the everyday tooling. A proven change reaches every project with easy opt-in by default. - **Enables:** Convergence without manual copying, trials and opt-in built into the system, and improvements that compound. - **Deficiencies:** Automated propagation can spread noise as signal, and it needs guardrails on what auto-adopts. - **Cost to maintain:** Building and tuning the propagation, plus reviewing what it spreads. - **Techniques:** - **Config-Driven Deployment.** Config plus install scripts that propagate a skill or doc from one place to every project, so an update in one repo lands in all of them. - **Skills.** Reusable prompt documents that encode a workflow, its rules, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. - **Pattern Libraries.** Catalog documents that encode the choices already made for a domain: which libraries to use, how to structure things, and the anti-patterns to avoid. - **Convergence Trial.** A low-friction, reversible way to try a new practice, so adopting a change is a small opt-in step rather than a project. ## The technique catalog The concrete moves that take a discipline up the tiers. A technique listed under a discipline materially moves it. Many move more than one, which is the point: the map overlaps. ### 01. Skills Reusable prompt documents that encode a workflow, its rules, its templates, and its anti-patterns, so a task starts from a runbook instead of a fresh explanation. A skill is written once and reused across sessions, projects, and people. **Addresses:** Collection (Expert-led); Disclosure (Expert-led); Delegation (Expert-led, Documented, Structural); Environment (Expert-led, Documented, Structural); Prioritization (Structural); Convergence (Expert-led, Documented, Structural). **In practice.** The outcome a skill buys is consistency: two engineers on two laptops produce work that reads like it came from the same team, because it followed the same skill. That is what lets you hand more of the routine work to an agent without the quality drifting, and it is why skills recur across so much of the map, from how context gets gathered to how a change gets shipped. Without them, every engineer reinvents the same prompt and gets slightly different output, and the variance lands in review. The principle underneath is input as the lever: a skill is built once, upstream of the run, and it pays off on every run after. ### 02. Orientation Docs Auto-loaded files the agent reads before anything else (CLAUDE.md, AGENTS.md, README), stating your stack, your conventions, and the patterns you have already ruled out. The agent opens every session already oriented instead of guessing. **Addresses:** Collection (Expert-led, Documented); Management (Expert-led, Documented). **In practice.** The outcome is rework that drops and decisions that land right the first time: a good orientation doc roughly halves the “why did it do it that way” review comments, because the agent stops making plausible but wrong assumptions a human then has to catch. It is the cheapest of Context's high-return moves, which is why it is worth having from the moment one engineer writes it down through the point where the team keeps it current. Skip it and you pay for the missing context on every session, quietly, in review. ### 03. Domain Knowledge A directory of knowledge files, one per service or entity, each written to the same shape: purpose, responsibilities, boundaries, key paths. The consistent structure is what makes the corpus navigable, so the agent finds the boundary and stays inside it. **Addresses:** Collection (Documented); Management (Documented). **In practice.** The outcome is faster orientation and fewer wrong turns: “what does the billing service own” gets answered from a file in seconds rather than reconstructed from a Slack thread. Because every file has the same shape, the agent reads the boundaries and works within them instead of guessing from whichever file it grepped first. This is where Collection and Management meet: the same one-file-per-entity structure that makes the corpus complete also makes it something a named owner can keep current. ### 04. Index Files Lightweight “when to read” tables that route the agent from a task to the one knowledge file it needs, instead of loading everything. A routing layer, not the knowledge itself. **Addresses:** Management (Documented); Disclosure (Expert-led, Documented, Structural). **In practice.** The outcome is leaner sessions and lower spend: the agent reads the index, pulls the two files the task calls for, and skips the other forty, so the context window stays small without losing reach. That shows up as fewer “I think I know what you mean” answers and more “I read the auth module's spec and here is what it says.” An index is also the cheapest way to keep disclosure honest as the corpus grows, which is why it carries Disclosure from one person's habit all the way to a default no one has to remember. ### 05. Prompt Templates Template files with placeholders for the runtime context a workflow needs (issue IDs, git state, prior session history), so the same workflow adapts to a new task without a rewrite. The stable instruction lives in the template; the variable parts get filled in per run. **Addresses:** Management (Expert-led). **In practice.** The outcome is a workflow the second engineer can run without remembering what the first one did: a team with templates spins up a new contributor on a workflow in an afternoon. Without them you end up with five private variations of the same prompt, none written down, and the workflow walks out the door when its author does. A template is a small, upstream investment that keeps a Context practice from living in one person's memory. ### 06. Contracts and Prohibitions Explicit do-not lines written into skills and orientation docs, ruling out a whole class of action rather than asking the agent to weigh it. Rules, not guidance. **Addresses:** Capability (Documented); Delegation (Expert-led, Documented, Structural). **In practice.** The outcome is fewer incidents of a specific kind: “never run git push --force on main” is more reliable than “be careful with destructive git commands,” and the whole class of mistake simply stops happening, without anyone remembering to remind the agent. This is the Humans-make-decisions principle drawn as a hard line: the high-stakes call is reserved, and the prohibition holds it even when no one is watching the run. It carries Delegation from one lead's rule of thumb to a team norm to a contract nothing is allowed to cross. ### 07. Permission Allow-Lists A settings file that enumerates exactly which commands an agent may run, started from zero and widened only as the work needs it. The agent's reach becomes a reviewed decision instead of whatever the tool shipped with. **Addresses:** Capability (Expert-led, Documented, Structural); Delegation (Structural). **In practice.** The outcome is a smaller blast radius: the list length is a posture statement, and an unbounded one is the liability, because a destructive command or a wrong-account push is otherwise one prompt away. A short list says a human approves anything past a trusted surface; a longer one says you have watched the agent work and know what it needs. Either is fine. Getting the reach wrong on the open side is what turns a confident mistake into an incident, so this is where Capability starts and where Delegation's reserved actions get enforced. ### 08. Verification Scripts Standardized host scripts (lint, test, e2e) the agent runs and reads the exit code from, so the project's own checks fire before a human sees the output. The script absorbs the project-specific complexity, so the agent's skill stays portable. **Addresses:** Management (Structural); Capability (Structural); Environment (Expert-led, Documented, Structural); Evaluation (Expert-led, Documented, Structural). **In practice.** The outcome is quality caught where it happens instead of in review: the agent runs bin/lint, reads the result, and self-corrects before anyone looks, so the review comments a linter could have caught disappear. It is one of the framework's true bridges, the same practice that keeps Context current, hardens Control's environment, and gives Improvement a fixed standard to measure against. That last use is the Fluency-isn't-value principle in practice: a check the system cannot talk its way past. The same scripts can also be pointed at the environment itself: preflight and baseline checks before any run, so a broken workbench or a pre-existing failure surfaces before it can be blamed on the change. ### 09. Structured Comments Typed comments with parseable markers (plan, progress, fix, review, complete) posted as the work runs, each in a defined format a downstream reader can pull from. The record of what happened is written as the work happens, not reconstructed after. **Addresses:** Environment (Documented, Structural); Capture (Expert-led, Documented); Availability (Expert-led, Documented); Trace (Expert-led, Documented, Structural). **In practice.** The outcome is a run you can reconstruct and a review you can scan: on a healthy team you can read an issue's comment thread and follow the whole arc in under a minute. A structured comment does four jobs at once, announcing what a session is doing, leaving a trail for the next one, signaling to parallel agents not to step on the work, and giving a reviewer something legible. That reach across Observability and Control is why so much of the map rests on it: the low-cost way a record exists before anyone needed it. ### 10. Progress Trackers A JSON state file that is decomposition, controller, and handoff at once: a new session reads it to know what is done, what is next, and what the last session learned. The tracker holds the state; sessions are stateless workers against it. **Addresses:** Disclosure (Documented); Delegation (Expert-led, Documented, Structural); Capture (Expert-led, Documented); Availability (Expert-led, Documented); Trace (Expert-led, Documented, Structural). **In practice.** The outcome is work that does not stall when a person steps away: the next session, human or agent, picks it up exactly where the last one stopped, so long-running work survives session boundaries without a coordination tax. It is also half of durable Delegation, the checkpoint an agent pauses at, and half of a Trace, the record of what each step decided. A tracker is a small artifact that keeps continuity from depending on one person holding the thread in their head. The same shape can be lifted into a PM tool's workflow states and enforced by a dispatcher: a tier climb of this technique, not a new one. ### 11. Session Prompts A standalone protocol document for picking up work in a fresh session: read the state, take the next step, repeat in a fixed loop. The prompt defines the loop; a progress tracker defines the state. **Addresses:** Disclosure (Expert-led, Documented, Structural). **In practice.** The outcome is something close to a durable workflow: the prompt tells a new session to read the tracker, take the next pending item, follow these steps, and mark it done, and the same prompt works across sessions, projects, and agents. A fresh session starts empty, so this is how you re-prime it with what the last one knew instead of briefing it by hand every time. Paired with a progress tracker, it turns picking up cold into a routine the agent runs itself. At the far end the handoff runs without a person: the harness detects the context ceiling, gathers the finished-and-pending state from the structured record, and writes the next session's briefing itself. ### 12. Tool Wrappers Scripts that wrap standard CLI tools with guardrails (credential routing, identity checks, confirmation on destructive operations), so the safety lives in the tool rather than in the prompt. The wrapper encodes what you wish the underlying tool did by default. **Addresses:** Capability (Expert-led, Documented, Structural); Delegation (Structural); Environment (Expert-led, Documented, Structural). **In practice.** The outcome is a class of incident that stops recurring: a bin/gh wrapper that routes credentials per client means the agent never has to reason about which account it is posting from, and “the agent committed to the wrong repo” simply stops happening. The fix lives in the wrapper, not in every skill that calls the tool, which is why it hardens both Capability and the wider Environment. It is the Input-is-the-lever principle applied to tooling: put the guardrail in the thing the agent reaches for, and it holds on every run. ### 13. Pattern Libraries Catalog documents that encode the choices already made for a domain: which libraries to use, how to structure things, and the anti-patterns to avoid. Each library is the team's accumulated taste, written down. **Addresses:** Prioritization (Expert-led, Documented, Structural); Convergence (Expert-led, Documented, Structural). **In practice.** The outcome is a codebase that stops drifting and a lesson you stop re-teaching: new contributors, human or agent, reach for the same primitives in the same way. A pattern library is also a learning instrument, which is why it sits in both Prioritization and Convergence: when the same anti-pattern keeps surfacing in review, that is the signal to add it to the library rather than flag it one PR at a time, and once it is written down the fix reaches everyone instead of dying with the reviewer who caught it. ### 14. Database Audit Trail Append-only, queryable logging at several granularities (tool-call events, orchestrator events, session summaries), so the record can be watched live and interrogated long after the run. A system of record with a memory you can query. **Addresses:** Capture (Documented, Structural); Availability (Structural); Trace (Structural); Evaluation (Documented, Structural). **In practice.** The outcome is a question that costs a query instead of a favor: how often the agent retries a tool, which sessions touched a service, which prompt produced the most rework, where the spend concentrated. Without it every retro is anecdotal and every incident review runs on memory; with it, “the agent did something weird” becomes something you can actually answer. It is where Observability's record becomes reachable and Improvement's evidence base comes from, the difference between believing a change worked and reading whether it did. ### 15. Config-Driven Deployment Config plus install scripts that propagate a skill, doc, or control from one place to every project, so an update in one repo lands in all of them. The deployment problem is solved once. **Addresses:** Management (Structural); Disclosure (Structural); Capability (Structural); Environment (Structural); Convergence (Structural). **In practice.** The outcome is an improvement that shows up everywhere without anyone copying files: fix a skill in one repo and it lands in the other twelve, and a hardened control applies to every project rather than the one it was written in. It is the technique that carries a change across the whole estate once sharing is built into the repos themselves, which is why so many disciplines list it at Structural. Worth naming its limit: it propagates artifacts, not the observation in one engineer's head, so people-level sharing still runs through the Convergence practices. ### 16. Source Connectors Live integrations that pull artifacts straight from the systems of record: the repo, the PM tool, the wiki, the transcript store. The agent sees today's state instead of last month's export. **Addresses:** Collection (Structural). **In practice.** The outcome is context that stays current on its own rather than because someone remembered to update it: when those systems feed the agent directly, a new session opens on reality, and the gathering stops being a chore anyone can forget. The engineering cost is real, one integration per source, so this is an investment you make once you know which artifacts actually matter. It is what Collection looks like as infrastructure rather than a habit. ### 17. Automatic Capture Hooks that record new artifacts as they land (a merged PR, a closed ticket, a fresh transcript), so the corpus never falls behind the work the team is doing. Completeness without anyone maintaining it. **Addresses:** Collection (Structural). **In practice.** The outcome is a setup that does not quietly rot: each new artifact is recorded as it happens, so the material the agent reads keeps pace with the team. The failure mode it removes is the worst kind of stale, the artifact that exists but never made it into what the agent can see, which is exactly the gap that undermines an otherwise complete corpus. Paired with source connectors, it is how Collection reaches the tier where nobody has to remember to gather. ### 18. Doc Currency Hooks Hooks in the commit and CI path that update or validate docs against the code as the work lands, so orientation files and indexes cannot silently drift. A doc that names the old stack fails the check that ships the new one. **Addresses:** Management (Structural). **In practice.** The outcome is drift caught at the commit that caused it rather than three sprints later by an agent heading down an abandoned path: the work itself keeps the docs honest, instead of a person remembering to. The cost is real integration and tuning, and an over-strict rule will nag on changes that did not actually invalidate a doc, so it belongs on the docs everything else depends on, where staleness is expensive. The hook does not have to live in the commit path: a harness can trigger the update from its own lifecycle, dispatching an agent session to refresh an index when a slice of work completes, with a failed update never blocking the delivery. ### 19. Escalation Affordance A built-in path for the agent to stop and ask a human when it is stuck, instead of looping or silently giving up. Asking for help is built into the harness rather than left to the agent's judgment. **Addresses:** Delegation (Structural). **In practice.** The outcome is a run that pauses and asks a sharp question instead of one that burns an afternoon down a dead end: most setups either barrel ahead on a bad assumption or error out after a few retries, and an escalation affordance gives the agent a third option, surfacing the block to a person before the wasted work compounds. The framework names mid-work escalation as its most under-addressed Control gap: even teams that have built their delegation rules into the tooling are usually still missing this piece. It is Humans-make-decisions applied to the moment the agent realizes it is out of its depth. ### 20. Session Telemetry Hooks Hooks that record each session's operational metadata (cost, token spend, model, platform) automatically as it runs, so the record stays complete without anyone choosing to log. **Addresses:** Capture (Structural). **In practice.** The outcome is cost and behavior as a live readout instead of a monthly surprise: when every session's spend and model land in the record on their own, the budget conversation runs on numbers, and no incident is unreconstructable because someone forgot to log. It is what takes Capture to the tier where completeness no longer depends on anyone's diligence. The cost is the instrumentation and the storage it fills; the payoff is a trail that is complete by default. ### 21. Evaluation Harness Automated evals that score output against a fixed standard of good on every run, so efficacy is measured continuously instead of judged after the fact. A standard the system cannot generate for itself. **Addresses:** Evaluation (Structural). **In practice.** The outcome is knowing your AI investment is paying off rather than believing it: a regression shows up as a failing eval the day it lands, not as a vague sense months later that quality slipped. This is the Fluency-isn't-value principle made structural: the standard lives outside the model, so fluent output cannot pass for good output. The engineering is real, a good evaluator is its own project, which is why it belongs where Evaluation runs as a built system rather than a first experiment: it is worth building where a silent regression is expensive. ### 22. Agent Devil's-Advocate Pass A separate agent pass that argues against the work, hunting regressions, weak spots, and unstated assumptions, so confirmation bias is caught before a human signs off. The agent that wrote the work is the worst judge of it. **Addresses:** Evaluation (Expert-led, Documented, Structural). **In practice.** The outcome is a review queue where the obvious problems are already flagged: a fresh agent goes at the output with one job, find what is wrong, and it catches the confident mistake the implementer talked itself into. It also structures the work for a human reviewer who would otherwise fatigue and rubber-stamp, which protects the judgment call the framework insists a human keep. This is diverge-then-converge with a second agent as the convergence step, and it belongs in Evaluation from the first tier because it costs little and catches a lot. ### 23. Landscape Scan / Research Digest A written scan of the external AI landscape (new tools, techniques, threads) that ends in a recommendation rather than a pile of links. Keeping up, made deliberate. **Addresses:** Prioritization (Expert-led, Documented, Structural). **In practice.** The outcome is a team that stays current on purpose instead of drifting toward obsolescence without noticing. The field moves weekly; someone reads widely and writes up what actually matters, so the team adopts the change worth adopting instead of chasing whatever thread was loudest. It is the intake that feeds Prioritization at every tier, from one person's reading to a standing digest the whole team works from. ### 24. Prioritization Rubric A written rubric that ranks candidate changes against shared criteria, so what to invest in next is chosen legibly rather than by whoever pushes hardest. The criteria are explicit and the ranking is visible. **Addresses:** Prioritization (Documented, Structural). **In practice.** The outcome is a roadmap you can defend: without a rubric the next investment goes to the best arguer, not the change that compounds, and a rubric makes the trade-off visible so a leader can point at why one bet beat another. It also lets the team revisit a call when the facts change, rather than relitigating it by volume. This is the Map-not-a-scorecard principle applied to spending: you are choosing where to invest across the framework, and the rubric is how that choice stays legible. ### 25. Convergence Trial A low-friction, reversible way to try a new practice, so adopting a change is a small opt-in step rather than a project. A proven change, packaged so a team can back out cleanly if it does not fit. **Addresses:** Convergence (Documented, Structural). **In practice.** The outcome is a win found by one engineer that actually reaches the others: good practices die when trying them costs too much, and a trial turns adoption from a commitment into an experiment. It is the carrot to Control's stick, the easy opt-in rather than the enforced standard, which is why it lives in Convergence and not in a gate. Lowering the cost of trying is often the whole difference between a change that spreads and one that stalls on the friction of switching. ### 26. Share Ritual A recurring retro or demo where engineers surface what worked, so a win reaches the whole team instead of dying in one person's setup. The standing venue for people-level sharing. **Addresses:** Convergence (Documented). **In practice.** The outcome is a good idea that reaches everyone by the next cycle rather than whenever someone happens to mention it: tools propagate skills across projects, but not the observation in one engineer's head, and a share ritual is what carries that. It is the people-level half of Convergence that config-driven deployment cannot reach, the reason the discipline needs a human venue and not just a pipeline. The ritual is where what worked gets said out loud, and said is the first step to spread. ### 27. Intent Markers The agent requests each side effect through a typed, parseable marker in its output, and the harness performs the action in deterministic code, with an end-of-session pass that commits any work the agent forgot to request. The risky operation is always executed by a fixed code path; the agent never holds the tool at all. **Addresses:** Delegation (Structural); Capability (Structural). **In practice.** The outcome is delegation that stops requiring belief: when the merge, the push, or the ticket transition is performed by deterministic code on the agent's signal, the record shows an observed exit code instead of the agent's claim that it happened. The split also turns prohibitions into facts: an agent that does not hold the credentials cannot force-push, and its worst case collapses to a malformed marker that fails to parse. Where a tool wrapper puts guardrails around a tool the agent still runs, an intent marker takes the tool away entirely: the agent proposes, a fixed code path disposes. That is what lets unattended scope grow while the agent's direct reach shrinks. ### 28. Bounded Fix Loop On a failed check, the harness dispatches a fresh, narrowly scoped repair session carrying the failure output and the record of every prior attempt, so approaches are not repeated. The cycles are capped at a fixed number that ends in escalation to a human, never an unbounded retry. **Addresses:** Environment (Structural). **In practice.** The outcome is a failure that repairs itself or surfaces legibly, instead of a loop that burns the night retrying one bad idea. Each repair session starts fresh, reads the failing output and what the previous attempts already tried, and is scoped to repair only: no refactors, no new features. The cap is the point: when the attempts run out, the loop halts with the reason recorded, so a person arrives to a diagnosis rather than a mystery. Teams that skip this build the alternative without deciding to, the unbounded retry, which fails silently and expensively. ### 29. Model Tiering Each role in a multi-session workflow is assigned its own model and reasoning budget in one config file: the strongest models on the judgment-heavy roles like planning and review, cheaper ones on execution. The spend is decided per role instead of defaulting everywhere. **Addresses:** Delegation (Structural); Prioritization (Documented). **In practice.** The outcome is spend that follows judgment weight: the expensive reasoning goes where a wrong call costs a whole run, planning and review, and the routine execution runs on a model that costs a fraction as much. Set once in config, the allocation holds across every run and every project instead of being re-decided per session. Model names and prices shift quarterly; the durable part is the decision structure, a per-role budget a team can read, argue about, and change in one place. It is a small file that makes the cost posture legible. ### 30. Judge of Judges A model set over the AI graders, stronger than them or from a different family, that surfaces the calls the bench disputes, plus a random sample of the calls it agrees on, as an arbitration queue for a human. The graders' blind spots become a queue a person actually works. **Addresses:** Evaluation (Documented, Structural). **In practice.** The outcome is graders that stay trustworthy instead of quietly drifting: disagreement on the bench is the cheapest signal that the standard is fuzzy right where it matters, and the random sample catches the calls the whole bench got wrong together. The judge earns its seat by being unlike the graders, a stronger model or a different family or both, because models that share a lineage share blind spots and make the same mistakes with the same confidence. Each arbitrated verdict does double duty, a quality read on the system and fresh calibration for the judges themselves. It is the Humans-make-decisions principle applied to the evaluation layer: machines scale the judging, and a person stays the court of appeal, and at Structural the queue assembles itself instead of being curated by hand. ## Domain overlaps Where investing on two disciplines compounds. Each pairing says what funding both buys, and what funding only one costs. The real work overlaps; investing on both sides beats either alone. ### Within a domain #### Collection and Management Collection gets an artifact in the door; Management keeps it true. The wider Collection reaches, the more surface Management has to hold current, and gathering aggressively without the upkeep just delivers stale context at scale. Fund both and a broad corpus stays trustworthy; fund only the first and you have paid to mislead the agent faster. #### Collection and Disclosure Collection decides what exists for the agent to see; Disclosure decides what actually reaches a given session. A complete corpus is only an asset if the routing can find the one slice a task needs inside it, and one dumped whole is worse than a small, well-aimed one. The value is in the pair: gather widely, then serve narrowly. #### Management and Disclosure Management makes sure an artifact is still true; Disclosure makes sure the agent reads it at the right moment. The leanest, best-routed context is still wrong if it points at a doc that went stale months ago. Currency without routing buries the truth; routing without currency delivers a falsehood on time. #### Capability and Delegation Capability sets what the agent can reach; Delegation sets what it is trusted to decide on its own. Connect a tool but reserve every call about it and you have a locked drawer; hand over a decision for a system the agent cannot touch and you have named a responsibility it has no way to exercise. Buy both in step, and reach and judgment cover the same work; buy one ahead of the other and you pay for a surface no one is cleared to use. #### Delegation and Environment Delegation draws the line on what the agent decides alone; Environment is the harness that catches the call when it goes wrong. How far you can safely move that line depends on what the system behind it catches: solid gates let you delegate more, because a bad decision meets a test failure instead of a customer. Widen the line with no harness and every handed-over call rides on the agent being right; harden the harness but never widen the line and the safety net goes unused. #### Capability and Environment Every tool you connect through Capability raises the stakes Environment has to hold. A wider reach is more the agent can break, and the harness of gates and isolation is what turns that reach into throughput instead of cleanup. Grow the surface faster than the guardrails and each new connection is one more unwatched way to cause damage; grow the guardrails with no new reach and you have hardened a cage around work the agent cannot do anyway. #### Capture and Trace A trace is only ever a reading of what Capture bothered to write down: reconstruct a decision and you are replaying the record, not the run, so reasoning-level Trace demands reasoning-level Capture. Instrument only the prompts and the tool calls and the trail can say what happened but never why, which is the half of an incident that actually costs you. Invest in both and a cold case reopens down to the judgment behind it. Under-fund Capture and Trace becomes a courtroom with no evidence: the event on the record, the reason for it gone. #### Availability and Trace Availability and Trace answer opposite questions of the same record. Availability reads across every run at once while Trace goes vertical on one of them, the full walk of a single case down to the reasoning behind it. You need both postures, because a query that flags a spike in failures cannot tell you why any single one broke, and reconstructing that one broken run cannot tell you whether it is a fluke or a pattern. Fund only breadth and you are watching a trend you cannot diagnose; fund only depth and you are explaining a case you cannot place. #### Capture and Availability Capture writes the record; Availability decides whether anyone can ever ask it anything. A session logged and then sealed away is the worst line on the bill: you paid for the instrumentation and the storage and bought back nothing, because the answer is technically in there and functionally gone. The two only pay off together: capture wide, then make it reachable. Log nothing and there is nothing to ask; log everything into a pit and you have funded a haystack no one will search. #### Evaluation and Prioritization You cannot rank what you have not measured. Evaluation turns a quarter of AI work into numbers a leader can defend, and Prioritization is how those numbers get spent. Measure without ever choosing from what you found and the evidence just sits there, an expensive read nobody steers by. The real failure is the other order: a team that chooses without measuring hands the invest-next call back to the loudest voice in the room, the one habit Prioritization exists to break. #### Prioritization and Convergence Prioritization decides which change earns the team's energy; Convergence gets that change out of one head and into everyone's workflow. Pick the right bet and never converge on it and you have a correct decision quietly decaying unused, re-earned by hand each time someone new needs it. Converge hard with nothing ranking what spreads and you propagate whatever got tried last, noise moving as fast as signal. Held together, they are the difference between a good idea becoming standard practice and a good idea becoming a story about the one engineer who had it figured out. #### Evaluation and Convergence Convergence is a multiplier, and a multiplier only helps if what it multiplies was worth spreading. Evaluation is the check that makes that call: it holds a change against a real standard of good before the team commits to it everywhere. Without that check, Convergence works perfectly on the wrong input, rolling out to every repo the change that merely felt faster, and the better your propagation the wider that mistake travels. The point of running them together is a team that standardizes on what was proven, not on what demoed best last week. ### Across domains #### Management (Context) and Environment (Control) A convention starts in Management as a documented pattern the agent is meant to follow, and crosses into Environment the day you build it into a gate that will not let a change pass without it. It is one rule at two postures: written down it informs and can be ignored, enforced it stops the work until it is met. Leave it in the doc and you are trusting everyone to remember it on a busy afternoon; promote it to a gate with no doc behind it and you get a red build no one can explain. The rule migrates across the line as the stakes climb, so keep the convention current where it lives and enforce only the few that have earned it. #### Delegation (Control) and Trace (Observability) You can only safely hand over a decision you are able to reconstruct later. Delegation is how much you let the agent settle on its own; Trace is what keeps those solo calls something you can still answer for when one goes wrong. Push the line out past what you can trace and every autonomous decision becomes a call with no path back to the reasoning behind it, which is exactly the call an incident or an auditor will make you explain. So the honest ceiling on how far you delegate is how deep you can reconstruct: widen the line and the record together, or the reach you handed over is reach you cannot account for. #### Trace (Observability) and Convergence (Improvement) A reconstruction tells you exactly what went wrong, and by itself changes nothing about what happens next. Trace is the practice that produces the finding: the walk back through a run to the decision that broke it. Convergence is the practice that turns that finding into something the whole team now does differently. Same evidence, two jobs: fund the first alone and you accumulate immaculate postmortems on incidents you are still on course to repeat, because a lesson learned in one investigation is not a lesson the team has adopted until someone makes it spread. #### Capture (Observability) and Evaluation (Improvement) Capture supplies the raw record of what the agent did; Evaluation is the judgment of whether it was any good. The record is the evidence Evaluation reasons from: with nothing captured to compare against, a claimed improvement is just a better-feeling week, and the evals fire on data someone had to record first. But a full log still cannot render the verdict, because good is judged against a standard the record does not contain: the rubric, a senior engineer's read, whether the output served a customer the log never saw. Capture the work without ever judging it and you are archiving activity no one grades; judge with no captured record and you are grading from memory, the exact feeling Evaluation exists to replace. #### Convergence (Improvement) and Environment (Control) The same good practice can travel two ways: Convergence makes it the easy default people opt into, and Environment makes it the gate no run is allowed to skip. Carrot and stick on one standard. Offer only the carrot and the holdouts never come; build only the stick and you enforce a habit no one was persuaded was worth it, which just teaches people to route around the gate. The deeper tie is that these are one risk posture expressed at two scales: the enforced check around a single agent's run and the norm you spread across the whole team are the same call about how much you are willing to trust, made once at the machine and once at the org. Set the posture first, then reach for the carrot where persuasion is enough and the stick where the stakes will not wait for it. #### Availability (Observability) and Prioritization (Improvement) Prioritization runs on the signal it can actually pull, and Availability is what decides whether that signal is reachable at all. A queryable record answers the questions a sound investment call rests on: where the spend is piling up, which failures recur, what a workflow really costs the team. Leave the record unreachable and Prioritization has nothing to read, so it quietly falls back to whoever argues hardest, the very failure mode both disciplines are built to prevent. Make the record queryable and the next bet gets chosen from evidence anyone can check, not from the meeting's center of gravity. #### Capability (Control) and Disclosure (Context) What an agent can actually do in a single session is a smaller thing than everything your org made reachable: it is the tools Capability connected and the slice of context Disclosure served this run, and nothing outside that. A capable agent handed a briefing that never mentions the tool for the job will not reach for it, and perfectly aimed context is inert the moment it depends on a system Capability keeps off the table. Reach and briefing have to describe the same working set. Connect a tool without ever surfacing it and you have paid for an ability the agent is never told it has; serve a workflow built on access it does not have and you have written an instruction it cannot follow. #### Collection (Context) and Capture (Observability) Collection and Capture are one instinct aimed in opposite directions: Collection gathers what an agent needs before a run, Capture records what it did after. They rhyme because the record Capture writes down (the decision made, the reasoning behind it, the transcript of how the work actually went) is exactly the kind of artifact Collection exists to feed the next agent. Instrument every run but never gather from what you logged and a true account of how your team works with AI piles up unread; gather diligently while capturing nothing and you curate only the sources that predate your agents, missing the richest one. The payoff is a loop: the exhaust of one session becomes the context that orients the next. #### Environment (Control) and Evaluation (Improvement) A gate and an eval both stand between a change and the team's trust, and they judge by different lights. Environment is the deterministic check that blocks: the build goes red or green, and red stops the work where it stands. Evaluation is the judged measure that grades whether the change was worth making at all, a question no test suite can answer. The same lint and test run serves both ends: once as the gate a break cannot pass, again as the before-and-after baseline an improvement has to argue from. Ask the gate alone to speak and you ship on green, forgetting a passing build is not the same as a good outcome; keep one check binary and the other honest, and the build tells you it still works while the eval tells you it works better. ## Building an evaluation practice _How a team iterates from flying blind to finding the failures nobody reports, touring all four domains of Working with Robots._ A traditional program that breaks looks broken. An alert lands in Slack, an error bar on a dashboard goes red. An AI assistant that is wrong looks almost exactly like an AI assistant that is right: same tone, same polish, same air of authority. The mistake is invisible until someone who knows the real answer happens to look. And when someone does look, it always ends the same way. The answer feels off, they dig in, and they come away calling the system a liar, whether it lied or just lacked the context to know better. Either way, the trust is gone. **Our working example.** An assistant that answers your customers' support questions: "where's my refund?", "why did my bill go up?" Behind the scenes it pulls their account and applicable company policies, and writes back a friendly, authoritative reply. When it is right, nobody looks twice. When it is wrong, it is a confident, well-written promise founded on nothing, and a customer holds you to it. Evaluation is the discipline of answering "is this system actually doing its job?" systematically and continuously, instead of by spot-checking and hoping. This guide walks that practice in stages, from one person reading transcripts to a system that hunts down its own hidden failures, each stage opening on the general lesson it taught. The ending up front: maturity is not about removing humans. It is about aiming scarce human judgment at exactly the things that need it, and letting AI carry everything else. The stages ahead follow one team, the engineers who built the assistant and own its quality. Their journey is told in the vocabulary of Working with Robots, Go Between's framework for how an engineering org decides where to invest in AI: four domains of work, divided into disciplines, each with an investment tier that describes how deep a team has gone. Every stage ahead closes with a readout of where this team's investments sit. One principle runs under everything that follows. Polish is not evidence, so every stage adds a way to check the output against something outside the system. The framework names this principle fluency-isn't-value, and [the framework itself](https://gobetweenlab.com/working-with-robots) is the place to go deeper. ### General lessons The whole guide in miniature: skim the lessons without the story, or follow one into its stage. - **Stage 0, Flying blind.** Every evaluation practice starts with no way to answer whether the system is doing its job, and for a prototype that is fine. The moment real decisions start riding on the output, shipping and hoping is the biggest risk you have. - **Stage 1, Mechanical checks and human review.** You cannot judge what you did not record, so evaluation starts in Observability. Then cheap, deliberately dumb checks ask whether the machine behaved on every run, and human judgment is only required on the few runs worth further investigation. - **Stage 2, Judgment at scale.** Whether an answer is actually any good is a judgment call, so scale it with an AI grader: write the standard down, put the grader on live traffic immediately, and treat everything it says as noise until its agreement with the human verdicts clears a bar. Even then a person keeps arbitrating where the graders disagree, and each verdict sharpens both the system and its judges. - **Stage 3, Agents propose, humans decide.** To make the whole system better faster, point AI at improving it. Make its restraint structural: it proposes, never applies, and every change clears evaluations the proposer cannot edit, plus a human decision. - **Stage 4, Finding the failures nobody reports.** The most dangerous failures are the ones that look right. Check answers against facts the answers cannot fake, and mine recorded behavior for the dissatisfaction that never reaches a feedback button. Followed to the end, the lessons build a steady state that spans all four domains of the framework. - **Context.** - What the hunting surfaces gets written down: vocabulary and boundaries the agent reads instead of guessing. - **Control.** - Agents propose improvements and humans decide what ships, behind an automatic gate. - **Observability.** - Every run leaves a record, queryable enough to mine for behavior and to reconstruct any single run end to end. - **Improvement.** - Cheap mechanical checks read every run and pre-filter what humans see. - Calibrated graders score quality at scale; a judge over them routes disagreements to the standard-setter for arbitration. - The system hunts its own hidden failures, and human attention is aimed at the runs with the most to teach, never flooded. - An approved improvement becomes team practice everywhere it applies. ### Stages #### Stage 0: Flying blind > Every evaluation practice starts with no way to answer whether the system is doing its job, and for a prototype that is fine. The moment real decisions start riding on the output, shipping and hoping is the biggest risk you have. The starting state is the same everywhere: the system is live, and the only quality signal is complaint. Nobody can say whether it is doing its job except by waiting for someone to notice that it is not. Efficacy is a gut feeling, and nobody stops to check the gut. The system's work is a black box the team pays for monthly, and when someone asks what it did last week, the honest answer is that nobody knows. > **In our example.** The assistant is live and answering customers every day. The only way the team learns an answer was wrong is a customer coming back angrier, quoting the assistant's own reply, and every fix is a scramble. No run is recorded anywhere. - **Human judgment:** The firefighter. Quality problems reach them after a customer has already found one. - **AI labor:** Nothing yet. Every check that happens is a person reading output by hand. - **Framework investments:** Nothing funded yet. Every discipline sits at Exploratory, which is a posture, not a grade: the tier you invest your way out of. - **Techniques:** none yet (manual and ad hoc). #### Stage 1: Mechanical checks and human review > You cannot judge what you did not record, so evaluation starts in Observability. Then cheap, deliberately dumb checks ask whether the machine behaved on every run, and human judgment is only required on the few runs worth further investigation. Evaluation is a discipline of the Improvement domain, and someone on the team will want to start with the discipline itself. The temptation is to go straight to AI graders. But a grader needs something to read and a way to know whether to believe it, and this team has neither yet. Both start with the record, so the journey starts in Observability. Every run has to leave a record: what the system was asked, what it did, what it answered, what it cost. And the record has to land somewhere the team can query, not in a vendor console nobody opens. With the record in place, the first checks are deliberately dumb. Before you judge whether an answer is good, you can cheaply check whether the machine behaved: did it finish at all, or time out and return nothing? Did it do the work its answer claims, or answer without doing it? Pure rules, no model involved, instant and perfectly repeatable. Checks this dumb still catch real failures. > **In our example.** For the assistant, the record is the customer's message, the account and policy it pulled, the reply it sent, and the cost. A telling dumb check is whether a question about an account actually touched the account, or was answered without ever looking it up. Those checks exposed replies invented from nothing, and the failures clustered late in long conversations, so the fix was to cap how much context a conversation carries. A patch, not a cure, but it worked. On their own these checks cannot tell you whether an answer is right. Their real job is to be a pre-filter. Execute them on every run and nobody has to read thousands of transcripts. Instead, one person on the team reads the few hundred the checks flagged. They record a verdict on each with a reason. The reasons matter as much as the verdicts: writing one forces the reviewer to say why the system worked or broke, and read in aggregate they are where the failure patterns first show up. The verdicts become the standard everything later is measured against, one the system cannot generate for itself. The team also drew its first delegation boundary. Machines flag, a person judges. Nobody wrote that rule down yet, but Stage 3 will build it in. - **Human judgment:** The grader. Reviews the flagged runs and sets the standard, one verdict at a time. - **AI labor:** The sifting. Cheap checks read every run and decide which few deserve a person's scarce attention. - **Framework investments:** - **Capture.** The first real investment. Every run leaves a raw record, because nothing downstream works without one. - **Availability.** The record lands where the team can query it. Capture and Availability climb toward Documented before Evaluation has properly begun. - **Evaluation.** Moves from Exploratory to Expert-led. One person carries the read, and the read is real. - **Delegation.** The first boundary is drawn without ceremony: machines flag, a person judges. - **Techniques:** - **Verification Scripts.** Deterministic checks pointed at agent runs: did it finish, did it look up the account, did the plumbing hold. Rules that read every run before a person does. - **Automatic Capture.** Hooks that record each run as it happens, so the record the checks read is complete without anyone choosing to log. - **Database Audit Trail.** An append-only, queryable log of runs and verdicts. The store the review reads from, and the evidence base every later stage builds on. #### Stage 2: Judgment at scale > Whether an answer is actually any good is a judgment call, so scale it with an AI grader: write the standard down, put the grader on live traffic immediately, and treat everything it says as noise until its agreement with the human verdicts clears a bar. Even then a person keeps arbitrating where the graders disagree, and each verdict sharpens both the system and its judges. Plumbing checks cannot say whether an answer was helpful, well-caveated, and plausible. That is a judgment call, and judgment is exactly what does not scale when a human has to make it on every run. This stage has a model grade the output against a written rubric. The trap is obvious the moment you say it out loud. Why would you trust one AI to grade another? You would not, and you should not. So the defining discipline of this stage is calibration. A new grader starts life as an untrusted opinion. Put it on live traffic immediately, because nothing exposes where it is deficient faster, but expect the early scores to be noisy and treat them as exactly that: nothing it says counts toward evaluating the system until it has been checked against the human verdicts from the last stage and its agreement with the expert clears a bar. > **In our example.** The assistant's rubric spells out what a good answer to "why did my bill go up?" has to do: pull the right account, cite the policy it leans on, and promise nothing the policy does not. Its first grader read live traffic the same week and promptly overpraised confident replies that cited no policy at all; nothing it said counted until its scores lined up with the hand review's verdicts. This is what the hand review was building. Those verdicts are the training signal that makes automated judgment trustworthy. Humans are not replaced by the graders; humans are what make the graders worth trusting, and the standard still lives outside the model. Human judgment does not retire when the graders arrive; it moves up. The mechanically flagged runs from Stage 1 still get read, even as their causes get fixed and that pile shrinks. And the graders open a new queue of their own: set a judge over the judges, a stronger model or one from a different family, so its blind spots are not their blind spots. Every disagreement on the bench goes to the standard-setter for arbitration, along with a random sample of the unanimous calls, because the whole bench can still be wrong together. Each arbitrated verdict does double duty, a quality read on the system and fresh calibration that sharpens the graders themselves. For now the team runs that refinement loop by hand; making it run itself is the next stage. > **In our example.** The assistant's two graders kept splitting on hedged replies, the ones that cited the right policy but would not commit to what it meant for the customer's bill. One grader read the hedging as careful caveating, the other as a failure to answer; the standard-setter ruled that an answer which never answers is a failure, and the ruling became a new rubric line that ended the split. The random sample earned its keep too, surfacing a reply both graders had waved through because the policy it cited was real but did not cover the customer's case, the first glimpse of the failure class Stage 4 hunts down. Writing the standard down pays a second dividend: the team has agreed what good means before any single piece of work gets argued about, and a junior engineer can push back on a senior's call by pointing at it. It is also the first glimpse of shared vocabulary the org never had, a gap the journey returns to later. - **Human judgment:** The standard-setter. Their verdicts arbitrate grader disagreements, calibrate the graders, and recalibrate them when the system shifts underneath. - **AI labor:** The judging. One expert's calibrated taste, applied to every run, at no marginal human cost. - **Framework investments:** - **Evaluation.** Climbs from Expert-led to Documented. The rubric writes the expert's standard down, and calibrated graders apply it to every run. Structural waits until the graders themselves are maintained by the system. - **Techniques:** - **Evaluation Harness.** Graders and deterministic checks that score every run against the written standard, so efficacy is measured continuously instead of argued after the fact. - **Judge of Judges.** A stronger model, or one from another family, reads the graders' calls and flags the disputes; the standard-setter arbitrates those plus a random sample, and each ruling recalibrates the graders. - **Database Audit Trail.** The recorded verdicts read back as a calibration set, the proof a new grader agrees with the expert before it counts. #### Stage 3: Agents propose, humans decide > To make the whole system better faster, point AI at improving it. Make its restraint structural: it proposes, never applies, and every change clears evaluations the proposer cannot edit, plus a human decision. By now the team can measure quality well. The next frontier is improving it, and there are a lot of levers: the system's prompts, the tools it is given, the way the work is orchestrated, even the graders' own instructions. Hand-tuning all of that is slow, and it does not scale past the one person who knows where the knobs are. The answer is a system whose job is improving another system, what some teams call a meta harness. It reads the record of how the work has actually been going and drafts candidate changes across every lever, including the standard itself. It emits proposals, never edits. What makes that safe is Control. The meta harness's gates are built in, not left to restraint the proposing agent is trusted to show, and every proposal has to clear two of them before it ships: an automatic evaluation gate, "does it improve the measured scores without breaking anything?", and a human approval. Nothing applies itself, and that rule is deliberately absolute. An auto-applied change is a vibes-based change, and "but I told it not to make mistakes!" is not a gate. The delegation boundary drawn casually back in Stage 1 is now written down and built in: agents propose, humans decide. Building that enforcing machinery is its own journey; the sibling [harness field guide](https://gobetweenlab.com/working-with-robots/field-guides/harness) walks it. > **In our example.** The context cap from Stage 1 was one person's patch, found by hand and applied by hand. This loop is where fixes like it come from now, drafted from the record of the assistant's real runs, shown to a person for review with the measured scores attached, and applied only after their approval. The same stage attacks another human bottleneck: attention. The arbitration queue the standard-setter curated by hand in Stage 2 is now the system's job, and it surfaces the dozen runs that will teach it the most. The case where two graders disagreed, the random sample that keeps the review honest, the run a user reacted to. And when a proposal is approved, it does not land in one engineer's setup and leave when they do. It lands in the shared configuration every run reads. - **Human judgment:** The decider. Approves or rejects proposed changes, and labels a curated few high-value cases instead of a firehose. - **AI labor:** The drafting and the triage: reading everything, proposing the improvements, ranking the mountain down to a dozen. - **Framework investments:** - **Evaluation.** Finishes the climb to Structural. The graders and the standard are now maintained by the improvement loop itself, not one person's hand-tuning. - **Delegation.** The propose-and-decide rule stops being a habit and becomes structure: Documented to Structural, baked into the tooling. - **Environment.** The evaluation gate joins the checks a change must clear before it lands. - **Prioritization.** Lights up for the first time. What deserves attention next is chosen by written criteria, not the loudest voice. - **Convergence.** An approved improvement becomes team practice everywhere it applies, instead of staying in the one setup where it was tried. - **Techniques:** - **Contracts and Prohibitions.** The do-not lines that keep a proposer a proposer: nothing it drafts applies itself. - **Escalation Affordance.** The harness carries every proposal to a human, instead of trusting the agent to ask. - **Prioritization Rubric.** The written criteria that rank candidate changes and pick the handful of runs worth a human's label. - **Config-Driven Deployment.** An approved change propagates from one place to everywhere it applies, so a win ships once and sticks. #### Stage 4: Finding the failures nobody reports > The most dangerous failures are the ones that look right. Check answers against facts the answers cannot fake, and mine recorded behavior for the dissatisfaction that never reaches a feedback button. Calibrated graders still miss a whole class of failure: the wrong answer that looks good. A confident, plausible, well-formatted answer that happens to be wrong sails past a plausibility grader the same way it sails past any reader who does not know the business deeply enough for their intuition to trip. An attentive user might feel something is off, but the feeling never reaches the team: no error, no complaint, no thumbs-down. Just a wrong answer someone is now acting on. The first counter is deterministic: find a fact outside the answer that can contradict it. "Is this grounded in reality?" breaks down into checks that run even when eyeballing the answer tells you nothing. A failure class is real when a check catches it on a real run, not when a reader gets a suspicion. Run those detectors across the whole corpus and they surface the silent failures, the ones no reviewer would ever have been pointed at. > **In our example.** For the assistant, the grounded-in-reality checks come straight from the sources. Re-read the policy the reply cites and compare what it actually says with what the reply claims. Check the reply against the account: does the refund it promises match an order that exists, does it contradict what the same customer was told a week ago. Ask whether what the reply needed, an order to refund or a policy that covers the case, even existed, or whether the assistant quietly filled the gap. The second counter starts from an uncomfortable fact about feedback: the thumbs-down button lies by omission. Most dissatisfaction never touches it. It shows up as behavior in the conversations themselves. The user pushes back in words, retries the question, or abandons the thread. Mining the record for that behavior yields far more signal than the button ever will. Run it on a cadence so the result is a trend, and a trend only means something if every month is scored by the same definition of dissatisfaction: when the team sharpens the definition, it rescores the old months under the new one. That is the Stage 1 investment in Observability paying out a second time, because you can only re-mine what you captured. > **In our example.** In the assistant's transcripts the sharpest tell is a customer asking, mid-conversation, where the policy says that, and never raising it anywhere the team would see. That mining surfaced the failures the detectors had no rule for yet. The team found more of them through mined behavior and investigation than through any grader. The hunting turns up more than wrong answers; it turns up work for disciplines well outside Evaluation. Each find is tagged by what it needs and routed to the right expert. A term the system keeps reading differently from run to run is missing shared vocabulary, a Context gap closed by writing the term down where the agent reads it. A boundary that keeps getting crossed needs a rule. And each detector that proves itself gets ported back into the cheap, always-on checks from Stage 1, where it runs forever. - **Human judgment:** The expert. Makes the subtle correctness calls, the ones that require knowing the business, not just reading the transcript. - **AI labor:** The hunting. Detectors and miners rank a haystack of thousands down to the needles worth an expert's time, including the ones nobody knew to look for. - **Framework investments:** - **Evaluation.** Runs at Structural as a steady state, and starts generating work for disciplines outside its own domain. - **Trace.** The subtle calls need a single run reconstructed end to end, with the reasoning alongside the decisions. - **Collection.** Evaluation surfaces the missing shared vocabulary; gathering it and writing it where the agent reads it is Collection's work, a Context investment. - **Environment.** The proven detectors come home as inline checks: the gate a change must clear just grew a new layer. - **Techniques:** - **Database Audit Trail.** The mining's raw material. Months of recorded conversations, queryable enough to read for behavior instead of button presses. - **Domain Knowledge.** Where a term that needs defining lands. Canonical vocabulary the agent reads instead of guessing. - **Pattern Libraries.** Where a boundary that needs a rule lands. The ruling written down once, with the anti-patterns to avoid. - **Verification Scripts.** The port-back target. A detector proven in the hunt reruns forever as a cheap deterministic check. ### The human's responsibility climbs Step back from the stages and one thing is constant: the human never leaves the loop. The human's job changes. At every stage, AI absorbed the work below and the human moved up to something only a person can do. The responsibilities form a ladder: grader, standard-setter, decider, expert. Each is harder to replace than the one below it, and the journey shows it from the bottom. Stage 2 automated the grader's work, Stage 3 drafts what the standard-setter used to hand-tune, and no stage touched the decider or the expert. The decider stays human because of the rule built in Stage 3. Nothing applies itself, so every change that ships still passes through a person. The expert stays human because the hardest correctness calls rest on knowing the business, and that knowledge is not in the system: a model can read every conversation ever recorded and still not know which promises the company cannot afford to keep. That is what maturity actually means here. It was never automation replacing judgment. It is automation concentrating judgment onto the decisions that need a person, and multiplying each of those decisions across the entire system. The one true ladder here is the human's. The stages themselves are a loop that runs continuously, the later stages feeding the earlier ones: the detectors Stage 4 invented to catch plausible-but-wrong answers were ported back into the always-on pre-filter from Stage 1, a discovery made in the last stage becoming a permanent part of the cheapest, earliest one. That is the framework's diverge-then-converge principle at practice scale, exploring at the edge and folding what proves out back into shared practice. In steady state all of it runs at once, and no domain sits downstream of another. ## Building a harness _How a team iterates from taking their agent's work on faith to building a system that doesn't require trust, touring all four domains of Working with Robots._ Hand an agent a bug to fix, some data to analyze, or an incident to writeup, and it already does a convincing job. Execution is the part that looks solved. The question that follows is how to hand it more, and the default answer is to ask more of the agent itself: a bigger task, more access, less oversight. One prompt that says build the whole thing and make no mistakes. Sometimes that works, which is why every team tries it. But eventually the agent's context window fills and the middle of the work goes missing. Splitting the job across sub-agents trades one failure for another: they cooperate imperfectly, and a single hallucinated sub-step quietly throws off the run. So a person stays camped beside the agent, redirecting by hand, and every increase in ambition is paid for in more manual intervention. The way to keep the ambition without the price is a harness: a mix of agentic inference and deterministic code. Agent sessions run where inference is genuinely needed; ordinary code runs before, between, and after. The harness routes work, validates output, feeds checked results back in, and executes every step that needs no judgment at all. Everything added to a harness is capability you can engineer. The deterministic parts are tested like any other code, the inference parts are evaluated against a standard, and the line between them is a decision you own: what to infer and what to simply execute. How much an org can hand its agents, and how sure it can be of what comes back, is a property of the harness, not of the model. This guide follows a team whose agent writes software. They ended up with everything the default story promises: real features shipping overnight and unattended. They got there by giving the agent less. The inference multiplied into more sessions with narrower jobs, while capabilities left the agent one by one and came back as ordinary code in the harness. Building it was how they learned, stage by stage, where inference belongs and where plain code will do, and how to structure and orchestrate the inference itself. **Our working example.** A coding agent in a harness that picks up planned issues from a three-person team's tracker, writes the code, opens the PR, and merges it. When a run fails overnight, nobody is there to catch it; the team finds out in the morning. Everything it builds is proof-of-concept software: real products, no production users. That scope was a person's decision, made before the first run. Harness engineering is the practice of increasing what agents can take on while keeping the quality of what they produce demonstrably high. This guide walks that practice in stages, each opening on the general lesson the stage taught. The ending up front: the team never came to trust the agent more. They built a system that does not require trust. The stages ahead are told in the vocabulary of Working with Robots, Go Between's framework for how an engineering org decides where to invest in AI: four domains of work, divided into disciplines, each with an investment tier that describes how deep a team has gone. Every stage ahead closes with a readout of where this team's investments sit. One principle runs under everything that follows. What comes out of an agent is a function of the system around it and of what is handed to it. The framework names this principle input-is-the-lever, and [the framework itself](https://gobetweenlab.com/working-with-robots) is the place to go deeper. ### General lessons The whole guide in miniature: skim the six lessons without the story, or follow one into the stage that taught it. - **Stage 0, A loop you run by hand.** Every harness begins as a loop that stops when its person does, and is defined by the act it performs without waiting for a person. Choose that act before the first run, and fence it: bound what it may touch so the worst case stays small. - **Stage 1, A shared source of truth.** Humans and machines should keep one set of books. Whatever record your harness runs on belongs on one surface both can read and write, and every handoff between person and machine should happen through it. - **Stage 2, Verify, don't trust.** An agent's claim to have done the work is not evidence it did. Never let the agent be the reporter of record: have it request side effects, let deterministic code perform them, and check what happened by re-running the harness's own checks, not by reading the agent's summary. - **Stage 3, Write the contract down.** Write down which rules the system enforces and which are only requests in a prompt, then close the gap. A boundary that is only implicit is discovered by the next incident. - **Stage 4, The harness leaves home.** Run the harness in a second home, a project it has never seen, and it tells you what it actually is. The context agents work from belongs to the home; the machinery that injects it belongs to the harness, and the interface you are forced to declare is exactly that boundary. - **Stage 5, Built for absence.** A harness is proven on its bad days. List what can break while nobody is watching, and give each failure its machinery: bounded repair attempts, stops that record their reason, a record to read in the morning. Followed to the end, the lessons build a steady state that spans all four domains of the framework. - **Context.** - Sessions are fed the issue, the index file the task needs, and the handoff state, assembled from records that survive between sessions. - The system maintains its own memory of the codebase, a record people read but never have to keep. - **Control.** - Side effects are requested by agents and performed by deterministic code, and every enforced rule is also written where people can read it. - The workbench is verified before work starts and healed when broken; every limit ends in a stop with its reason recorded. - **Observability.** - Every run leaves a typed trail in the tracker and a queryable database: people read both, and so do the machines. - **Improvement.** - Judging the harness stayed deliberately Expert-led, and what proved out propagated to every project at install time. ### Stages #### Stage 0: A loop you run by hand > Every harness begins as a loop that stops when its person does, and is defined by the act it performs without waiting for a person. Choose that act before the first run, and fence it: bound what it may touch so the worst case stays small. Start smaller than feels serious. A harness begins the day the team stops redirecting an agent by hand and writes that hand-work down as code. The machine does the work; the person does everything around the work: aiming the loop, feeding it, checking it, keeping the books. > **In our example.** The earliest version is a few hundred lines of code that run one feature per invocation: an implementation session, then a fresh-context review session that critiques, fixes, and merges. The person specs every feature and its verification steps, marks the state by hand when a session fails, and keeps the books in two flat files, a JSON truth file for the machine and a prose mirror for people. The products it built from the first week were disposable, iterated past and deleted; the harness was the thing the team kept. > > One detail breaks the default story, in which the riskiest acts are handed over last, once the agent has earned them. The review session already pushes its branch, opens the PR, and merges it. The merge button was handed over on day one, and the fence around it is what made that defensible: proof-of-concept software with no production users, so the worst case is a bad merge in a codebase nobody depends on yet. The team would not take this stance on an app with production use. That delegated act, the consequential thing the system does with no person in the loop, is what makes this a harness; a system where every consequential action waits for sign-off is a tool. The assistant in the sibling [evaluations field guide](https://gobetweenlab.com/working-with-robots/field-guides/evaluations) holds an act of its own, answering a customer unsupervised. Swap one act for the other and the stages ahead barely change. Every delegated act needs its fence, and the biggest delegation decision is which class of work the loop may touch at all. That decision stays with a person. So does the other one: what to build is never handed over. That is the framework's humans-make-decisions principle at its earliest and its largest. - **Human judgment:** The operator. Runs the loop one feature at a time, keeps the books by hand, and marks the state when a session fails. - **AI labor:** The feature itself: implementation and a fresh-eyes review, merge included. - **Framework investments:** - **Delegation.** Both decisions visible from day one. The review session merges its own PR, and what to build never leaves the person. Expert-led: one person's rules. - **Environment.** A reset script returns the workbench to a known state; the test suite is the only gate. - **Capture.** An append-only log that exists because one person keeps it: Expert-led. - **Techniques:** - **Progress Trackers.** The JSON state file the harness treats as authoritative: structured state resists quiet rewriting better than prose. - **Verification Scripts.** The full test run as the one gate a feature must clear before the merge. #### Stage 1: A shared source of truth > Humans and machines should keep one set of books. Whatever record your harness runs on belongs on one surface both can read and write, and every handoff between person and machine should happen through it. Every harness runs on a record. A delivery harness works from a queue of planned work; the evaluations guide's practice runs on a log of every customer conversation its assistant has handled. Yours runs on something else. The stage-zero arrangement, machine state in one file and a human-readable copy in another, is double bookkeeping. The copies drift: an agent told to keep both current will sometimes update only one. Collapse them into one record that humans and machines both read and write, kept where the team already looks. A shared record does three jobs. The machine's work becomes visible for free, because the state is the status and nobody writes status updates. The machine gains a surface it can be held to: a transition either happened on the record or it did not. And the record becomes the counter where person and machine trade work. People put in what the machine should do, and read out what it did. Every handoff, in either direction, happens through it. > **In our example.** The flat files collapsed into the team's tracker: a small workflow of states the harness reads and writes. The loop immediately stopped needing a person between features. It drains the backlog on its own, and the person's job compresses to the two ends, filling the board and reading the results. Filling the record is its own craft, a planning conversation that turns intent into scoped issues, and it is the only place a harness takes work from. - **Human judgment:** The backlog author. Fills the tracker through a human-led planning conversation, presses start, and watches. - **AI labor:** Finding its own next piece of work: the loop drains the backlog without being re-aimed. - **Framework investments:** - **Delegation.** What to do next is read from the board: the first responsibility delegated to the system rather than the session. - **Availability.** State becomes visible where the team already looks. - **Collection.** The issue becomes the context feed. Every session opens on the ticket pulled live. - **Techniques:** - **Progress Trackers.** The state file's shape lifted into the tracker's workflow states, now enforced by machinery instead of kept by habit. - **Source Connectors.** The live integration that makes the tracker readable and writable from inside the loop: the board is the state, not a copy. #### Stage 2: Verify, don't trust > An agent's claim to have done the work is not evidence it did. Never let the agent be the reporter of record: have it request side effects, let deterministic code perform them, and check what happened by re-running the harness's own checks, not by reading the agent's summary. A harness can run unattended long before it can be trusted to. The gap is made of claims: the agent reports that it ran its checks, performed its side effects, updated the record, and it is sometimes wrong, confidently. The machinery re-runs the checks itself and reads the exit codes. The agent's account of what happened is never the gate again. The counterintuitive half is what to do with the agent's tools: not extending the agent's reach as trust grows, but reversing it. The agent stops executing side effects entirely. It requests them in a structured form its output carries, and deterministic harness code performs them. The immediate payoff is that a side effect performed by code is an observation, not a report. A prohibition becomes a fact: "may not touch the PR" is a request while the agent holds the tools, and an impossibility once it no longer does. The mechanical layer becomes engineerable, too. Retries, race guards, and idempotent re-runs are fixed once in ordinary code rather than in prompts, where guidance is only ever followed sometimes. > **In our example.** The gap arrived within days of the first overnight merges, the team's own notes reading "We trust agent claims that work was completed." The whole reversal took one week, the same week the harness's unsupervised scope grew most. The record came with it: a plan posted to the issue before any code, a typed, machine-readable comment from every session, cost metered per issue, a queryable database underneath. None of it depends on anyone remembering to keep it. Every capability left inside the agent is paid for in trust on every run; every capability moved into the harness is paid for once, in engineering. That is the input-is-the-lever principle, applied to actions instead of information. - **Human judgment:** The mechanic. Fixes the harness between the agent's overnight PRs, and leaves the happy path to the machine. - **AI labor:** The whole delivery path, executed rather than claimed: checks re-run by the machine, side effects performed by deterministic code on the agent's signal. - **Framework investments:** - **Environment.** The verification gate climbs to Structural. The checks fire inside the machinery on every run. - **Delegation.** The agent requests side effects, software performs them. Unsupervised scope went up the same week the agent's direct reach went down. - **Capture.** From one person's prose log to a record nothing can skip: typed comments on the issue, a database underneath. Structural. - **Techniques:** - **Intent Markers.** The agent asks for each side effect through a typed marker in its output; the harness performs the action in deterministic code. - **Verification Scripts.** Lint, tests, and end-to-end checks re-run by the harness after every session, exit codes as the gate. - **Database Audit Trail.** Every message, tool call, and harness event is written to an append-only local database. #### Stage 3: Write the contract down > Write down which rules the system enforces and which are only requests in a prompt, then close the gap. A boundary that is only implicit is discovered by the next incident. Enforcement accumulates faster than anyone writes it down. By now, what a harness enforces versus what it merely asks for is implicit, scattered across prompts and harness code. So write the contract: one document a person can read, naming the roles the machine plays, what each may and may not do, and which rules are backed by code. Its standing rule holds anywhere. If a rule is enforced in code, it is a hard requirement; if it lives only in a prompt, it is guidance the agent may or may not follow. The inform-versus-enforce boundary is written down first, then built. Each role deserves a tool surface chosen on purpose, and what a role is denied matters as much as what it is granted. The most deliberate denial is context: a reviewer that cannot see the implementor's reasoning cannot inherit the implementor's assumptions. A machine can only hand work between its own sessions through state that survives them. Nothing in a context window does. > **In our example.** Four sessions run the loop: a planner that drafts the issue's plan and may not write files; an implementor and a reviewer that may not touch git, the PR, or the board; and a fixer, a narrow session dispatched when a check fails, that may not refactor or add features. The reviewer is the fresh-context skeptic, denied the implementor's context on purpose. > > The harness watches a session's context usage and ends the session before the window fills. The successor is briefed from the issue's typed comments, which say what is finished and what is pending, with no person writing the briefing. The shared record from Stage 1 and the typed trail from Stage 2 make the handoff possible at all. - **Human judgment:** The rule-writer. Writes the contract and decides which of its rules the harness will also enforce in code. - **AI labor:** Its own working memory: the machine ends a session running low on context and hands the successor the state. - **Framework investments:** - **Delegation.** The boundary becomes Documented in one place, each written rule paired with Structural enforcement: written where people read it, enforced where the machine acts. - **Capability.** Each session's tool surface is decided and written down. No session holds a tool its contract does not grant. - **Disclosure.** A fresh session is fed exactly the state it needs, assembled by the machine from the record. - **Techniques:** - **Contracts and Prohibitions.** One document naming each session's entry and exit states, required outputs, and do-not lines, with the code-enforced rules marked. - **Session Prompts.** The harness detects a session nearing its context ceiling and writes the fresh session's briefing itself from the typed comments. - **Bounded Fix Loop.** On a failed check, a narrow repair session carries the failure output and every prior attempt. The loop is capped at a fixed number of cycles and ends in a human's hands. #### Stage 4: The harness leaves home > Run the harness in a second home, a project it has never seen, and it tells you what it actually is. The context agents work from belongs to the home; the machinery that injects it belongs to the harness, and the interface you are forced to declare is exactly that boundary. The tell that a harness is ready to leave home is fixes ferried by hand between the copies of it living inside each project. Leaving home forces the system to declare what it needs from any host, and to surrender everything it had merely gotten used to. The second home does not have to be a codebase: for the evaluations guide's assistant, it is a second product's customers and policies. And a harness that will only ever have one home still profits from asking what it would need from one, because the interface it would have to declare measures what it has coupled to. > **In our example.** The harness lived vendored inside one project template, four commits in its history titled some variant of "backport." The extraction itself took a single day: the harness refactored into its own standalone system, its prompts turned into templates, the vendored original deleted. From then on, an install script drops it into any host project. > > What it truly needs turned out to be small: five executables the host presents for init, dev, lint, test, and end-to-end, each answering with an exit code, plus a handful of conventions. A preflight check verifies the interface before any session runs, so a bad workbench fails fast instead of failing weird. Stack knowledge left the prompts and landed in each host's own orientation files. The harness stopped knowing what framework it is driving, and kept the machinery that reads the right file into every session. That is the line extraction draws. A harness never runs in a vacuum; it runs primed with its home's context. The context belongs to the home, and the mechanism that injects it belongs to the harness. Structural investment survives people turnover; extraction makes a harness survive project turnover. Even institutional memory moves into the machinery, as an agent-maintained index of what has been built, which humans consult but never have to keep. - **Human judgment:** The installer. Sets up each host's side of the interface: the check scripts, the orientation docs, the conventions. - **AI labor:** Knowing the codebase: the index is written by the machine and read by the machine; humans consult it instead of maintaining it. - **Framework investments:** - **Environment.** Verification points at the workbench itself. The host interface is checked before any session runs. - **Management.** The index is kept current by a hook in the harness's own lifecycle, and the updater is itself an agent session: nobody's chore. - **Convergence.** An improvement made once reaches every host at install time. - **Techniques:** - **Orientation Docs.** The host's own orientation file, read at the start of every session: stack knowledge lives with the stack. - **Index Files.** An agent-readable index of what has been built, routing a session from its task to the one file it needs. - **Config-Driven Deployment.** The install script re-copies the harness into a host, so one repo's improvements ship everywhere on the next install. - **Model Tiering.** Each session type gets its own model and reasoning budget in one config file: the strongest models on planning and review, a cheaper one on execution. #### Stage 5: Built for absence > A harness is proven on its bad days. List what can break while nobody is watching, and give each failure its machinery: bounded repair attempts, stops that record their reason, a record to read in the morning. Budget a closing stretch of work that exists only for the bad days. None of it adds capability; all of it teaches the machine to have its bad day alone. > **In our example.** The closing stretch reads like a catalogue of refusals to trust externally held state: do not believe the tracker until the transition is observed, notice when a human cancels an issue mid-run, never resume an overflowed session, retry a flaky network handshake instead of crashing at two in the morning. A broken main heals itself with up to three repair dispatches before the loop stops to ask. The record laid down in Stage 2 is what absence runs on. Not every discipline needs the climb; a deliberate stop is an investment decision too. > **In our example.** The one investment the team deliberately never automated is judging the harness itself: it stayed Expert-led the entire journey, expert eyes on the record and no written benchmark. For a three-person team's internal tooling, with the record making the expert's read cheap, Expert-led was the right tier. The map-not-a-scorecard principle in practice: a tier says where you are, not how you rank. Evaluation is the discipline [the evaluations field guide](https://gobetweenlab.com/working-with-robots/field-guides/evaluations) climbs all the way to Structural. > **In our example.** Seven months in, the harness's own development goes quiet. The proof comes a month later, when a wholly new product ships through it, run by someone who did not build it. The tally: six features, twenty-three PRs, roughly twenty-two hours of wall clock, zero human touches mid-run. By this final run, a planned issue lands with its scope fenced, its acceptance criteria itemized, and its references pointing at the exact artifacts a session should read. And the run went unattended only because people had done the scoping in advance. - **Human judgment:** Absent. Before the run, planning the backlog; after it, reading the record; during it, nothing. - **AI labor:** The bad day: broken main, state races, cancelled issues, network failures, its own stuck sessions. - **Framework investments:** - **Environment.** Structural as a steady state. The workbench is checked and healed before work begins, and every limit ends in a recorded stop. - **Evaluation.** Deliberately Expert-led from first stage to last: no written standard, no benchmark. The record made the expert's read cheaper; the standard stayed in their heads. - **Delegation.** The closing readout. Unsupervised scope at its maximum, the agent's direct hold on side effects at its minimum. - **Techniques:** - **Escalation Affordance.** The harness counts the attempts, stops the loop, and records why, so a stuck run surfaces as a request a person can act on. - **Database Audit Trail.** One record read three ways: a live monitor, the expert's after-run queries, and the agents' own lookups. ### What steady state can do Step back and the human's seat climbs a ladder as the machine absorbs the work below: operator, backlog author, mechanic, rule-writer, installer, and finally nobody inside the loop at all. The history runs opposite to the story the guide opened on. The most dangerous action was delegated on day one, the agent's direct hold on side effects only ever shrank, and what grew for seven months was the structure around it. The team never came to trust the agent more; they no longer needed to. The structure alone never bought the unattended hours. The final run needed both: a harness that let the hours pass unattended, and a scoped, criteria-fenced backlog that made them productive. A perfect harness fed vague intent ships vague software overnight. What comes out is a function of the system and of what was handed to it. The second factor is its own practice, and it deserves a guide of its own. The definition of done is neither a finished feature list nor a system that must be worked on forever. Development went quiet at month seven; the run a month later shipped a whole product anyway. A harness is done enough when its development stopping is not an event: the improvement loop idles, small flaws persist because they cost nothing, and the system keeps delivering. To place your own team, start with the question this journey turns on: what is your system's merge? Name the delegated act, the thing your system does with nobody watching, and the fence that makes its worst case survivable. Then ask what it would take to run your harness in a second home. Whatever that answer enumerates is your real interface; the rest is what your system has merely gotten used to. From there, read the tiers of [Delegation](https://gobetweenlab.com/working-with-robots/control/delegation) and [Environment](https://gobetweenlab.com/working-with-robots/control/environment) honestly. This guide walked one path; yours will hold a different act and touch the same disciplines in a different order.