Harness Engineering: How to Tell Whether Your Agent Harness Actually Works

Coding agents can inspect repositories, edit files, run commands, create pull requests, and keep working across dozens of steps. For engineering leaders, that changes what needs evaluating.

Agent performance depends on the model, the harness, and the environment in which it runs.

In Anthropic's infrastructure experiments, changing resource limits shifted Terminal-Bench scores by six percentage points while the model, harness, and tasks stayed fixed. Separately, SpatialBench reported a 23.3-percentage-point improvement from changing the harness around Claude Opus 4.5 on spatial biology tasks.

These findings show why benchmark comparisons need to account for harness design and infrastructure settings. The reported gains are specific to the tasks and configurations tested.

A useful harness enforces access limits, preserves verification evidence, and helps the agent recover from failures within a defined budget. More attempts, tokens, or pull requests tell you how much activity occurred. To judge whether the setup works, you need to follow those changes through validation, review, and delivery.

In this guide, we'll cover:

• What harness engineering is and how it differs from agent engineering
• Six signs your agent harness is working
• When to change a harness, and how to test whether a change worked
• Common harness failure modes, and how to connect agent results to delivery metrics

Let's get started!

What Is Harness Engineering?

Harness engineering is the practice of designing, testing, and improving the system around an AI model so an agent can complete tasks under defined controls. That system, the agent harness, manages context, tool access, execution, verification, recovery, and human intervention.

What Is an Agent Harness?

An agent harness is the runtime layer around a model that manages task specifications, context selection, tool access, permissions, memory, verification, tracing, recovery, and intervention rules.

It gives an AI agent a way to act on a workspace and handle the results. Its control surfaces are the settings and mechanisms you can change, such as:

• Tool interfaces
• Approval rules
• Context-loading policies
• Stopping conditions

Context engineering determines which information reaches the model and when. Prompt engineering shapes the instructions the model receives. Both contribute to harness design.

The practice developed before the terminology settled. Anthropic described a long-running harness in November 2025. Mitchell Hashimoto's February 2026 post named his habit of improving the environment after agent mistakes “harness engineering.”

OpenAI's February 2026 Codex write-up and Anthropic's subsequent work gave the practice broader industry attention. Birgitta Böckeler's framework on Martin Fowler's site then organized user-controlled harnesses around guides that steer work before action and sensors that check the results.

Anthropic calls the Claude Agent SDK a general-purpose agent harness. For your team, the relevant harness system also includes repository-specific instructions, checks, and access rules that you maintain around the coding tool.

So, that leads us to our next point...

What Is the Difference Between Harness Engineering and Agent Engineering?

Agent engineering is the broader work of building and operating agents, including planning, tool use, and execution. Harness engineering focuses on the context, controls, verification, and recovery mechanisms that support that work.

The boundary isn’t always clear.

An agent builder may design a repository-search tool, while harness work determines which repositories it can search, how failures are recorded, and what happens when access is denied. Both affect what the agent changes and how its work is checked.

The distinction matters when you evaluate vendor claims. So, ask which controls its product provides and which your team must maintain, including repository instructions, permissions, validation checks, and escalation rules.

Why Harness Engineering Matters for Software Teams

Harness engineering matters because coding agents need working environments, enforceable limits, and reliable checks before their output can contribute to team delivery.

DORA's 2024 research associated AI adoption with individual productivity benefits alongside lower delivery throughput and stability. Its 2025 findings, discussed again in March 2026, associated adoption with higher throughput and continued delivery instability. Individual gains therefore need checking against team results.

These studies did not isolate harness quality as the cause. However, for engineering leaders evaluating AI adoption, harness controls affect several operational risks:

• Agents modify real workspaces. Enforced permissions limit which files, repositories, and services an agent can read or change. In other words, these permissions determine whether a mistake affects a temporary branch, a shared repository, or a production service.
• Execution conditions affect performance. Missing dependencies, inaccessible tools, and incomplete verification can stop a capable model from finishing.
• The same model behaves differently across configurations. Instructions, available context, and feedback loops change how it approaches work.
• Poor controls make output expensive to handle. Repeated attempts and oversized changes consume compute and reviewer attention.
• More agent activity can increase the volume of work awaiting review. Track how much work meets acceptance criteria and whether it requires corrections after merge.

As we said above, the question is whether the model, harness, and environment can produce a change that is correct, reviewable, safe, and worth the cost. Producing a patch answers only part of that question.

So, how do you know if it's working? Let's cover that next...

How to Evaluate Agent Harness Performance: 6 Signs It’s Working

At Axify, we recommend grouping tasks by team, task type, and harness version, then comparing configurations on similar work. Use a consistent follow-up window to capture rework. A four-week intake period and 14-day follow-up are examples; extend them when tasks or defects take longer to resolve.

Evaluate each configuration across the following six dimensions.

1. It Improves Accepted Work

Start with all tasks assigned to the agent, including attempts that produced no PR. For each task, record:

• Whether the output met acceptance criteria
• Whether a PR was submitted and merged
• Whether it was rejected (and if yes, why)
• What corrective work was required

Define acceptance separately from merging.

An approved PR may remain unmerged because of integration or scheduling decisions. Report accepted tasks against all attempted tasks, and accepted and merged PRs against submitted PRs. Show pending reviews separately and apply consistent rules for excluding experiments and duplicates.

This distinction matters in practice. In METR's 2026 maintainer-review study, four maintainers assessed 296 AI-generated pull requests. After normalization against a human-patch baseline, maintainer acceptance scores were 24.2 percentage points lower than automated benchmark scores.

Maintainers assessed static patches without giving agents a chance to revise them in response to feedback. Still, the result supports measuring automated test success and maintainer acceptance separately.

That’s why we advise you to:

• Compare acceptance and rework rates with the baseline for the same task family.
• Report corrections before acceptance separately from verified fixes after merge.
• Review rejection reasons and follow-up fixes to identify missing context, unclear requirements, or unsuitable task scope.

2. It Produces Verifiable Outputs

Every completed run should leave evidence you can inspect. For code changes, that means the diff, relevant test results, validation logs, reproduction steps, and an explanation of any failed or skipped checks.

Report runs that passed every required check as a share of all attempted runs. Show failed, skipped, and unavailable checks separately so incomplete validation cannot be counted as a successful result.

Harness-Bench records final artifacts, execution traces, usage statistics, and validator outputs. That combination is useful because it lets evaluators examine both the result and how it was produced.

Let’s take the example of a customer-record import feature. In this case, you should verify valid records, malformed data, duplicate entries, and recovery after an interrupted import. A successful build does not establish that these behaviors work correctly.

Reviewers should be able to reproduce the checks against the exact revision submitted for review. “All tests passed” without the commands, results, or revision leaves reviewers unable to confirm what was checked.

3. It Keeps Reasoning Connected to Tool Feedback

Inspect whether the agent uses tool results to choose its next action. Look for:

• Repeated failing commands
• Nonexistent file paths
• Ignored test failures
• Unnecessary actions after success
• Edits unsupported by the observed workspace state

The Harness-Bench authors describe this as a breakdown in execution alignment: plausible reasoning becomes disconnected from tool feedback, workspace state, evidence, or the requirements used to verify the output.

Within each task family:

• Measure the share of tool failures followed by successful recovery or appropriate escalation.
• Inspect unresolved failures and compare required-check coverage so fewer recorded errors cannot hide skipped validation.

For example, if a database operation is denied by policy, the agent should stop or request authorization. Enforce the restriction across every access route, including shell tools and Model Context Protocol (MCP) servers, so the same operation cannot proceed through another interface.

Base this review on recorded actions, tool results, and workspace changes. Use the agent’s explanation as supporting context and verify it against the execution trace.

4. It Reduces Review Burden

1. Use PR timestamps to separate the wait from ready-for-review to first review from the time between first review and approval.
2. Then sample actual reviewer effort through a short time log or survey, since elapsed time also includes queues and author response delays.
3. For the same cohort, examine comment counts, reviewer confidence, senior intervention, and PR size and clarity.
4. Ask reviewers which corrections they repeatedly make. A short sequence diagram can help explain a change that crosses services. But an unrelated repository-wide cleanup makes that same PR harder to assess.

The 2026 study Why Are Agentic Pull Requests Merged or Rejected? found observable human intervention in 15.4% of 364 sampled merged agent PRs. That included feedback-and-revision loops and reviewer-applied commits.

Note: The researchers did not measure review time or test whether a harness change reduced it. This finding supports measuring human intervention alongside merge outcomes.

So, look for lower reviewer effort per accepted task while validation coverage and rework remain stable. Treat comment counts as supporting evidence (fewer comments alone can mean reviewers had less time or trusted the output too quickly). Require review by the appropriate technical owner for architectural decisions, high-risk behavior, and unclear requirements.

5. It Controls Cost Per Useful Outcome

1. Calculate cost per accepted task by dividing the total cost of all attempts in the cohort by the number of tasks that meet acceptance criteria.
2. Link usage records to task and PR identifiers, and include failed attempts and retries in the total.
3. Track token consumption alongside cost per accepted task to investigate expensive runs.
4. Separate input, output, and cached-token categories, and apply the relevant provider’s billing rules.
5. Also record expensive failures and the model used for each task type. Compare similar work, since a documentation change and a database migration require different effort.

The Harness-Bench report above found an average consumption of 68,700 tokens for NanoBot and 175,100 for NullClaw, with composite scores of 76.2 and 64.4, respectively. These results were averaged across 106 tasks and eight model backends. The higher-consumption harness scored lower in this comparison.

Note: These composite scores cover multiple agent workflow categories and do not measure coding-task acceptance or production delivery.

Still, that same outcome-based view should carry into the full cost calculation.

• Include API cost, execution infrastructure, and paid supporting tools.
• Report reviewer and remediation labor separately or convert recorded time using an agreed rate.
• Reconcile subscriptions and metered charges to avoid double-counting.

Retain a configuration when it lowers total cost per accepted task without increasing defects or human repair effort. Include reviewer and remediation costs when assessing whether cheaper runs produce a net saving.

6. It Improves Delivery Flow

Accepted changes still have to move through delivery. For the same team and review period, issue history, PR events, deployment records, and incident links show what happens after merge.

Each metric captures a different part of that path. Work-item cycle time runs from your agreed work-start state to completion. In Axify, PR cycle time runs from the first commit on a branch to merge. Work in progress (WIP) counts started but unfinished items, while throughput counts completed items per period.

Together with deployment frequency, change failure rate, and work waiting in review or QA, these metrics show where progress slows.

Link incidents to the responsible changes and deployments before classifying them as related to agent-assisted work. A deployment containing agent-generated code is not sufficient evidence that this code caused the incident.

The need to look beyond merge shows up in the September 2026 preprint Who Finishes the Job?. It compared 6,774 agent PRs with 5,044 human PRs from the same repositories and period. Merged agent PRs had 1.62 times the odds of receiving a verified follow-up fix. This observational finding concerns follow-up fixes, though it does not establish incident rates or a harness effect.

At Axify, we connect AI usage and cost with DORA and flow metrics. At the same time, value stream mapping (VSM) shows where work waits.

Compare throughput with rework and waiting time across the delivery process. Shorter PR cycles alongside growing QA queues may indicate that faster development is increasing pressure downstream. Check task mix, arrival rates, and QA capacity before expanding agent use.

How to Tell It’s Time to Change the Harness

Change an agent harness when a recurring problem points to a control your team can improve. Compare results within the same task family, and check whether task difficulty, staffing, infrastructure, or requirements changed before attributing the difference to the harness.

Use acceptance, review time, retries, and cost to decide where to investigate:

Skills deserve the same evaluation as other harness changes.

SWE-Skills-Bench evaluated 49 skills across 565 software-engineering task instances using Claude Code with Claude Haiku 4.5. Pass rates were unchanged for 39 skills, but 24 of those already had a 100% pass rate without a skill. Some skills also reduced token consumption without changing pass rates. The findings support evaluating skills for task success and cost in the configuration you use.

Before replacing an agent platform, identify who owns the failing control. A platform change is worth evaluating when the current system cannot provide the required permissions, verification, tracing, or recovery behavior. Define the result the replacement must improve before testing it.

How to Test Whether a Harness Change Worked

Start with a reproducible failure and one harness change you can isolate. Define the expected improvement and acceptable limits for cost, review effort, and rework before running the comparison. Preserve enough evidence for another engineer to reproduce the evaluation.

Use the following five steps to test the change and decide whether to retain it.

Pick One Task Family

Choose comparable dependency updates, API migrations, or small bug fixes from one product. Give legacy modernization its own group because unfamiliar dependencies and weak tests can change task difficulty. Record acceptance criteria and required checks first.

Run the Old and New Harness Against Comparable Tasks

Run both configurations on isolated copies of the same tasks and repository snapshots where possible. Hold the model, compute, and time budget constant, and change only the harness component under evaluation. Randomize run order, repeat runs, and keep evaluation tasks separate from tuning tasks.

Capture Traces, Costs, Validations, and Final Artifacts

For every attempt, retain the task ID, harness revision, instructions, tool calls, errors, usage, diff, and validation output. Include failed and interrupted runs. Protect evaluation checks from agent edits, and test a prohibited action in a sandbox to confirm the guard blocks it.

Compare Acceptance, Review Burden, Rework, and Cost

Apply the same acceptance rubric to both groups, with reviewers blinded to the configuration where possible. Include failed and unresolved attempts in task-level results. Report acceptance rates, reviewer effort, rework, and cost with sample sizes and uncertainty, using the same follow-up window.

The August 2026 AutoSaddler preprint reported a 9.6-percentage-point improvement on SWE-Bench Pro over its baseline harness. Its method combined failure-trace analysis, targeted harness updates, and validation-based selection. This benchmark result supports testing focused changes; improvements in your repository require a separate evaluation.

Keep the Change Only if It Holds Up Downstream

Reviewer corrections, follow-up fixes, and operating costs can offset an early gain. In a hypothetical rollout, commit-to-production lead time might fall from four days to three while checkout-related support tickets rise from four to seven per 1,000 orders. Investigate the affected releases before expanding the rollout.

That pattern alone cannot establish the cause. A small gain across a few tasks calls for more observation, with the previous configuration available to restore.

Common Agent Harness Failure Modes

Diagnose agent harness failures through tool-call traces, validation results, and final artifacts. The checklist below identifies what to inspect within a run; use those findings to explain the acceptance, review, and cost patterns measured across tasks.

• Context that adds cost without improving task resolution. Check whether repository instructions supply missing guidance, repeat existing documentation, or trigger additional work that does not improve the result. The September 2026 revision of Evaluating AGENTS.md found that automatically generated repository context files increased inference costs by an average of 20% on SWE-bench Lite and 23% on CTXbench, without statistically significant resolution gains in the tested configurations.
• Tools without enforced permissions. Verify which repositories, commands, and services the agent can access. Test whether denied operations remain blocked through alternative tools. Connecting a security service does not establish these permissions.
• Retries without a changed approach. Look for repeated failing commands that use the same inputs and conditions. Check whether retry limits and escalation rules activate when progress stops.
• Passing tests with poor product fit. Compare validation results with the intended user behavior. Identify business rules and exception cases that the tests do not cover.
• PRs too large to review safely. Compare the diff with the task’s acceptance criteria. Flag unrelated cleanup, unexplained dependency changes, and edits that expand the review scope.
• Changes outside the intended scope. Inspect the complete diff for modifications to restricted files, configurations, or services. Confirm that write boundaries are enforced outside the agent’s instructions.
• Missing execution evidence. Check whether the trace preserves the requirement, tool results, errors, and validation commands needed to reconstruct the change.
• High token spend with low accepted output. Identify repeated exploration, abandoned attempts, and duplicated work. Check these patterns before increasing capacity or reasoning budgets.
• Missing links to delivery metrics. Verify that run IDs connect to tasks, PRs, deployments, and follow-up fixes. Without those links, delays and rework cannot be reliably associated with agent-assisted work.

Functional validation also leaves security gaps. The 2026 study When Passing Tests Hides Vulnerabilities identified 170 confirmed security-related silent failures across 1,030 valid execution traces from seven frameworks using GPT-4o-mini. These outputs passed functional checks while retaining or introducing vulnerabilities. The study used security-focused tasks, so its findings are not a general failure rate for coding agents.

Treat harness controls as one part of your security solution. Security review should assess credential exposure, unsafe inputs, and exposure to prompt injection and unsafe tool calls across all generated artifacts. Software architecture and green tests provide useful evidence, but neither establishes that every security requirement has been met.

How Axify Helps Engineering Leaders Evaluate Agent Harnesses

At Axify, we help you examine whether an AI workflow changes the way your team delivers software. Our tool analyzes AI adoption, usage, and cost through the lens of delivery metrics so engineering leaders can identify patterns and choose where to investigate. Use controlled harness evaluations to assess specific configuration changes, then use team-level delivery data to monitor their effects after rollout.

With our AI Adoption and Impact capabilities, you can review supported tools such as Claude Code, GitHub Copilot, and Cursor alongside other connected usage sources.

You can:

• Review adoption by team, developer, and tool. See who actively uses the available tools and where adoption is limited, with an AI adoption score built on three pillars: reflexive use, reliance, and trust. Seat ownership doesn’t show whether a tool fits the work.
• Track AI spending. Examine usage and gross cost at provider list rates by tool, model, team, and developer to identify expensive patterns.
• Compare usage with delivery metrics. Review cycle time and deployment frequency alongside AI adoption and cost, using similar reporting periods before and after AI adoption.
• Locate delays through value stream mapping. Use cycle time, WIP, throughput, and other VSM metrics to see whether work accumulates in development, review, or QA.
• Use Axify Intelligence to investigate causes and implement actions. Our recommendations help you identify delivery issues and prioritize changes using the connected data, such as adjusting review ownership or reducing parallel work.

Developer-level usage helps identify adoption and support needs. It cannot establish each person's business value or justify ranking developers by tokens, commits, or PR counts. Shared review work and task difficulty affect those results.

Use your evaluation tooling to retain agent traces and verify individual outputs. Use Axify to monitor team-level delivery patterns and investigate bottlenecks. Establish whether a harness change caused an improvement through a controlled comparison.

Final Takeaway: How to Decide Whether to Scale an Agent Harness

Judge a harness by whether it makes AI-generated code safer to handle, cheaper to deliver, easier to verify, and more useful to the team. Simply analyzing autonomy cannot answer those questions.

Start with a recurring failure and test a targeted change against the current configuration. Expand the rollout when acceptance improves, or total cost falls while verification, review effort, and downstream quality remain within agreed limits. Reevaluate the configuration when the model, tools, or task mix changes.

Axify helps you run that last check with evidence by bringing AI adoption, AI cost, and delivery metrics into one view.

Want to see whether your agent rollout is improving delivery? Start your 14-day free trial of Axify.

FAQs

What is harness engineering?

Harness engineering is the practice of designing and improving the execution controls around a model. This includes the context, permissions, verification, and recovery mechanisms described in the main definition above.

Is harness engineering real?

Yes, harness engineering describes measurable engineering work around agent execution. LangChain increased its Terminal-Bench score from 52.8% to 66.5% through harness changes while keeping OpenAI's GPT-5.2-Codex model fixed. The SpatialBench and Anthropic examples above provide further evidence about harness and environment effects. These results remain specific to their evaluation setups.

Why is it called harness engineering?

It is called harness engineering because the surrounding system holds and guides model execution. The term also echoes software's test-harness tradition, where supporting code runs and checks another component. Usage is still evolving, so define the boundaries whenever you compare systems.

Should you compare harnesses using the same model?

Keep the model fixed when testing the effect of a harness change. If the model also changes, evaluate the combined configuration and avoid attributing the result to the harness alone. Record model versions and execution settings so the comparison can be repeated.

Is Claude Code harness engineering?

Claude Code is a coding-agent product that runs with a harness. Harness engineering is the work of designing or adapting those execution controls. Anthropic describes the related Claude Agent SDK as a general-purpose harness. Your repository-specific rules and checks extend that setup for your work. Our Claude Code best practices cover the instructions, permissions, and review rules that shape it.

How should you evaluate a harness for high-risk code?

Define security and correctness requirements before testing, then verify the changes in an isolated environment. Protect evaluation checks from agent edits and require review by the relevant technical owner. Expand access only after the configuration passes those checks and its failure behavior has been assessed.

How do engineering leaders control agent harness costs?

Engineering leaders control harness costs through task-level budgets, bounded retries, suitable model selection, and cost-per-accepted-outcome reviews. You can include failed attempts, infrastructure, supporting tools, and human repair effort, following the cost logic above. Our guides to measuring Claude Code impact and FinOps for AI explain usage and total-cost accounting in more detail.