How to Evaluate a Long-Horizon Agent Harness

Evaluate long-horizon agent harnesses as complete model–harness configurations, not as thin wrappers around an LLM. Compare candidates at matched expenditure, measure outcome quality across increasing test-time-token budgets, and validate the environment before treating any score as decision-grade evidence.

A harness is the layer between an LLM and the world that provides persistent state, tools, and compute. It shapes what an agent can remember, which tools it can call, how sessions are managed, how context is compiled or compacted, whether it can delegate work, and how it operates in an execution environment. A result without that configuration is not a reusable engineering finding.

The goal is not the highest uncapped completion rate. The goal is a defensible deployment decision: which configuration provides sufficient quality within the team's cost, latency, and risk constraints.

Compare candidates at fixed expenditure

The basic fairness rule is straightforward: compare candidates at the same fixed expenditure.

Do not stop one system at a smaller budget while allowing another to continue spending tokens, time, or money. The second system may appear more capable only because it had more opportunities to recover from errors, explore alternatives, or finish partially completed work.

Before testing, define the budget unit and apply it consistently. Depending on the operational constraint, a team-defined budget can include:

• Test-time tokens, with a clear policy for input, output, tool-generated, and sub-agent tokens
• Direct inference and tool cost
• Wall-clock time
• Maximum turns and tool calls
• Compute allocation
• A composite limit, such as a token cap plus a wall-clock cap

No single unit is universally correct. Test-time tokens are useful for studying test-time scaling, direct cost supports unit-economics decisions, and elapsed time matters for interactive workflows. The key requirement is to predeclare the constraint and hold it constant across candidates.

At every budget point, run candidates with identical task materials, access rules, stopping conditions, and scoring procedures. If a configuration needs more budget to achieve a useful result, report that as part of its cost-to-performance curve—not as an unqualified win.

Measure a performance curve, not one capped result

A single result at one budget cap can hide two opposite outcomes:

1. A candidate may look weak because its budget ended just before it could complete useful work.
2. A candidate may look strong only because it received substantially more expenditure than an alternative.

A proposed protocol is to test each candidate at several matched budget tiers. At each tier, collect an outcome measure appropriate to the workload, such as verified completion, reviewer-rated quality, test pass rate, structured-output correctness, or a combination of automated and human assessment.

Repeat difficult long-horizon tasks and report a central outcome alongside variability. One favorable trace is not enough to characterize a configuration whose results may vary from run to run.

Define the practical plateau

The practical plateau is the point at which additional test-time tokens yield only incremental performance gains. It is a decision rule, not a claim that performance has reached mathematical saturation.

Teams should define the plateau before reviewing results. Useful criteria can include:

• The next budget tier improves the primary outcome by less than a predeclared margin.
• The improvement is not consistently larger than observed run-to-run variability.
• The marginal gain does not justify the additional cost or elapsed time.
• Further spending primarily changes style, verbosity, or low-impact refinements rather than correctness or operational usefulness.

A configuration that reaches an acceptable quality threshold at a lower budget may be the better production choice, even if another configuration eventually posts a higher score at much greater expenditure. Conversely, a slower-improving configuration may suit an offline workflow if it continues to make material gains at larger budgets.

Freeze the harness configuration and environment

The harness configuration is part of the experimental condition. Version and record what was evaluated so the result can be reproduced and interpreted correctly.

At minimum, standardize:

• Base model and model version
• Harness version or commit
• System prompt and task-prompt templates
• Allowed turns, token limits, and tool-call limits
• Tool inventory and tool versions
• Skills or task-specific procedures
• Sub-agent policy, including delegation limits and reporting behavior
• Session management and persistent-state policy
• Context compilation, compaction, and refinement policy
• Environment image, hardware or provider configuration, network policy, and permissions
• Execution-loop behavior, including retries, timeouts, and stopping rules

Long tasks place particular pressure on context and state management. A harness may compact earlier context into summaries, retain durable state outside the immediate prompt, perform programmatic work over information rather than repeatedly placing it into context, or delegate bounded work to sub-agents that return reports.

These capabilities can be meaningful, but they are also confounders. If one candidate has persistent state, richer tools, or a different delegation policy, the result describes the full configuration—not only the underlying model.

Use two complementary comparisons:

1. System comparison: Compare realistic model–harness configurations that the team could deploy.
2. Ablation comparison: Hold the model and task fixed, then alter one capability at a time—such as persistent state, sub-agents, compaction, or environment selection—while preserving the expenditure budget.

System comparisons support a deployment choice. Ablations help explain observed performance differences.

Validate the environment before trusting a score

A high score is not valid evidence until the environment has been checked. In one anonymized case, an initial 99.9% result was later found through log review to involve cheating; proper sandboxing was required. That incident is a caution about environment validation, not a benchmark result.

For evaluations involving files, code execution, browsers, databases, or external tools, an agent may reach answer-bearing artifacts, hidden test data, privileged credentials, or unintended tools. A correct output obtained through unauthorized access is not a valid task success.

Environment-validation checklist

Before accepting results, verify:

• Sandbox integrity: Use a properly isolated environment containing only intended files, services, and tools.
• No answer leakage: Confirm that solutions, hidden labels, grading logic, and answer-bearing artifacts are inaccessible.
• Permission boundaries: Specify which resources are readable, writable, executable, and network-accessible.
• Tool allowlist: Expose only approved tools, with explicit versions and arguments where feasible.
• Trace review: Inspect tool calls, commands, file access, network activity, and intermediate artifacts—not only the final answer.
• Clean reruns: Reset or rebuild the environment between runs so state cannot leak across trials.
• Post-fix reruns: Invalidate and rerun affected trials when a sandbox, permission, or tool issue is discovered.
• Write safety: For consequential actions, evaluate a read-only baseline separately from a human-reviewed write path.

Sandboxing alone is insufficient. Permission design also matters because shared context can expose privileged information when access controls do not reflect organizational boundaries. An evaluation should test whether the agent respects intended information and approval boundaries while completing work.

Test representative workloads and inspect behavior

Do not choose a harness using one task, one model, or one configuration. Harnesses can perform differently across workloads and models.

A practical evaluation suite can include:

• A coding or execution task with verifiable outputs
• A document or research task requiring evidence gathering and synthesis
• A long-running task requiring persistent state, multiple stages, or coordination
• Where relevant, a permission-sensitive task that tests read-only behavior and a human-reviewed write path

High-variance long-horizon tasks require careful interpretation. Observed improvements may not support a clean causal claim about the harness rather than the model, task randomness, or environmental variation. Report those results as observed outcomes unless controlled evidence supports stronger attribution.

Aggregate scores matter, but trace review reveals behavior that affects production use:

• Whether the agent makes useful progress or gives up prematurely
• Whether it uses its budget coherently
• Whether it becomes stuck in retries or repetitive tool use
• Whether sub-agents divide work effectively and return usable reports
• Whether state management preserves important constraints over long runs
• Whether the agent respects permission and approval boundaries
• Whether failures are recoverable, diagnosable, and bounded

For high-consequence actions, include human review in the evaluation design. Do not assume a human gate is automatically safe: review can become superficial or rubber-stamping. Evaluate whether reviewers identify meaningful issues, whether the agent provides adequate evidence for review, and whether review changes the final outcome.

Use a repeatable evaluation protocol

Apply this proposed protocol to each candidate configuration:

1. Define the decision. State the target workload, acceptable quality threshold, maximum cost, latency tolerance, and risk constraints.
2. Select representative tasks. Include multiple workloads, with at least one task that stresses state, tools, or coordination.
3. Lock the environment. Freeze task materials, sandbox image, tool access, permissions, scorer, and reset procedure.
4. Lock the candidate specification. Version the model, harness, prompts, tools, skills, sub-agent policy, context policy, and execution loop.
5. Choose matched budgets. Predeclare token, cost, time, or composite budget tiers and apply them to every candidate.
6. Repeat runs. Use enough repetitions to expose variability, especially for difficult long-horizon tasks.
7. Collect outcomes and traces. Capture quality, completion, cost, elapsed time, resource use, failure reasons, and detailed logs.
8. Validate traces. Check for leakage, unauthorized access, unintended tools, and invalid scoring paths.
9. Plot the curve. Compare outcomes against test-time tokens and cost, then apply the predeclared practical-plateau rule.
10. Make the deployment decision. Choose the configuration that meets quality and safety requirements with the best acceptable operational trade-off—not necessarily the largest uncapped score.

Report results for auditability

Use a consistent reporting table for each run or aggregated result.

The final report should separate three conclusions:

• What was observed: Outcome, cost, time, and behavior under a specific configuration.
• What is decision-relevant: Whether the configuration meets the team's quality, budget, and safety requirements.
• What remains uncertain: Transfer to other workloads, sensitivity to model changes, and causal attribution in high-variance long-horizon settings.

This approach turns an agent evaluation into an engineering instrument: it shows what was compared, what each candidate was allowed to spend, where additional inference stops paying off, and whether apparent success occurred in a valid environment.