Defining Success Metrics
for Autonomous Agent Tasks
A framework for measuring what matters — from task completion to trust, alignment, and long-term value creation.
Task Completion Rate
The percentage of assigned tasks the agent successfully completes end-to-end without human intervention or rollback.
Efficiency & Speed
Wall-clock time and computational cost per task — accounting for token usage, API calls, and steps taken to reach the goal.
Goal Alignment
Whether agent outputs genuinely match the user’s intent — not just literal instructions, but the underlying desired outcome.
Safety & Refusal Quality
Accurate identification of out-of-scope, harmful, or ambiguous instructions. Neither over-refusal nor unsafe compliance.
Error Recovery
How gracefully the agent detects failures, corrects course, and surfaces uncertainty without cascading into larger mistakes.
Metric Taxonomy
Relative Importance by Domain
Goodhart’s Law Traps
When a metric becomes a target, it ceases to be a good metric. Agents optimize for measurable proxies, not true intent.
Eval-Train Leakage
Benchmark tasks that overlap with training distribution inflate scores without reflecting real-world generalization.
Ignoring Latent Costs
Speed metrics that ignore downstream rework, user confusion, or trust erosion give a dangerously incomplete picture.