Effectiveness · Is it working?
Are the agents accomplishing tasks — not just running fast and cheap?
Latency and cost say how fast and expensive a model is. This asks the question those miss: did the agent actually accomplish the goal? Success, human-intervention, and rework rates — per agent and per workflow.
Bottom line
Fleet success 89% across 1212 tasks — but only 63% of agents emit a measured signal (2 inferred, 1 dark). 8% of tasks needed a human. Biggest lift: support-copilot (~56 failed tasks).
Outcome signal — known vs guessed
Outcome signals need run-outcome instrumentation. 5/8 agents emit a measured signal; 2 are inferred from retries / abandons / errors; 1 are dark (shadow). Values are projected where signal ≠ measured — treat inferred rates as directional, not exact.
89%
fleet success rate
1212 tasks, signal-bearing
8%
needed a human
intervention / handoff
13%
rework rate
retried / redone
86
quality score
63% measured coverage
Fix this first — by failed-task volume
tasks × (1 − success) · biggest absolute liftWhere the most goals are actually missed — a high-volume agent at 82% can fail more tasks than a low-volume one at 48%. Fixing the top row moves the most outcomes.
By workflow
By agent
This is task success, not speed — latency & cost live in Performance →. Derived: effectivenessCoverage · fleetEffectiveness · lowestEffectiveness — from app/data/c16-fleet.ts (proposed run-outcome signal; measured / inferred / none provenance per agent).