Engineering · Effectiveness
Is it working? — are agents accomplishing the task
Fleet success 89% across 1212 signal-bearing tasks — but only 63% of agents emit a measured signal (2 inferred, 1 dark). 8% of tasks needed a human. Biggest lift: fix support-copilot first (~56 failed tasks).
Outcome signals need run-outcome instrumentation. 5/8 agents emit a measured signal; 2 are inferred from retries / abandons / errors; 1 are dark (shadow). Values are projected where signal ≠ measured — treat inferred rates as directional, not exact.
fleet success rate
1212 tasks, signal-bearing
needed a human
intervention / handoff
rework rate
retried / redone
quality score
63% measured coverage
Fleet success rate — 6-week trend
projected · signal-bearingUp +10 pts over 6 weeks (79% → 89%), still climbing this week.
Fix this first — by failed-task volume
tasks × (1 − success) · biggest absolute liftWhere the most goals are actually missed — a high-volume agent at 82% can fail more tasks than a low-volume one at 48%. Fixing the top row moves the most outcomes.
By workflow
Workflow | Tasks | Success | Intervene |
|---|---|---|---|
| data-sync measured | 540 | 97% | 2% |
| support-triage measured | 310 | 82% | 12% |
| code-review measured | 260 | 92% | 7% |
| content-gen inferred | 64 | 48% | 34% |
| infra-ops inferred | 38 | 66% | 21% |
By agent
Agent | Tasks | Success | Intervene | Quality |
|---|---|---|---|---|
| support-copilot measured | 310 | 82% | 12% | 79 |
| marketing-gpt inferred | 64 | 48% | 34% | — |
| data-pipeline-agent measured | 540 | 97% | 2% | 91 |
| Codex CLI inferred | 38 | 66% | 21% | — |
| Claude Code measured | 142 | 93% | 6% | 88 |
| Cursor measured | 96 | 90% | 8% | 84 |
| Claude Desktop measured | 22 | 95% | 5% | 90 |
| Zed Agent none | — | — | — | — |
This is task success, not speed. Latency and cost say how fast and cheap a model is; this asks whether the agent actually accomplished the goal. Latency & cost live in Performance →.
Derived: effectivenessCoverage · fleetEffectiveness · lowestEffectiveness — from app/data/c16-fleet.ts (proposed run-outcome signal; measured / inferred / none provenance per agent).