The paper reports simple harnesses outperforming five hand-crafted RCA agents under a shared RCAtlas setup.
Study finds simple LLM harnesses beat hand-crafted root-cause agents
2026-10-27Frontier watchWATCH
A study compares three simple LLM harnesses with five published root-cause-analysis agents plus statistical and causal baselines on RCAtlas. Across 90 validated incidents from eight public datasets, six models and three reasoning efforts, the authors report that simple harnesses reach 65–77% PR@1 on three mid-tier models versus 16–53% for the hand-crafted agents. The paper does not link a public implementation or independent reproduction.
What this means for you
For observability and incident-response teams, the result warns that fixed stages and role agents can hide decisive evidence and add cost. The comparison is broad enough to influence prototype design, but not yet strong enough to replace production evaluation.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Evaluation | RCAtlas: 90 validated cases from eight public datasets |
|---|---|
| Comparators | Three simple harnesses, five published LLM RCA agents, statistical and causal methods |
| Models | Six models and three reasoning-effort levels |
| Reported Results | Simple harnesses 65–77% PR@1 on three mid-tier models; published agents 16–53% |
| Cost Limit | Higher reasoning effort changed accuracy by only a few points while increasing time by up to 7x |
| Artifacts | Paper and TeX source; no implementation linked |
| Time Basis | arXiv v1 submitted 2026-10-26T17:12:02Z |
Technical impact 4
Engineering value 5
Adoption signal 1
Maturity 1
Verification cost 3
Risk 2