Summary
ParaRecover moves agent evaluation from final success to the process of diagnosing and recovering from failures across parallel tool branches. Its 10,626 instances cover 14 error types spanning dependency planning, tool selection and argument matching. The SDE rubric scores structural integrity, diagnostic reasoning and replanning strategy. Tests on more than ten models find persistent weakness in multi-turn error propagation, implicit failures and precise replanning; MIT-licensed code and data are public.
Why it matters
For teams operating parallel or multi-agent workflows, a final pass rate hides whether recovery is reliable or accidental. ParaRecover provides a regression suite for localization and replanning before these paths reach production. Because most scoring still depends on a research rubric and the repository has no visible adoption, it should complement environment-specific failure injection rather than replace it.
Technical details
| Dataset | 10,626 instances at two difficulty levels |
|---|---|
| Taxonomy | 14 error types across planning dependencies, tool selection and argument matching |
| Rubric | SDE: structural integrity, diagnostic reasoning and evolutionary/replanning strategy |
| Model Coverage | more than ten mainstream LLMs |
| Availability | MIT-licensed GitHub repository with data and evaluation code |
| Adoption | 0 GitHub stars at scan cutoff |
Tags
agent-evaluationparallel-tool-useerror-recoveryreplanningbenchmark