SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI

Summary

SWE Marathon is a benchmark designed to evaluate the coherence of coding agents over extended, project-scale tasks, moving beyond simple bug fixes to owning entire projects. Building on lineage from HumanEval and SWE-bench, it aims to measure if agents can perform complex actions like rebuilding software from scratch or translating codebases, with a key takeaway being the critical need for robust, multi-faceted verification systems to prevent agents from exploiting weak testing during long execution horizons.