SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius

Summary

This presentation emphasizes the critical importance of rigorous evaluation for coding agents, using the SweetRebench leaderboard as an example. The speaker, a dentist by training, draws a parallel between the high stakes of medical errors and software engineering failures, underscoring that intuition is insufficient for production. The key takeaway is that regular, "fresh" evaluations, particularly time-split benchmarks, are essential to combat data contamination and accurately assess model performance in real-world software engineering tasks.