Production Evals For Agentic AI Systems - Nishant Gupta, Meta Superintelligence Labs

Summary

The main theme is the evolution of evaluation for agentic AI systems, moving beyond simple model benchmarking to assessing production behavior. Key subjects include agentic systems, training and inference infrastructure, and the distinction between model capability and system behavior. The practical takeaway is that evaluation must shift from judging answers to evaluating complex workflows and their robustness in production environments, accounting for a wider range of failure modes beyond hallucinations.