Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

Summary

This tech transcript discusses improving LLM evaluation by borrowing concepts from psychology and psychometrics, specifically moving beyond simple accuracy counts. It highlights the limitation of treating all questions equally and proposes using item difficulty (B parameter) to better assess model performance. The practical takeaway is to adopt more sophisticated methods from IQ testing to gain deeper insights into LLM capabilities.