Why Your LLM is a Slot Machine

Summary

The transcript discusses the underlying computational reasons why ChatGPT generates different responses to the same prompt, specifically how GPU processing, batch requests, and floating point math create variability. The key technical explanation revolves around how GPUs distribute processing across multiple cores, which introduces different addition orders and rounding errors that ultimately affect token selection. The practical takeaway is that response randomness isn't intentional but a byproduct of complex computational load-balancing, essentially meaning users are "gambling" on the number of concurrent users and hardware allocation when submitting queries.