3 New PCs, One Giant AI Model… This Shouldn’t Work

Summary

The main theme is running a large 70 billion parameter AI model on consumer hardware, specifically the ASUS NUC 16 Pro with Core Ultra X7 processors and Arc GPUs. While initially attempting to split the model proved disappointing, the practical takeaway is that replicating the full model on each device and directing requests to different machines significantly speeds up performance, achieving approximately two and a half times the original speed.