Scaling Laws
Empirical relationships predicting model quality from compute, data, and size.
Scaling laws describe how model loss falls predictably as parameters, data, and compute increase. They turn model design into a budgeting exercise: given a compute budget, how big should the model be and how much data should it see?
The Chinchilla result refined these laws, showing most large models were undertrained and should be trained on far more data for their size.