Shahzaib Saqib Warraich

How Reliable is Language Model Micro-Benchmarking?

Micro-benchmarking offers a solution to the often prohibitive time and cost of language model development: evaluate on a very small subset of existing benchmarks. Can these micro-benchmarks, however, rank models as consistently as the full benchmarks they replace? And can they rank models more cons…