Adding Error Bars to Evals

Key findings

• Treats eval questions as samples from a distribution and derives confidence intervals, clustering and paired-difference tests - Many reported model differences are within noise

Sources

• arxiv.orghttps://arxiv.org/abs/2411.00640