Benchmark data and the pernicious “Law of Small Numbers”

According to the Law of Large Numbers, “you can have a high degree of confidence in the average value of a sample if the sample includes a very large number of observations.” As explained in the NY Rev. of Books, Oct. 8, 2009 at 30, therefore, the more legal departments contribute data to a benchmark study, the more you can rely on the resulting metrics (See my post of Feb. 7, 2008: crowdsourcing depends on the Law of Large Numbers.).

The problem arises when surveyors stop way short of a sufficient number of observations but show high confidence in the results anyway. The facetious invocation of the Law of Small Numbers refers to our tendency to put too much faith in too small a slice of data.

Even while I so much desire even shreds of benchmarks, I bark all the time at sample size (See my post of Dec. 9, 2005: margin of error and sample size; Oct. 31, 2007: formula for confidence levels; April 22, 2007: power tests and sample size; March 28, 2005: number of respondents; Dec. 19, 2007: few participants; and April 9, 2005: few respondents from a large invitee pool.). Still, in a data desert, better Small Numbers than no numbers.