Many surveys of law departments ask questions to be answered on a scale, such as “How satisfied are you with the technology used by your primary law firms?” The scale goes from 1 (“not very satisfied”) to 2 (“reasonably satisfied” to 3 (“very satisfied”). Most of us don’t think twice about the methodological validity of the resultant headline, “Law departments scorn law firm tech: average score of 2.2.”
Unfortunately, this methodology treats a non-interval scale like an interval scale, as though the unit between each number – 1 to 2 or 2 to 3 – were an objective, agreed-to difference between “reasonably” and “very.” Such scalar questions find favor because they are familiar and easy, but they can mangle the actual complexity of answers respondents would like to give. At the least, offer seven intervals. As is, however, commonly done, a three-point scale leaves nuance shaved off, insights squashed into crude averages or medians. This admonishment comes from Deirdre N. McCloskey, Bourgeois Dignity: Why Economics Can’t Explain the Modern World (Univ. Chic. 2010) at 63.