Multiple cautions about multiple-choice questions on surveys

Surveys of law departments frequently have multiple-choice questions, even though such questions are beset with methodological traps. Here are my blog posts on the topic.

Strive to cover every reasonable choice and not have choices overlap (See my post of June 16, 2007: the test of Mutually Exclusive, Comprehensively Exhaustive.). To do so, selections need to be defined tightly and worded carefully. Recognize that some plausible choices are likely to have been omitted and cover them with a selection for “Other” (See my posts of April 12, 2006 and July 14, 2005: importance of “Other” as a choice.).

The wording, number, and order of the choices make a difference. List selections neutrally, such as alphabetically, and make no effort to influence people by the order of choices (See my post of Dec. 20, 2005: the order of questions.). Avoid disjunctives such as “Knowledge of industry or experience” since you can’t untangle the combined concepts. Limit any one question to no more than seven or eight selections, since the number of selections has an effect on how people answer them (See my post of Nov. 13, 2006: decision difficulties when there are too many choices.).

Don’t invite respondents to “choose all that apply” (See my posts of July 21, 2005 and Dec. 20, 2005: criticize the methodology; and March 13, 2006: “choose more than one.”). Better to ask respondents to pick their top three or four of a large set (See my posts of Aug. 14, 2005 on this variation; Nov. 5, 2006; July 4, 2006: “pick the top 5”; and Nov. 5, 2006: “pick your top three from ten.”).

Even better is to ask respondents to rank their choices or their top few (See my posts of Dec. 20, 2005 and March 31, 2007: prioritize choices.). One problem with this step if you ask for all the choices to be ranked is that even choices undesirable to the respondent will be ranked. They can also misread the instructions and reverse their ranking from what you wanted.

Best of all for multiple-choice questions is to ask survey respondents to allocate 100 points among the choices (See my post of July 4, 2006: sophisticated technique.). They can give as many points as they want to their most favored choice and sprinkle fewer points among the remainder as they see fit. The downside is that respondents must think more and take more time, which discourages participation.