The challenges and rewards of content analysis for survey responses

Multiple-choice questions on surveys of law departments are ubiquitous yet problematic (See my posts of June 16, 2007 on MECE; Dec. 3, 2007 for five improvements; and Dec. 20, 2005 on wording.). An alternative form of question asks respondents for text comments. The challenge then is to turn the comments into something digestible and into statistics.

With the technique known as “content analysis,” someone reads the text answers and clusters them according to their content. For example, if a question on an employee morale survey invites open-ended suggestions, that is a good situation for content analysis.

The technique is hard to do because you must make decisions and not let your views warp how you cluster and interpret the responses. The advantages of the technique include that you learn what you did not even think to ask about in a multiple-choice question, you get more nuance in context, and you can produce metrics. You can say that “30 percent of the lawyers want more desk top space.”