The Litigation Cost Survey of Major Companies, presented at the Searle Center on Law, Regulation, and Economic Growth, (Northwestern Univ. School of Law, May 10-11, 2010), can tutor all of us who do surveys of law departments. Several of the steps it took conform to what I believe are solid methodologies for surveys.
They drew on survey expertise. “AustinTrends, a consulting firm specializing in survey administration, assisted in the design of the online survey instrument and administration of the survey.”
They tested the survey before they unleashed it. “The survey was beta tested with four companies that provided feedback on potential ambiguity in questions and likely availability of data, as well as potential unanticipated barriers or excessive costs that might be incurred in attempting to respond to the survey.”
At the start, they sent a hardcopy of the survey to the companies they invited to participate. The report notes (at 4), however, that “Hard copy raises more confidentiality concerns. For these reasons, the respondents answered questions via a secure website.”
They stated the response rate and distribution, so readers can determine for themselves whether the number and representativeness is acceptable. “Out of the Fortune 200 companies that received the survey, 37 responded to at least portions of the survey. The response rate was therefore 18.5 percent.” This was followed by a comparison of respondents per industry to the percentage of that industry in the Fortune 200.