Law Department Management Blog
This blog, and lawyers in general, often use or refer to surveys. The Manual for Complex Litigation (US Gov’t Printing Office, 3rd Ed. at 101-103 lists seven criteria for deciding whether or not a survey is trustworthy, according to Economic Approaches to Intellectual Property: Policy, Litigation, and Management (NERA Econ. Consulting 2005, at 125-126). Those who study law department management, an area awash in surveys (See my post of Oct. 17, 2005 listing some survey sources.), should evaluate survey data against these criteria.
1. The population was properly chosen and defined.
2. The sample chosen was representative of the population.
3. The data gathered was accurately reported.
4. The data were analyzed in accordance with accepted statistical principles.
5. The questions were clear and not leading.
6. The survey was conducted by qualified persons following proper interview procedures.
7. The process was conducted so as to assure objectivity.
For surveys of law departments, the shortcomings are numbers 2 (because only those who choose to respond are included, and that self-selected group may not be representative), 4 (because there may not be enough data to be reliable, or no one applies statistical analysis), 5 (because unambiguous, unbiased questions are extremely hard to craft), and 7 (because often vendors, consultants or interested parties conduct the surveys). Not a pretty picture, I guess.