The style of a question robs its answers of much meaning (plans for matter management systems)

People who conduct surveys – I raise my hand high – should be wary when they throw stones at other surveys’ methodologies. Even at risk, I will continue to chide others on how they word questions and expect rocks through the windows of my own benchmark survey in turn.

So, consider this question from the Third Annual Law Department Operations Survey: “Are there plans to improve or evaluate a new matter management and/or e-billing system in the next 12 months?”

Start with praise, Rees: The topic, updating two important classes of software, deserves a question and the findings could be revelatory. Further, to put a time frame on the question gives it more precision – “in the next 12 months.”

But the question has four flaws. Start with it being a yes/no question (See my post of July 17, 2007: need a choice of “neutral”.). A binary choice in a complex world is very crude. A conjunctive question, the second problem, leaves interpretation of the results almost worthless and this one even manages has two “or’s”: “improve or evaluate” as well as “matter management or e-billing.” You can’t figure out which branch an answer sits on.

Third, the question leaves a gap, since it asks about “new” matter management or e-billing while but not about upgrades to an “existing” package. Fourth, “plans” is so vague a notion that any self-respecting administrator could say yes, if the thought flitted across their mind four months ago. Had the question asked a provable, operational question such as “Has money been budgeted to invest in improvements in the next 12 months?” that makes the answer more credible.