Impossible to calculate return on knowledge-management investments

A thoughtful piece by Don Cohen in the Harv. Bus. Rev., Vol 84 December 2006, at 28 doncohen@rcn.com starts by acknowledging that “businesses have been trying — and mainly failing — to calculate the return on knowledge-management investments for more than a decade.” The short piece discusses why the measurement problem has not been solved and then offers two ideas.

Cohen suggests that “the more successful management projects target particular roles, groups, or processes and expected outcomes are articulated upfront.” For example, a law department might target the process by which its staff reviews marketing materials.

Second, those organizations with the most vibrant KM programs “approach the measurement problem by accepting soft indicators that knowledge management is earning its keep rather than demanding hard numbers that may be misleading.” For example, general counsel might accept a telling anecdote or two as better indicators of success than a precise but irrelevant number, perhaps such as the number of documents loaded in a database.

Dr. Dan Kirsch says:

September 13, 2007 at 9:31 pm

Hi Rees,
I can’t say that I agree with the premise that you cannot measure return on investment (ROI) for knowledge management (KM)..but that’s probably a “trick question.”
Firstly, ROI is really a pretty lousy “metric” to begin with. Sure lots of folks use it — but it is a quick ratio and there are more games to play with ROI than with employee performance appraisals and curves. In short, tell me what you’d like ROI to be and I can come up with a dozen legitimate ways to game it to accomplish that goal.
Secondly, most organizations that insist upon ROI metrics from something like KM don’t really “get it” to begin with. Implementation of KM is all about and has a lot to do with organizational change. If you’re not currently creating, transferring and utilizing knowledge…then you’re implementing a change to get the organization to do so.
And the significance of that is that an organizational change cycle is typically going to be somewhere in the neighborhood of 5-8 years (Kotter). So in short, you cannot sit down and begin KM implementation today and utilize traditional ROI measures which seek to (in most corporate ROI “competitions”) prove that return in 6-12 months.
One reason that this takes place with regard to KM is that most organizations still don’t seem to understand that KM is NOT about implementing IT (information technology). Sure you can implement some new fangled IT system and reasonably expect to achieve ROI targets over short periods. But KM is not the same.
Now having said that, let me suggest that there ARE in fact great ways to measure a sense of ROI in KM. We use things like “Knowledge Value-Added” (KVA) for that. KVA is based on the basic concept that if the knowledge within a process increases and that results in a increase in the outcome, then you can in fact measure the contribution that knowledge has to the process.
But related to things that would for example, impact the legal profession, there is a much easier way to measure the value of knowledge management. We pretty well accept (in the KM community) that about 50% of the time a “knowledge worker” (defined as someone who develops or uses knowledge — think for example, any documentation, briefs, etc.) will recreate their own document because they are unable to find it on the shared drive.
Based on that you’re looking at a potential 50% rework rate on knowledge documents. It doesn’t take an overactive imagination to realize that you can readily demonstrate a return.
And in fact, this might be a very easy way to get senior partner buy-in to the whole notion of KM implementation. If they don’t pay attention to what you have to say after you suggest that there is a 50% rework rate on documents…may as well give it up.
Another one that impacts the legal community — despite having spent thousands and thousand on one search appliance after another…we still see search failure rates as high as 95%. Search “failure” can be defined as failing to find the exact results you need in the first set of returned results (and we all know how we hate to scroll through page after page in hopes of finding what we need). Again, a lack of knowledge management. The search appliances are much less effective when the documents themselves aren’t properly keyworded. Both PDF and MS Word documents — the two most frequently found file types in a knowledge-base (add in Word Perfect for things related to courts that still utilize that) are typically saved to the knowledge-base without consideration given for including keywords to the document meta tags (most folks don’t seem to even know about that feature).
Frankly, demonstrating the value of KM efforts has a lot to do with properly implementing KM. KM needs to be tied to the firm’s overall strategy. Do that and the proper metrics begin to suggest themselves.
Regards,
Dan
Dr. Dan Kirsch, CKM, MKMP, CKMI
COO & Board Member
Knowledge Management Professional Society (KMPro)
Email: COO@KMPro.org