The Corrupting Influence of Metrics
There is, every year, it seems, a bigger focus and greater reliance upon metrics. Metrics can, of course, provide a much-needed focal point for management objectives, and provide valuable performance feedback. One of the continuing issues with metrics is the ‘gaming” behavior that metrics can drive. Metrics inadvertently become the end, rather than a measure of the means. Stories abound of unintended consequences. What we see time and time again is a corrupting effect of metrics on what the metrics are purporting to measure. Has our mad rush to quantify and measure everything resulted in the unintended consequences becoming the rule rather than the exception?
I have spoken with many companies about metrics, in the course of advising them about using software and technology more effectively in their operations to optimize their business processes. In nearly all of these conversations, the issue of gaming comes up.
We all know what ‘gaming the system’ is. If we are not directly in involved in work tasks in which we need to meet specified metrics, most of us are familiar with the strategy of under-promising and over-delivering – gaming our expectation-setting to make our subsequent results look better. Exceeding expectations has become so important that it, well, has come to be the expectation.
Metrics in the Healthcare Industry
Two (2) recent articles have caught my attention.
- “U.S. Ties Hospital Payments to Making Patients Happy”, Janet Adamy, Wall Street Journal, October 14, 2012 (subscription required)
- “Blue Cross speeds up its customer service for one week“, Robert Weisman, The Boston Globe, October 20, 2012.
These two articles, appearing together within a week of one another, highlight these issues in a very poignant way.
Janet Adamy, in her article, enlightens us on how the results of a 27-question survey will be used to decide how some $1B of payments are allocated among hospitals. The survey, administered by the Hospital Care Quality Information from the Consumer Perspective (HCAHPS), in Baltimore, MD., has been in use since 2006, but will only now impact hospital revenues.
What is the survey measuring? Well, one question is: “During this hospital stay, how often was the area around your room quiet at night? (Never / Sometimes / Usually / Always ).” Really? Are nurses now responsible for enforcing a quiet time at the hospital time now?
With hundreds of thousands to millions of dollars at stake, who among us does not think that hospitals will manage to these 27 questions? As patients, do we want our medical staff focusing and attending to our medical needs, or monitoring noise levels at night?
Full disclosure: I have no medical background, training, or experience.
Hospitals are also complaining about the lack of relevance of the metrics and the unintended consequences. Doctors who feel that they are in the life-saving business are now being told they are in the customer service business, requiring the staff to reallocate time spent providing medical treatment to time spent delivering customer service – to a customer that does not generally understand the product or the implications of the metrics.
At least one hospital has adding flat screen TVs for patients to solicit higher scores. Another hospital has installed mini waterfalls in some patient rooms. Another hospital added ESPN because the consumption of pain medicine goes down during football games.
One hospital nurse in the Wall Street Journal story relates the story of a patient, who, after receiving treatment for a life-threatening stroke, dinged the hospital for cold meals. Is a meal warmed to the proper temperature as important as a life-saving medical treatment?
The healthcare industry may prove an exceptional case study on the questionable use of metrics and the law of unintended consequences.
A study earlier this year discovered, in fact, according to its abstract, that patients with higher customer satisfaction scores experienced “greater inpatient use, higher overall health care and prescription drug expenditures, and increased mortality.” (Subscription required for full access to the study. )
The other article, from the Boston Globe’s Robert Weisman, alleges that Blue Cross Blue Shield of Massachusetts is boosting up its customer service processes for the one week that coincides with a survey of its practices. While Blue Cross may claim that its revised customer service practices are pilots for new processes, the timing is certainly curious, to say the least, leading to the unavoidable suspicion that there is an effort afoot to juice the survey results by boosting customer responsiveness.
We see reports all the time of aberrations like this, most often when there is an inordinate focus on the end results, and a “no excuses” pressure on “making the numbers”.
Metrics in the Broader Context
The healthcare industry is, of course, a special case, particularly so given recent extraordinary changes in the industry.
In the broader context, Andrew Stein points out, in The Trouble With Metrics, that there is a significant distinction between “change”, and “transformation”.
Andrew Stein argues that if you want to drive behavior a certain way to achieve specific results, better to think in terms of transformation, and use metrics to measure the behaviors over time. Focus on metrics as results, and expect behaviors to change short-term to achieve the specified metrics, without any long-term transformative effects.
While there is often an almost irresistible urge, particularly for management, to focus on and measure near-term results, and manage by the numbers, longer-term success requires that a focus on quantitative results be balanced with metrics that focus on shaping the cross-functional process behaviors that drive excellence.
There are other things to watch out for when instituting a set of metrics:
- Having too many metrics
- Understanding what the metrics meant and were trying to measure
- Anticipating how specific metrics are likely to drive, rather than measure, behavior
- Cultural resistance over the definition of metrics
- Constant churn in the metrics being used
- Data availability to measure the metrics quickly and consistently
- Stovepiping: metrics that encourage siloing
- People gaming the system to generate the desired metrics, compromising sound practices
Metrics and The Customer Experience
Both the Wall Street Journal article and the Boston Globe are addressing customer-oriented processes. And when it comes to customer-oriented processes, quantitative measurements may not even be the best way to measure the effectiveness of these processes.
There has been increasing focus over the past several years on something called “the customer experience”. For a variety of reasons, and in response to several trends, there has been an increasing focus on adopting a formalized “outside-in” view and discipline in order to orient and refine internal processes to be more responsive to the customer.
As Forrester Research has stated, however:
“While NPS (Net Promoter Score) and other quantitatively focused measurement techniques help firms understand the current state of their customer experience, they’re simply not the right tools to understand what kinds of experiences customers truly want and need.” (2012 Customer Experience Predictions, Forrester Research, by Kerry Bodine, January 6, 2012)
It is all the more interesting, then, from multiple perspectives, why more companies are not more actively adopting a strong approach to the Customer Experience. This has been mystifying to me for as long as I can remember.
Forrester has been reporting that the amount of churn in the top ranks of people overseeing the Customer Experience seems to indicate that many companies are still not as serious about this Customer Experience focus as they could be.
If the Wall Street Journal and Boston Globe stories are any sort of indication, companies might be better served addressing the underlying processes themselves, rather than trying to set up measurements that result in such efforts to game the results.
Best practices for Using Metrics
There is no single set of Best Practices than can address all of the possible uses of metrics. However there are a set of general Best Practices that can help to maintain a set of metrics that have more integrity and more accurately measures the targeted behaviors:
- For best results, use metrics to drive longer-term behavior transformation. In the shorter-term, metrics may encourage more short-term behavior modification and gaming of the system, with less long-term sustained results.
- Do not make the mistake of thinking that all metrics are results-oriented. The best metrics may not only be process-focused, but cross-functionally process-focused. Results-oriented metrics are the metrics that may be most susceptible to being co-opted as the numbers to be achieved at all costs, rather than as a target or measure of the results to be achieved.
- Think long and hard about the behavior(s) that you are trying to drive, and then structure metrics in such a way that gaming the system is at least as difficult as the target behaviors.
- Use metrics that are measurable, and measurable in an easy and consistent a manner as possible.
- Anticipate how your metrics will encourage responsibility-shifting and -dumping, and then try to refine the metrics to encourage a more responsibility-assuming behavior, where appropriate.
- Focus metrics, where appropriate, on the ideal client experience, and how metrics can be used to drive behavior that is more focused on continual improvements to the client experience.
- Be sure to apply the metrics at the point where there is responsibility and authority to transform behavior. Applying a metric to a result or behavior over which someone has no control or authority is likely to be very counter-productive.
- Follow up your metrics regularly and frequently early on to assess whether the metrics are shaping behaviors as intended. Refine the metrics where necessary to minimize or stop instances where results are being compromised by efforts to game the metrics using unwanted behavior(s).
While there is a lot of effectiveness and efficiency that can be driven through an organization through the use of appropriate metrics, the effort should not be underestimated. Metrics need to be established by people who know what they are doing, who are proficient in understanding and managing efforts to game the system, and who can recognize and quickly respond to unintended consequences. Metrics will seldom be any substitute for a sound focus on the Customer Experience.