There is, every year, it seems, a bigger focus and greater reliance upon metrics. Metrics can, of course, provide a much-needed focal point for management objectives, and provide valuable performance feedback. One of the continuing issues with metrics is the ‘gaming” behavior that metrics can drive. Metrics inadvertently become the end, rather than a measure of the means. Stories abound of unintended consequences. What we see time and time again is a corrupting effect of metrics on what the metrics are purporting to measure. Has our mad rush to quantify and measure everything resulted in the unintended consequences becoming the rule rather than the exception?
I have spoken with many companies about metrics, in the course of advising them about using software and technology more effectively in their operations to optimize their business processes. In nearly all of these conversations, the issue of gaming comes up.
We all know what ‘gaming the system’ is. If we are not directly in involved in work tasks in which we need to meet specified metrics, most of us are familiar with the strategy of under-promising and over-delivering – gaming our expectation-setting to make our subsequent results look better. Exceeding expectations has become so important that it, well, has come to be the expectation.
Metrics in the Healthcare Industry
Two (2) recent articles have caught my attention.
- “U.S. Ties Hospital Payments to Making Patients Happy”, Janet Adamy, Wall Street Journal, October 14, 2012 (subscription required)
- “Blue Cross speeds up its customer service for one week“, Robert Weisman, The Boston Globe, October 20, 2012.
These two articles, appearing together within a week of one another, highlight these issues in a very poignant way.
Janet Adamy, in her article, enlightens us on how the results of a 27-question survey will be used to decide how some $1B of payments are allocated among hospitals. The survey, administered by the Hospital Care Quality Information from the Consumer Perspective (HCAHPS), in Baltimore, MD., has been in use since 2006, but will only now impact hospital revenues.
What is the survey measuring? Well, one question is: “During this hospital stay, how often was the area around your room quiet at night? (Never / Sometimes / Usually / Always ).” Really? Are nurses now responsible for enforcing a quiet time at the hospital time now?
With hundreds of thousands to millions of dollars at stake, who among us does not think that hospitals will manage to these 27 questions? As patients, do we want our medical staff focusing and attending to our medical needs, or monitoring noise levels at night?
Full disclosure: I have no medical background, training, or experience.
Hospitals are also complaining about the lack of relevance of the metrics and the unintended consequences. Doctors who feel that they are in the life-saving business are now being told they are in the customer service business, requiring the staff to reallocate time spent providing medical treatment to time spent delivering customer service – to a customer that does not generally understand the product or the implications of the metrics.
At least one hospital has adding flat screen TVs for patients to solicit higher scores. Another hospital has installed mini waterfalls in some patient rooms. Another hospital added ESPN because the consumption of pain medicine goes down during football games.
One hospital nurse in the Wall Street Journal story relates the story of a patient, who, after receiving treatment for a life-threatening stroke, dinged the hospital for cold meals. Is a meal warmed to the proper temperature as important as a life-saving medical treatment?
The healthcare industry may prove an exceptional case study on the questionable use of metrics and the law of unintended consequences.
A study earlier this year discovered, in fact, according to its abstract, that patients with higher customer satisfaction scores experienced “greater inpatient use, higher overall health care and prescription drug expenditures, and increased mortality.” (Subscription required for full access to the study. )
The other article, from the Boston Globe’s Robert Weisman, alleges that Blue Cross Blue Shield of Massachusetts is boosting up its customer service processes for the one week that coincides with a survey of its practices. While Blue Cross may claim that its revised customer service practices are pilots for new processes, the timing is certainly curious, to say the least, leading to the unavoidable suspicion that there is an effort afoot to juice the survey results by boosting customer responsiveness.
We see reports all the time of aberrations like this, most often when there is an inordinate focus on the end results, and a “no excuses” pressure on “making the numbers”.
Metrics in the Broader Context
The healthcare industry is, of course, a special case, particularly so given recent extraordinary changes in the industry.
In the broader context, Andrew Stein points out, in The Trouble With Metrics, that there is a significant distinction between “change”, and “transformation”.
Andrew Stein argues that if you want to drive behavior a certain way to achieve specific results, better to think in terms of transformation, and use metrics to measure the behaviors over time. Focus on metrics as results, and expect behaviors to change short-term to achieve the specified metrics, without any long-term transformative effects.
While there is often an almost irresistible urge, particularly for management, to focus on and measure near-term results, and manage by the numbers, longer-term success requires that a focus on quantitative results be balanced with metrics that focus on shaping the cross-functional process behaviors that drive excellence.
There are other things to watch out for when instituting a set of metrics:
- Having too many metrics
- Understanding what the metrics meant and were trying to measure
- Anticipating how specific metrics are likely to drive, rather than measure, behavior
- Cultural resistance over the definition of metrics
- Constant churn in the metrics being used
- Data availability to measure the metrics quickly and consistently
- Stovepiping: metrics that encourage siloing
- People gaming the system to generate the desired metrics, compromising sound practices
Metrics and The Customer Experience
Both the Wall Street Journal article and the Boston Globe are addressing customer-oriented processes. And when it comes to customer-oriented processes, quantitative measurements may not even be the best way to measure the effectiveness of these processes.
There has been increasing focus over the past several years on something called “the customer experience”. For a variety of reasons, and in response to several trends, there has been an increasing focus on adopting a formalized “outside-in” view and discipline in order to orient and refine internal processes to be more responsive to the customer.
As Forrester Research has stated, however:
“While NPS (Net Promoter Score) and other quantitatively focused measurement techniques help firms understand the current state of their customer experience, they’re simply not the right tools to understand what kinds of experiences customers truly want and need.” (2012 Customer Experience Predictions, Forrester Research, by Kerry Bodine, January 6, 2012)
It is all the more interesting, then, from multiple perspectives, why more companies are not more actively adopting a strong approach to the Customer Experience. This has been mystifying to me for as long as I can remember.
Forrester has been reporting that the amount of churn in the top ranks of people overseeing the Customer Experience seems to indicate that many companies are still not as serious about this Customer Experience focus as they could be.
If the Wall Street Journal and Boston Globe stories are any sort of indication, companies might be better served addressing the underlying processes themselves, rather than trying to set up measurements that result in such efforts to game the results.
Best practices for Using Metrics
There is no single set of Best Practices than can address all of the possible uses of metrics. However there are a set of general Best Practices that can help to maintain a set of metrics that have more integrity and more accurately measures the targeted behaviors:
- For best results, use metrics to drive longer-term behavior transformation. In the shorter-term, metrics may encourage more short-term behavior modification and gaming of the system, with less long-term sustained results.
- Do not make the mistake of thinking that all metrics are results-oriented. The best metrics may not only be process-focused, but cross-functionally process-focused. Results-oriented metrics are the metrics that may be most susceptible to being co-opted as the numbers to be achieved at all costs, rather than as a target or measure of the results to be achieved.
- Think long and hard about the behavior(s) that you are trying to drive, and then structure metrics in such a way that gaming the system is at least as difficult as the target behaviors.
- Use metrics that are measurable, and measurable in an easy and consistent a manner as possible.
- Anticipate how your metrics will encourage responsibility-shifting and -dumping, and then try to refine the metrics to encourage a more responsibility-assuming behavior, where appropriate.
- Focus metrics, where appropriate, on the ideal client experience, and how metrics can be used to drive behavior that is more focused on continual improvements to the client experience.
- Be sure to apply the metrics at the point where there is responsibility and authority to transform behavior. Applying a metric to a result or behavior over which someone has no control or authority is likely to be very counter-productive.
- Follow up your metrics regularly and frequently early on to assess whether the metrics are shaping behaviors as intended. Refine the metrics where necessary to minimize or stop instances where results are being compromised by efforts to game the metrics using unwanted behavior(s).
While there is a lot of effectiveness and efficiency that can be driven through an organization through the use of appropriate metrics, the effort should not be underestimated. Metrics need to be established by people who know what they are doing, who are proficient in understanding and managing efforts to game the system, and who can recognize and quickly respond to unintended consequences. Metrics will seldom be any substitute for a sound focus on the Customer Experience.
Manufacturing has been on its heels, in Massachusetts and elsewhere. There can be no argument about this. The reasons for this (as well as the reasons, pro and con, for manufacturing itself) have been discussed to exhaustion elsewhere. Massachusetts is taking some welcome fresh steps to revitalize the manufacturing sector. Only time will tell if these new programs mean that we are finally going “on offense” with some strategic long-term solutions that signal a genuine new-found respect for a sector that is the source of so much innovation and vitality.
I have promised to breathe some new life into Netting It Out, and I will be doing so shortly.
In the meantime, I have been busy in several other venues:
- Procurement and Other Issues in Public Housing
Nick Harris and Bob Eastman, on the Calyptus Consulting blog
- A Conversation with Calyptus Consulting’s George Harris
Bob Eastman and George Harris, on the Calyptus Consulting
- Outsourcing IT Lets Small Businesses Focus on the Business
Heather B. Hayes, on the Biztechmagazine website
- Additional work published on the SMB Research website
Reposted from The SMB Research Blog…
Have you read enough pieces yet on what makes a great website?
You have probably seen quite a few articles and posts addressing this. We here at SMB Research know we have seen a lot of material. (SMB Research includes at least one fairly good article on SMB Research’s Top 20 Favorite Reads of 2010.)
Many of these articles seem to take a cookie-cutter approach: it should go without saying at this point that a website needs to have good navigation, and that a website needs a “Call to Action”.
We think there are a number of other critical website features and best practices that get too little attention.
What was the best SMB or technology-related piece you read this past year?
When we at SMB Research started talking a few weeks ago about what we thought were some of the most interesting and valuable articles from 2010, we realized that once again this year there has been a great deal of very good (however you define this) research and writing. If you struggle to find the time to read to read everything that deserves your attention, we share your pain.
To help you out, SMB Research decided to take a look back at some of what has been written and published this past year, and to identify some of our favorite reads.
SMB Research’s Top 20 Favorite Reads, 2010:
New post over at the (unaffiliated) Saint Joseph’s University Track & Field and XCountry website, http://sjutrackxcountry.wordpress.com. (I am not an alumnus, just a fan, and a Saint Joseph’s University parent.)
Nancy and I traveled to New York City on Saturday (20th) to watch the St Joseph’s University’s (almost) last Cross Country meet for the season: the IC4A / ECAC (InterCollegiate Association of Amateur Athletes of America) / (Eastern College Athletic conference) at Van Cortlandt Park in Bronx, New York.
Over on my alter-ego, The SMB Research Blog (re-posted with my own permission…)
While you will have to wait just awhile longer, we don’t mind giving you a sneak peek at some of the great learnings you can expect to get from the finished product.
SMB Research was very pleased to be able to attend the Greater Boston Chamber of Commerce’s 26th Annual Small Business of the Year Awards yesterday (Oct 14, 2010) at the Colonnade Hotel. Emceed by WBZ Radio’s co-Anchor of “WBZ Morning News” Ed Walsh, the event itself was flawlessly put together and efficiently executed, for a very enjoyable event. Better yet, the winner’s stories and remarks were compelling and inspiring.
Interesting to see who St Joseph’s University is inducting into its Athletics Hall of Fame tomorrow, October 16: The roster includes Vince Papale (class of 1968), whose story was told in the movie, Invincible. While the move tells the story of his improbable football career with the Philadelphia Eagles, the story of Mr. Papale’s cross-country and track achievements at St. Joseph’s University (where my son now attends) in Philadelphia is less well known. Among the many sources that fill in some of these blanks, one of the more interesting I have now seen is at Chasing the Frog.
Congratulations are due as well to the other notable inductees to Saint Joseph’s University’s 9th Athletics Hall of Fame Celebration:
John Butz ’66, Baseball
Bill Campbell, Sportscaster
Tony Costner ’84, Men’s Basketball
Renee Hykel ’01, Women’s Rowing
Christina Kowalski ’00, Softball
Michael McDermott ’97, Golf
Susan Moran ’02, Women’s Basketball
Tom Pippet ’73, Men’s Soccer