In our data driven word, you know what’s around the corner. Report cards for doctors based on clinical outcomes. But, statistics is a funny thing. Michael Blastland, a journalist with BBC, made precisely this point with his Go Figure Chance Calculator.
His premise:
“Imagine you are a hospital doctor. Some patients die. But how many is too many before you or your hospital are labeled killers?”
He created a calculator to demonstrate that chance alone might create such an outcome.
Blastland’s acknowledged that different surgeons have different abilities. But, for his thought experiment, he controlled all the rules. He assumed every surgeon had exactly the same judgment and surgical prowess (just like your hospital, right?).
In the first calculator exercise:
- The calculator showed 100 hospitals each performing 100 operations
- The probability that a patient died was initially fixed at five in 100
- The government, meanwhile, states death rates 60% worse than the norm are unacceptable (in red)
- So any hospital which has eight deaths or more out of 100 ops – when the expected average is only five – is in trouble.
By running this statistical exercise a few times, you learn that some hospitals look great; some look like crap. The only thing truly differentiating them is chance. Every time you run this experiment, there are new winners and losers.
If you play around with some of the assumptions, the outcomes change. If the underlying mortality rate for a procedure is much higher, say 12%, very few are 60% worse than the norm.
Make the underlying mortality rate much lower, say 1%, and many more institutions are 60% worse than the norm.
You’ll look closer to the norm if you stick to the most dangerous procedures.
Now, increase the number of procedures done each year ten-fold, chance variation decreases. And vice versa.
Try the calculator yourself.
Of course, there are differences between surgeons and institutions – and such differences may causally impact outcomes.
I understand the motivation to publish report cards on doctors. But, the public has limited understanding of statistics. And, unless the variables are controlled to minimize the effects of “chance”, the results will be uninformative.
Mark Twain once said the citizen who does not read the newspaper is uninformed. The person who reads the newspaper is misinformed.
Imagine then, what the web (which is nothing but newpapers on steroids) can and will do to Medicine. Today’s real issue is which information can be relied upon to make truly informed decisions. It doesn’t matter whether you are a patient or a doctor, which source of information do you trust? Which doctor would you go see if you had condition “X” if you happened to be the patient?
Stratification of hospital quality based on one statistical variable is inherently so flawed that it should never be taken seriously. For example, if risk adjustment is not considered for severity of illness the calculation is meaningless. Also lets not equate a hospital report card with a physician report card. That would be comparing apples to watermelons.