Chat with us, powered by LiveChat

Medical Justice

Making healthcare safe for doctors


A Report Card on Report Cards. Risks of Publicly Reported Surgical Outcomes

10/10/18 10:33 AM

Skeptical Scalpel publishes an excellent blog. He recently tackled the topic of whether the public even understands how to interpret surgical complication rates and surgeon report cards. He graciously gave us permission to distribute his post. Read on. 

Some data is better than no data at all 

Do you believe that?  

I heard it frequently when the infamous Propublica Surgeon Scorecard first appeared three years ago. Back then I blogged about it saying “To me, bad data is worse than no data at all.” 

A recent study in BJU International confirmed my thoughts about this type of publicly posted data and identified a previously unreported issue. The paper attempted to determine whether the public was able to accurately interpret statistics used in the Surgeon Scorecard. It turns out they were not very good at it.  

Investigators from the Department of Urology at the University of Minnesota surveyed 343 people who attended the Minnesota State Fair in 2016. Those who took the survey had a median age of 48, were 60% female, 80% white, and 60% college educated. Their median annual income was $26,550 with an interquartile range of $22,882-$32,587. 

The authors showed individuals the figure below on a tablet computer with the accompanying statement “This graph shows the individual surgeons’ complication rates after 28-35 cases. Surgeons A, B and C raw complication rates are A = 1/35 or 2.9%, B = 1/34 or 3.8% and C = 1/28 or 3.6%.” 

In case you aren’t sure, understand the complication rates for these three surgeons are not significantly different due to the small numbers of cases and complications. 

The most surprising finding of the study was although the surgeons’ complication rates were clearly stated above the figure, just 15.2% of the participants could correctly identify surgeon C’s complication rate. The participants thought the average complication rate for surgeon C was 25% (range 3.6% to 50%). Regarding surgeon B, they were better at estimating the complication rate, but still only 34.9% got it right. 

The subjects were asked multiple-choice questions related to the surgeons’ complication rates. When asked to choose a surgeon for a hypothetical procedure, 192 (56%) picked surgeon A, 30 (8.7%) picked B, and 19 (5.5%) selected C; 102 (29.7%) said they didn’t have enough information to decide. 

Here’s the new wrinkle on the potential harm of misinterpreting data. The subjects were then told that their insurance would only pay if they used surgeon C, and if they wanted to use one of the other surgeons, they would have to pay out of their own pockets. Almost two-thirds said they would pay an average of $5754 in order to have their surgery done by surgeon A or B. 

Those willing to switch were significantly poorer, had a significantly higher incidence of a history of cancer, and misinterpreted the complication rates significantly more often. 

What this means is that the people who could least afford to switch surgeons were the most likely to do so. 

Bottom line: People may misinterpret published data on surgical complication rates which could result in financial harm to them.

What do you think? 


Posted by Medical Justice | in Blog | 4 Comments »

Leave a Reply

4 Comment threads
0 Thread replies
Most reacted comment
Hottest comment thread
4 Comment authors
EasyEJoe HortonRetiredAri Recent comment authors
newest oldest
Notify of

I unfortunately find that virtually anything reported in the lay media that is related to the medical field is rife with mischaracterization, oversimplification, or just rank inaccuracy. Often these attributes may arguably have been deliberately introduced to the story to fan the flames of hysteria and thus promote readership.

Not all complications are equal, nor are all preoperative protoplasms. I agree with you that there is no way that attempting to simplify a statistic like this could result in truly informed patient-doctor selection.


1) There is no way to know for certain, but I seriously doubt that 60% of the survey sample was college educated with a median income of $26,000. The median income reported today for all of WV was $43,000 and WV is one of the poorest states. So right off the bat we have problems with the statistics of the people who were surveyed. 2) As far as the analysis is concerned, there is no way to know if the questions were normed, there is no way to know if surveyor bias was involved since these were presumably done live,… Read more »

Joe Horton
Joe Horton

Each surgeon had a total of one complication. What’s not specified is whether it was early in his experience. But with the lowest possible nonzero numerator, it’s impossible to make a judgment about which surgeon is the best–if one is best–given the number of cases. If one had 1 out of 2, it’s 50%, but that might have been the surgeon who ended up doing the largest number of cases and therefore ended up with the lowest rate over time. It’s obvious that the question were constructed with this in mind. Questions like this are explored in Dan Kaneman’s excellent… Read more »


This survey does not mention any specific procedure, just number of procedures. Complications are more common with complicated patients and their surgeries. I believe this data unfairly discriminates against those who take on the most challenging cases.