Dangerous pitfalls of database research

By Physician’s Weekly blogger, Skeptical Scalpel

Two papers using the same data about the same topic were published in the same surgical journal one month apart. They came up with completely opposite conclusions.

It happened last year in Surgery, one of the leading general surgery journals.

This incident was described in a JAMA Surgery Viewpoint last week. The subject was Laparoscopic appendectomy—specifically whether the placement of the excised appendix in a retrieval bag before removing it through a small incision results in fewer postoperative infections.

The National Surgical Quality Improvement Program (NSQIP) database for the year 2016 was used in both papers. The numbers of patients included in the studies were 11,475 in what I will call Paper A and 10,357 in Paper B.

Paper A analyzed 10,578 (92.2%) laparoscopic appendectomy patients with the use of a retrieval bag, and 897 (7.8%) without.

The rate of superficial infections was not significantly different. Using multivariable analysis, Paper A found using a retrieval bag was associated with a 2.7% incidence of intra-abdominal abscesses in cases where a bag was used compared to 3.8% without retrieval bags—relatively 40% fewer intra-abdominal abscesses.

Superficial infection rates were not different, but using multivariable analysis, Paper A found a 3.8% incidence of intra-abdominal abscesses when a bag was not used. The rate of abscess formation was 2.7% when a bag was used—a relative 40% drop and a significant difference.

Paper B said a bag was used in 9,585 (92.6%) of patients and not used in 772 (7.5%). Postoperative surgical site (superficial, deep, and organ space) infections occurred in 3.6% of those in whom a bag was used and 4.2% who did not have the specimen placed in a bag prior to its removal from the abdomen. The difference was not statistically significant.

How did both of the studies look at the same data and come up with different results? From the Viewpoint article: “… the studies use different inclusion and exclusion criteria, outcome measures, sample sizes, and covariates. These analytic decisions led to opposite findings.”

The documentation of bag usage in the NSQIP database is derived from operative dictations, which may not always be accurate. For example, Viewpoint authors Christopher P. Childers MD PhD and Melinda Maggard-Gibbons MD MSHS looked at data from their own institution, the David Geffen School of Medicine at UCLA, and found when the operative note did not mention the use of a bag, the perioperative nursing log said a bag was used in 10 of 11 cases.

Abscesses most commonly occur in patients whose appendix has perforated before the operation was performed. Via email, I asked Dr. Childers how a bag could prevent abscess formation in such patients. He said “If you are going to propose that retrieval bags reduce rates of abscess in all types of appendicitis, you have to be able to defend the biologic basis of that hypothesis.”

The editor of Surgery responded to a letter criticizing the journal for having published them both and said Paper B “was reviewed by 5 reviewers, none of whom recognized the similarity of this paper with the previous one.” He acknowledged that he also did not notice the resemblance and accepted the blame.

I asked Dr. Childers if his Viewpoint called into question the value of most if not all database research papers. He said, “In a word, yes.” However, he said there were many groups performing valid health services and database research that can be trusted.

Many statistical programs are user-friendly and analyzing the large amount of data available today is easy. “One can run a regression model, find a positive finding, and publish it,” he said. “Many trainees have just enough competence with statistical software to be dangerous.”

My unscientific Twitter poll found 79.8% of 168 respondents always use a retrieval bag when performing a laparoscopic appendectomy; 1.8% never use a bag; 18.5% say they decide about bag usage depending on the appearance of the appendix at surgery.

The cost of a single use laparoscopic retrieval bag ranges from $50 to $60. At least 250,000 appendectomies are done in the U.S. yearly. Using a bag in every case would come to $12.5 million. It would be nice to know if bags really do prevent infections.

A randomized prospective trial would be a good way to find out, but I doubt an institutional review board would allow it because they might consider it unethical. I think most surgeons would not want to include their patients.

 

Skeptical Scalpel is a retired surgeon and was a surgical department chair and residency program director for many years. He is board-certified in general surgery and a surgical sub-specialty and has re-certified in both several times. For the last 9 years, he has been blogging at SkepticalScalpel.blogspot.com and tweeting as @SkepticScalpel. His blog has had more than 3,700,000 page views, and he has over 21,000 followers on Twitter.