False Reassurance Based on Non-Significant Results

Article information

Clin Endosc. 2017;50(6):617-617
Publication date (electronic) : 2017 August 18
doi : https://doi.org/10.5946/ce.2017.091
Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
Correspondence: Douglas G. Altman Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, OX2 6UD, Oxford, UK E-mail: doug.altman@csm.ox.ac.uk
Received 2017 June 7; Accepted 2017 July 3.

A journal article with a title that begins with “There is no …” immediately suggests that there may have been a misinterpretation of a non-significant result from a small study. Non-significant results, expressed either as a p-value greater than 0.05 or a confidence interval (CI) that includes the value representing no effect, are often falsely interpreted as evidence of equivalence [1]. What matters far more than the p-value is whether the CI includes values of clinical importance.

My attention was drawn to the meta-analysis of Amin et al. [2] because of its title. Those authors conducted a systematic review and found 9 studies that examined the outcomes of patients receiving different interventions for transmural endoscopic drainage of pancreatic fluid. Three drainage methods were considered, but, for no obvious reason, only two methods were analyzed: transmural drainage (TM) alone or in combination with transpapillary drainage.

Even if the studies included in the meta-analysis were of high quality, the wide CIs indicated that the results are compatible with a large difference in outcomes. For example, the odds ratio for recurrence was 1.49 in favor of TM, but the 95% CI was 0.53 to 4.21, which is consistent with halving or quadrupling of the risk. Only 6 studies contributed to this analysis, with a total of 34 events.

Further, 6 of the 9 observational studies were found to be of poor quality regarding the comparability of the groups, one of the most fundamental aspects of study quality. Other problems were also identified. The authors used a 9-item version of the Newcastle-Ottawa scale, as against the 7-item version they cited.

I noted that these authors also conducted the largest study included in their meta-analysis [3]. Despite being based on even less data, that study too concluded that transpapillary drainage has no benefit on treatment outcomes when performed in addition to TM.

The authors sensibly call for better, larger studies. Meanwhile, these two strategies for drainage have certainly not been demonstrated to be clinically or technically equivalent, as is wrongly claimed in the titles of both articles.


Conflicts of Interest: The author has no financial conflicts of interest.


1. Stang A, Poole C, Kuss O. The ongoing tyranny of statistical significance testing in biomedical research. Eur J Epidemiol 2010;25:225–230.
2. Amin S, Yang DJ, Lucas AL, Gonzalez S, DiMaio CJ. There is no advantage to transpapillary pancreatic duct stenting for the transmural endoscopic drainage of pancreatic fluid collections: a meta-analysis. Clin Endosc 2017;Apr. 17. [Epub]. https://doi.org/10.5946/ce.2016.091.
3. Yang D, Amin S, Gonzalez S, et al. Transpapillary drainage has no added benefit on treatment outcomes in patients undergoing EUS-guided transmural drainage of pancreatic pseudocysts: a large multicenter study. Gastrointest Endosc 2016;83:720–729.

Article information Continued