False Reassurance Based on Non-Significant Results

A journal article with a title that begins with “There is no …” immediately suggests that there may have been a misinterpretation of a non-significant result from a small study. Non-significant results, expressed either as a p-value greater than 0.05 or a confidence interval (CI) that includes the value representing no effect, are often falsely interpreted as evidence of equivalence [1]. What matters far more than the p-value is whether the CI includes values of clinical importance.

My attention was drawn to the meta-analysis of Amin et al. [2] because of its title. Those authors conducted a systematic review and found 9 studies that examined the outcomes of patients receiving different interventions for transmural endoscopic drainage of pancreatic fluid. Three drainage methods were considered, but, for no obvious reason, only two methods were analyzed: transmural drainage (TM) alone or in combination with transpapillary drainage.

Even if the studies included in the meta-analysis were of high quality, the wide CIs indicated that the results are compatible with a large difference in outcomes. For example, the odds ratio for recurrence was 1.49 in favor of TM, but the 95% CI was 0.53 to 4.21, which is consistent with halving or quadrupling of the risk. Only 6 studies contributed to this analysis, with a total of 34 events.

Further, 6 of the 9 observational studies were found to be of poor quality regarding the comparability of the groups, one of the most fundamental aspects of study quality. Other problems were also identified. The authors used a 9-item version of the Newcastle-Ottawa scale, as against the 7-item version they cited.

I noted that these authors also conducted the largest study included in their meta-analysis [3]. Despite being based on even less data, that study too concluded that transpapillary drainage has no benefit on treatment outcomes when performed in addition to TM.

The authors sensibly call for better, larger studies. Meanwhile, these two strategies for drainage have certainly not been demonstrated to be clinically or technically equivalent, as is wrongly claimed in the titles of both articles.