Cochrane childbirth reviews riddled with statistical errors

Lay people love Cochrane pregnancy and childbirth reviews. They always include plain language summaries, are systematic reviews or meta-analyses that are easy to understand, and are generally written by volunteers, many with an natural childbirth ax to grind.

Doctors are not nearly so enamored of Cochrane pregnancy and childbirth reviews. Although they agree in principle with the aims of the Cochrane project (synthesizing scientific evidence), the reviews are limited by the fact that papers included in a review are often poorly done, underpowered and differ markedly from each other in what results are measured and how they are measured. Moreover, Cochrane Childbirth reviews are often written by self-selected volunteers with an ax to grind, and therefore suffer appear to start with the conclusion and work back to include only papers that support it.

As a general matter, systematic reviews and met-analyses suffer serious limitations, some of which can be overcome with appropriate statistical analysis. However, as a new paper on the Cochrane Childbirth Reviews reveals, most are riddled with serious errors of statistical analysis that render their conclusions suspect or even useless.

Statistical methods can be improved within Cochrane pregnancy and childbirth reviews by Riley, Gates, Neilson, and Alfirevic was published in this month's issue of the Journal of Clinical Epidemiology. Coincidentally, I recently referenced Alfirevic as the author of the Cochrane Review on electronic fetal monitoring (EFM), the review that he acknowledged was underpowered to determine if EFM saves lives.

The intrinsic problems of systematic reviews have been summarized elsewhere as follows:

• There are numerous ways in which bias can be introduced in reviews
and meta-analyses of controlled clinical trials.

• If the methodological quality of trials is inadequate then the findings
of reviews of this material may also be compromised.

• Publication bias can distort findings because trials with statistically
significant results are more likely to get published, and more likely to
be published without delay, than trials without significant results...

• Criteria for inclusion of studies into a review may be influenced by
knowledge of the results of the set of potential studies...

These limitations can be summarized by the pithy phrase "garbage in, garbage out." A meta-analysis or systematic review is only as good as the quality of the papers reviewed.

The Cochrane Childbirth Reviews suffers from these problems and more:

There are deficiencies in the use of statistical methods within the Cochrane Pregnancy and Childbirth Group (CPCG) reviews. The issue of publication bias is
rarely addressed; the process of measuring, investigating, and accounting for heterogeneity is often limited or inadequate; and random-effects analyses are
not correctly interpreted. The large number of metaanalyses per review also raises the concern of multiple testing. These problems need to be urgently
addressed...

Improved use of statistical methods is urgently needed within Cochrane reviews. Although we have only assessed CPCG reviews in the article, our findings have general implications for all Cochrane reviews... The Cochrane Collaboration must seek to engage more statisticians and methodologists within individual reviews ...

The problems identified in the Cochrane Reviews were not limited to a small subset of the reviews. For example, in assessing publication bias, the authors note:

Just 6 (7%) of the 75 reviews stated in their Methods section how they would assess publication bias; only 7 (9%) described a publication bias assessment in their Results or Discussion section or justified why publication bias assessments were not possible; and only 3 reviews described a publication bias assessment plan in their Methods section and subsequently reported an assessment in their Results or Discussion section...

The authors acknowledge that errors such as these seriously limited the validity of Cochrane pregnancy and childbirth reviews:

... It is clear that CPCG reviews must now consider the issue of publication bias in more detail, both when planning their review and when interpreting their results. This is particularly important for their primary analyses, as else misleading or overly strong conclusions may be made...

Unless and until these issues are addressed, Cochrane pregnancy and childbirth reviews will continue to dazzle lay people with incorrect conclusions, and be dismissed by doctors as poorly done and riddled with statistical errors.


This piece first appeared on The Skeptical OB in June 2011.