Posted on November 3, 2017
Assumptions for normal meta-analysis methods do not hold true in these scenarios, as there can be no events in one or both of the comparison arms. This is often the case with serious but uncommon adverse events, making it essential to get the analysis method correct as studies have shown that using a different meta-analysis method can change the final effect estimate considerably(1).
The Cochrane handbook has a separate chapter on special statistics (chapter 16), which contains a devoted section on reviews dealing with rare events (section16.9). Though it discusses the validity of the different methods available within the literature, the accompanying RevMan software only permits the use of Mantel-Haenszel odds ratio (OR) method using a 0.5 zero-cell corrections or Peto’s OR, as also acknowledged within the manual(2).
The Mantel-Haenszel OR, using the 0.5 zero-cell corrections, have been repeatedly shown to give biased results (2,3) while the Peto´s OR has generally been considered robust when there is not a large imbalance in the total group size between the different comparison arms, and the expected effects are not large (1-3). This has meant the Peto OR has become the method of choice for most.
However, if the latter assumptions are not met, which is often the case, then the results have a higher potential to be biased. Moreover, the Peto OR model is unable to handle studies where both the arms have no events, and effectively removes them from the analysis. Therefore this further limits its applicability and ethically, patients who have been recruited to these double-zero studies, have a right that their data is included in meta-analyses (4-6).
We therefore undertook a study to identify the validity and robustness of effect estimates across different meta-analysis methods for rare binary events data, using the example of serious rare adverse events from antidepressants trials. We compared the four rare adverse outcomes of all-cause mortality, suicidality, aggressive behaviour and akathisia across the Peto method, the generalized linear mixed model (GLMM), conditional logistic regression, a Bayesian approach using Markov Chain Monte Carlo (MCMC) method and finally the beta-binomial method.
Our results, recently published online (7) in the Journal of Clinical Epidemiology, showed that though the estimates for the four outcomes did not change substantially across the different analysis methods, the Peto method underestimated the treatment harm and overestimated its precision, especially when the estimated OR deviated greatly from 1. For example, the OR for suicidality for children and adolescents was 2.39 (95% CI 1.32 to 4.33) using the Peto method, but increased to 2.64 (1.33 to 5.26) using conditional logistic regression, to 2.69 (1.19 to 6.09) using beta-binomial, to 2.73 (1.37 to 5.42) using the GLMM and finally to 2.87 (1.42 to 5.98) using the MCMC approach. Moreover when we consider absolute numbers and that these are very serious harms, the minor changes in the effect estimates make a difference.
Therefore, the method used for meta-analysis of rare events data influences the estimates obtained, and the exclusion of double zero-event studies can give misleading results. To ensure reduction of bias and erroneous inferences, sensitivity analyses should be performed using different methods. Other methods, in particular the beta-binomial method that was shown to be superior through simulation studies (4,7), should be considered as an appropriate alternative.