Strategy word shown behind a magnifying glass

Comparing the validity and robustness of different statistical methods for meta-analysis of rare event data

Posted on November 3, 2017

Tags: , , , , , ,

Meta-analyses of rare events or sparse data should employ different methods compared to regular data.

Assumptions for normal meta-analysis methods do not hold true in these scenarios, as there can be no events in one or both of the comparison arms. This is often the case with serious but uncommon adverse events, making it essential to get the analysis method correct as studies have shown that using a different meta-analysis method can change the final effect estimate considerably(1).

The Cochrane handbook has a separate chapter on special statistics (chapter 16), which contains a devoted section on reviews dealing with rare events (section16.9). Though it discusses the validity of the different methods available within the literature, the accompanying RevMan software only permits the use of Mantel-Haenszel odds ratio (OR) method using a 0.5 zero-cell corrections or Peto’s OR, as also acknowledged within the manual(2).

Mantel-Haenszel vs Peto odds ratio

The Mantel-Haenszel OR, using the 0.5 zero-cell corrections, have been repeatedly shown to give biased results (2,3) while the Peto´s OR has generally been considered robust when there is not a large imbalance in the total group size between the different comparison arms, and the expected effects are not large (1-3). This has meant the Peto OR has become the method of choice for most.

However, if the latter assumptions are not met, which is often the case, then the results have a higher potential to be biased. Moreover, the Peto OR model is unable to handle studies where both the arms have no events, and effectively removes them from the analysis. Therefore this further limits its applicability and ethically, patients who have been recruited to these double-zero studies, have a right that their data is included in meta-analyses (4-6).

We therefore undertook a study to identify the validity and robustness of effect estimates across different meta-analysis methods for rare binary events data, using the example of serious rare adverse events from antidepressants trials. We compared the four rare adverse outcomes of all-cause mortality, suicidality, aggressive behaviour and akathisia across the Peto method, the generalized linear mixed model (GLMM), conditional logistic regression, a Bayesian approach using Markov Chain Monte Carlo (MCMC) method and finally the beta-binomial method.

Our results, recently published online (7) in the Journal of Clinical Epidemiology, showed that though the estimates for the four outcomes did not change substantially across the different analysis methods, the Peto method underestimated the treatment harm and overestimated its precision, especially when the estimated OR deviated greatly from 1. For example, the OR for suicidality for children and adolescents was 2.39 (95% CI 1.32 to 4.33) using the Peto method, but increased to 2.64 (1.33 to 5.26) using conditional logistic regression, to 2.69 (1.19 to 6.09) using beta-binomial, to 2.73 (1.37 to 5.42) using the GLMM and finally to 2.87 (1.42 to 5.98) using the MCMC approach. Moreover when we consider absolute numbers and that these are very serious harms, the minor changes in the effect estimates make a difference.

Conclusion

Therefore, the method used for meta-analysis of rare events data influences the estimates obtained, and the exclusion of double zero-event studies can give misleading results. To ensure reduction of bias and erroneous inferences, sensitivity analyses should be performed using different methods. Other methods, in particular the beta-binomial method that was shown to be superior through simulation studies (4,7), should be considered as an appropriate alternative.

References

  1. Sweeting M, Sutton A, Lambert P. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Statistics in Medicine 2004; 23(9): 1351-75.
  2. Higgins JG, S (editors). Cochrane Handbook for Systematic Reviews of Interventions [updated March 2011]. Available from www.cochrane-handbook.org.: The Cochrane Collaboration; 2011, (accessed May 2016).
  3. Bradburn M, Deeks J, Berlin J, Russell Localio A. Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events. Statistics in Medicine 2007; 26(1): 53-77.
  4. Kuss O. Statistical methods for meta-analyses including information from studies without any events—add nothing to nothing and succeed nevertheless. Statistics in Medicine 2015; 34(7): 1097-116.
  5. Friedrich JO, Adhikari NK, Beyene J. Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data. BMC Medical Research Methodology 2007; 7(1): 1-6.
  6. Keus F, Wetterslev J, Gluud C, Gooszen HG, van Laarhoven CJHM. Robustness Assessments Are Needed to Reduce Bias in Meta-Analyses That Include Zero-Event Randomized Trials. Am J Gastroenterol 2009; 104(3): 546-51.
  7. Sharma T, Gøtzsche PC, Kuss O. The Yusuf-Peto method was not a robust method for meta-analyses of rare events data from antidepressant trials. Journal of Clinical Epidemiology. 2017; Aug 9. pii: S0895-4356(17)30785-0.  doi: 10.1016/j.clinepi.2017.07.006. 
Tarang Sharma

Tarang Sharma

Tarang Sharma is a PhD Fellow at the University of Copenhagen in Public Health and Epidemiology, a Researcher, at the Nordic Cochrane Centre and a consultant with the Knowledge Management, Evidence and Research for Policy-Making unit at the World Health Organization Regional Office for Europe, in Copenhagen, Denmark. Previously she has worked as a Health Economist for Novo Nordisk A/S, as a Senior Analyst for the National Institute for Health and Care Excellence (NICE) in the UK and as a Research Associate for the Health Economics and Decision Science unit at the University of Sheffield, amongst other roles. Her research focuses on improving systematic review methodology especially in relation to adverse effects and publication bias. She is interested in the translation of evidence into policy and practice and the use of patient-reported outcomes, patient experience and patient involvement in research and decision-making.

More Posts - Website

Related Post

creative commons license
Comparing the validity and robustness of different statistical methods for meta-analysis of rare event data by Tarang Sharma is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. Unless otherwise stated, all images used within the blog are not available for reuse or republication as they are purchased for Students 4 Best Evidence from shutterstock.com.

Leave a Reply

Your email address will not be published. Required fields are marked *