The Pitfalls of Meta-Analysis Should be More Widely Recognized and Acknowledged.

The Pitfalls of Meta-Analysis Should be More Widely Recognized and Acknowledged. (from a US government report) Our evidence report draws heavily on six study-level meta-analyses of glucosamine/chondroitin and five of viscosupplementation. While we used a validated instrument to appraise the quality of the systematic reviews, the instrument does not address the question of when meta-analysis is appropriate to a systematic review. Meta-analysis is a technique with underlying assumptions that may or may not hold when a particular collection of results are pooled. Furthermore, metaanalyses may fail to convey the real uncertainty and potential bias accompanying pooled estimates. Uncertainty in the magnitude of effects pooled is influenced by factors intrinsic to the underlying trials. Among these are variable patient characteristics, trial characteristics, and the indication that a few trial results were outliers and influential on pooled estimates. The metaanalyses frequently reported high inter-trial heterogeneity. Random effects models were used in the face of high heterogeneity, but a consequence is to increase the influence of smaller trials on the pooled results. The meta-analyses did not address a threshold question, one that has not been clearly resolved by practitioners of meta-analysis: when is heterogeneity too high to justify pooling trial results. A related concern is the practice of reporting on multiple outcome measures and time intervals, which may be represented by a small portion of studies, thus potentially introducing bias. Evidence Report/Technology Assessment Number 157 Treatment of Primary and Secondary Osteoarthritis of the Knee Prepared for: Agency for Healthcare Research and Quality U.S. Department of Health and Human Services 540 Gaither Road, Rockville, MD 20850 www.ahrq.gov Contract No. 290-02-0026 Prepared by: Blue Cross and Blue Shield Association Technology Evaluation Center Evidence-based Practice Center Chicago, Illinois Investigators David J. Samson, M.S. Mark D. Grant, M.D., M.P.H. Thomas A. Ratko, Ph.D. Claudia J. Bonnell, B.S.N., M.L.S. Kathleen M. Ziegler, Pharm.D. Naomi Aronson, Ph.D. AHRQ Publication No. 07-E012 September 2007