The Pitfalls of Meta-Analysis Should be More Widely Recognized and Acknowledged.

The Pitfalls of Meta-Analysis Should be More Widely Recognized and Acknowledged. (from a US government report) Our evidence report draws heavily on six study-level meta-analyses of glucosamine/chondroitin and five of viscosupplementation. While we used a validated instrument to appraise the quality of the systematic reviews, the instrument does not address the question of when meta-analysis is appropriate to a systematic review. Meta-analysis is a technique with underlying assumptions that may or may not hold when a particular collection of results are pooled. Furthermore, metaanalyses may fail to convey the real uncertainty and potential bias accompanying pooled estimates. Uncertainty in the magnitude of effects pooled is influenced by factors intrinsic to the underlying trials. Among these are variable patient characteristics, trial characteristics, and the indication that a few trial results were outliers and influential on pooled estimates. The metaanalyses frequently reported high inter-trial heterogeneity. Random effects models were used in the face of high heterogeneity, but a consequence is to increase the influence of smaller trials on the pooled results. The meta-analyses did not address a threshold question, one that has not been clearly resolved by practitioners of meta-analysis: when is heterogeneity too high to justify pooling trial results. A related concern is the practice of reporting on multiple outcome measures and time intervals, which may be represented by a small portion of studies, thus potentially introducing bias. Evidence Report/Technology Assessment Number 157 Treatment of Primary and Secondary Osteoarthritis of the Knee Prepared for: Agency for Healthcare Research and Quality U.S. Department of Health and Human Services 540 Gaither Road, Rockville, MD 20850 www.ahrq.gov Contract No. 290-02-0026 Prepared by: Blue Cross and Blue Shield Association Technology Evaluation Center Evidence-based Practice Center Chicago, Illinois Investigators David J. Samson, M.S. Mark D. Grant, M.D., M.P.H. Thomas A. Ratko, Ph.D. Claudia J. Bonnell, B.S.N., M.L.S. Kathleen M. Ziegler, Pharm.D. Naomi Aronson, Ph.D. AHRQ Publication No. 07-E012 September 2007

Am I biased?

A reader of my nutrition blog wondered if I may be biased. That's a fair question. Actually, I believe that everyone has to deal with the issue of bias, and the Scientific Method is supposed to help us all overcome these biases by focusing on valid, reproduceable data. Unfortunately, some of us seem to be trying harder than others to fairly represent unbiased data. For one example, my letter published by the cancer journal CA, the Journal of the American Cancer Society http://caonline.amcancersoc.org/cgi/eletters/55/5/319#176 rebutted an article positing that antioxidants should be avoided during cancer therapies. However, none of the references provided in that article showed any evidence of risk! In my rebuttal, I catalogued a number of studies that used nutrients with drugs or radiation therapies, which showed no harmful effects and in some cases even enhanced anticancer effects. I also pointed out the 40% of cancer patients who die of malnutrition while under their doctors' care, much of which may be preventable if physicians actually follow evidence-based medicine instead of clinging to conventional therapies and theories. Click on the title of this article to see the original report and my response. I think that you'll find that I presented relevant published scientific reports to counter a biased opinion that was not even supported by the author's references. How did that crappy opinion even get published in a peer-reviewed journal in the first place?