How Do We Know What Treatments Work?
How Do We Know What Treatments Work?December 18th, 2017 by Cindy Perlin
We’re often told that alternative medicine is unproven and pharmacological treatments are well validated. Is this true?
It’s important for patients and health care providers to understand the many factors that go into creating the body of medical evidence available, the degree of reliability of that information, and its applicability to their specific situation. I’ve included a brief discussion of research issues here to meet that need.
Since the early 1990s, there has been a push from many quarters for “evidence-based medicine” (EBM). According to the British Medical Journal, “The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.”[i]
According to EBM, all research is not created equal. The best evidence is considered to be meta-analyses of randomized clinical trials. A meta-analysis statistically pools results from many similar studies to draw conclusions about treatment effectiveness.
Second best, according to EBM, is evidence obtained from at least one randomized controlled clinical trial. A randomized controlled trial is a study in which participants are assigned, by chance alone, to receive one of several interventions. One of these interventions is the comparison, or control. The control may be a standard treatment, a placebo (a sugar pill or fake treatment), or no intervention at all.
Double-blind randomized controlled studies are held in even higher regard. In a double-blind study, no one—patient, researcher, or any other results evaluator—knows which participants received which treatment. This ensures that no biases or expectations will influence results. Of course, double-blind studies are not possible when the treatment is anything more complicated than a pill, as practitioners administering the treatment must know what they’re doing.
Nonrandomized studies and observational studies are considered less conclusive. Expert opinion is considered the lowest level of evidence for making treatment decisions.[ii]
In the EBM model, the evidence that comes from thousands of years of use and observation by healers across many cultures, such as in the use of acupuncture or herbs, is not considered valid, whereas results from a four- to eight-week study of a novel drug is given one of the highest rankings.
Flaws in the Research
EBM sounds good in theory, but its application to medical decision-making has left much to be desired. A 2014 article in the Journal of Evaluation in Clinical Practice reported that EBM has failed to achieve its main objectives: to improve health care outcomes and reduce health care costs. Summarizing critics’ concerns about EBM, authors Susanna Every-Palmer and Jeremy Howick wrote that one issue of major concern is that most studies available in medical literature are funded by parties that have a financial interest in the outcome, such as pharmaceutical companies. These entities manipulate the outcome of the research by their choice of issues studied and by manipulating the study design to favor their desired results. When the study still does not turn out as they had hoped, they fail to publish it and bury the results. Howick and Every-Palmer concluded that relying indiscriminately on industry-funded studies to make clinical decisions is like trusting politicians to count their own votes.[iii]
Howick and Every-Palmer cited the evidence for selective serotonin re-uptake inhibitor (SSRI) antidepressants as an example. More than 1,000 randomized trials of SSRI antidepressants have been conducted, resulting in seemingly overwhelming evidence that they provide clinically significant benefits.[iv] Doctors and patients were convinced that SSRI antidepressants (including Paxil and Prozac) worked. These blockbuster drugs posted global sales in 2011 of $11.9 billion.[v] SSRIs replaced older antidepressants and psychotherapy as the treatment of choice for depression. However, two independent analyses, published in 2008 and 2010, which included all published and unpublished studies on SSRI antidepressants, found that only favorable studies, and those that could be doctored to look favorable, were published. The pooled data from all studies showed that SSRIs are no more effective than placebo for the treatment of mild to moderate depression.[vi],[vii]
A review of pharmaceutical company-funded studies that compared newer, atypical antipsychotics—touted as safer and more effective than the older generation—revealed that the drug produced by the company funding the study was found to be superior 90% of the time.[viii] Studies were designed to make the study sponsor’s drug appear superior by such means as setting the dose of the competing drug too low to be effective or so high that the side effects would be intolerable.[ix]
Low-dose studies of natural treatments have also been funded by parties with an interest in discrediting competing treatments. This was the case with biofeedback, a treatment in which sensitive electronic instruments are used to measure the patient’s physiology. That information is used to teach the patient to control her physiology and resolve her symptoms without medication. Biofeedback is used to treat chronic pain, headaches, high blood pressure, anxiety, insomnia, and many other conditions. The 1986 book From the Ghost in the Box to Successful Biofeedback Training documented key “errors” researchers made when conducting these studies.[x] One research methodology flaw involved an insufficient number of training sessions. Successful biofeedback treatment requires the mastery of complex skills, which is highly unlikely in the limited number of training sessions allowed by many studies.[xi] Of 167 studies that took place before 1982 that used biofeedback to teach voluntary control of heart rate, 75% used only one to three sessions of biofeedback training.[xii] Other research flaws included insufficient length of training sessions; lack of homework exercises (standard in most types of biofeedback treatment); failure to instruct subjects that the goal was to learn to control their responses; failure to provide adequate rationales, instructions, and coaching; use of a relaxation control group for comparison to biofeedback training (biofeedback is relaxation training); and failure to train to mastery.[xiii] While initially biofeedback generated a great deal of excitement, after these flawed studies were published, the interest substantially waned.
Failure to disclose the risks of pharmaceuticals is another issue with published research conducted by the drug’s manufacturer. Pharmaceutical giants GlaxoSmithKline, Johnson & Johnson, Eli Lilly, and AstraZeneca have all been fined by the Food and Drug Administration (FDA) for hiding adverse effects of their products to make them more marketable.[xiv],[xv],[xvi],[xvii],[xviii] More specific to pain research, Purdue Pharmaceuticals was charged by the federal government with a criminal count of misbranding the narcotic OxyContin (oxycodone), with intent to defraud, and falsely promoting it as nearly addiction-proof.[xix]
In 2009, the world of pain medicine was rocked by the revelation that well-known anesthesiologist and researcher Scott Reuben had fabricated data in at least 21 published studies. Having received five research grants from Pfizer to study its drug Celebrex (celecoxib), Reuben went so far in some of the “studies” as to not bother enrolling patients and to make up all of the data. His findings naturally favored Pfizer. These falsified findings influenced surgeons and the way they treated postoperative pain all over the world, affecting the care of millions of patients.[xx] Reuben pled guilty to engaging in health care fraud. He was sentenced to six months in prison followed by three years of probation, a $5,000 fine, restitution of $361,932, and forfeiture of $50,000.[xxi] It’s unclear how much harm was done to patients by Reuben’s fabrications, but experts believe that implementation of his recommendations may have slowed surgical recovery.[xxii]
Another problem with the available body of evidence is that after research has been discredited, it is not retracted or labeled as fraudulent. The research studies remain available for the unwary to draw false conclusions regarding treatment safety and effectiveness.[xxiii]
The prevailing research models are not appropriate for many alternative therapies, such as acupuncture or homeopathy, in part because treatments are not the same for every patient who presents for treatment of a specific condition. The choice of intervention is based on an assessment of the whole person, not just the presenting symptoms.[xxiv] As a result, studies that give every patient with back pain, for instance, the same intervention are not valid measures of the effectiveness of these therapies.
Another related issue is the averaging of results in a study. What does or does not work for most people with a condition is not necessarily applicable to every individual in a study or in the overall population. Patients are individuals with different characteristics and combinations of symptoms, which might have different causes. Treatment decisions that don’t take individual differences into account are less likely to be effective and can cause harm.[xxv]
Bias in Funding
Another significant problem in evaluating evidence is that there is an uneven playing field when it comes to funding for research. Pharmaceutical companies, which can earn tens of billions of dollars in profit from patenting a single successful drug, can easily afford the hundreds of thousands of dollars it takes to conduct a drug study. Companies that sell foods, vitamins, or herbs that cannot be patented don’t have that kind of money to invest in research that will benefit all companies selling a similar product. Health care providers who provide hands-on treatment—such as chiropractic, massage, or psychotherapy—are even more financially constrained, as they are limited to charging for their services by the hour. They cannot possibly generate enough extra income from their clinical work to fund a large research study.
Adding to the problem is the limited government funding available for medical research, and the fact that this funding rarely goes to those who challenge either the prevailing paradigm or those with power and influence.
What Can Patients and Providers Do?
We need to recognize that claims of pharmaceutical safety and effectiveness are often overblown and the benefits of alternative therapies are often minimized. Alternative therapies are much safer than pharmaceuticals and, if accessible and affordable, are worth trying first.
My book, The Truth About Chronic Pain Treatments: The Best and Worst Strategies for Becoming Pain Free evaluates the evidence for a broad range of conventional and alternative treatments for chronic pain. My new online Alternative Pain Treatment Directory also provides information and resources for pain patients.
Cindy Perlin is a Licensed Clinical Social Worker, certified biofeedback provider and chronic pain survivor who lives and works in the Albany, NY area.
[i] Sackett D. L., Rosenberg W. M. C., Gray J. A. M., Haynes R. B., & Richardson, W. S. (1996) Evidence based medicine: What it is and what it isn’t. The BMJ, 312, 71-2.
[ii] U.S. Agency for Health Care Policy and Research guidelines
[iii] Every-Palmer, S. & Howick, J. (2014) How evidence-based medicine is failing due to biased trials and selective publication. Journal of Evaluation in Clinical Practice, 20, 908-14.
[iv] Ioannidis, J. P. (2008) Effectiveness of antidepressants: an evidence myth constructed from a thousand randomized trials? Philosophy, Ethics and Humanities in Medicine, 3, 14.
[v] CBI Research. (2012) Antidepressants market to 2018–Despite safety concerns, selective serotonin re-uptake inhibitors (SSRIs) continue to dominate in the absence of effective therapeutic alternatives. http://www.gbiresearch.com/Report.aspx?ID=Antidepressants-Market-to-2018-Despite-Safety-Concerns
[vi] Kirsch, I., Deacon, B. J., Huedo-Medina, T. B., Scoboria, A., Moore, T. J. & Johnson, B. T. (2008) Initial severity and antidepressant benefits: a meta-analysis of data submitted to the food and drug administration. PLoS Medicine, 5, e45.
[vii] Fournier, J. C., DeRubeis, R. J., Hollon, S. D., Dimidjian, S., Amsterdam, J. D., Shelton, R. C. & Fawcett, J. (2010). Antidepressant drug effects and depression severity: A patient level meta-analysis. Journal of the American Medical Association, 303, 47–53.
[viii] Heres, S., Davis, J., Maino, K., Jetzinger, E., Kissling,W. & Leucht, S. (2006). Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: an exploratory analysis of head-to-head comparison studies of second-generation antipsychotics. American Journal of Psychiatry, 163,185–94.
[ix] Smith, R. (2005) Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Medicine, 2 (5), e138.
[x] Shellenberger, R. and Green, J. A. (1986). From the Ghost in the Box to Successful Biofeedback Training. Greely, CO: Health Psychology Publications.
[xi] Shellenberger & Green. (1986). From the Ghost in the Box to Successful Biofeedback Training.
[xii] Banderia, M., Bouchard, M., & Granger L. Voluntary control of autonomic responses: A case for a dialogue between individual and group experimental methodolgies. Biofeedback and Self-Regulation, 7, 317-329.
[xiii] Shellenberger & Green. (1986). From the Ghost in the Box, 15-64.
[xiv] Roehr, B. (2012). GlaxoSmithKline is fined record $3 billion in US. British Medical Journal, 345, e4568.
[xv] Lenzer, J. (2006). Manufacturer admits increase in suicidal behaviour in patients taking paroxetine. British Medical Journal, 332(7551), 1175.
[xvi] Department of Justice. (2012) GlaxoSmithKline to plead guilty and pay $3 billion to resolve fraud allegations and failure to report safety data. http://www.justice.gov/opa/pr/2012/July/12-civ-842.html.
[xvii] Kmietowicz, Z. (2012). Johnson & Johnson to pay $2.2 bn to settle charges of false marketing on three drugs. British Medical Journal, 347, f6696.
[xviii] Tanne, J. H. (2012). US judge fines Johnson & Johnson $1.1 bn for misleading marketing of risperidone. British Medical Journal, 344, e2772.
[xix] Tanne. (2012). US judge fines Johnson & Johnson.
[xx] Borrell, B. (2009, March 10). A medical Madoff: Anesthesiologist faked data in 21 studies. Scientific American,
[xxi] Press release, FDA Office of Criminal Investigations/US Department of Justice. Anesthesiologist sentenced on health care fraud charge, June 24, 2010. http://www.fda.gov/ICECI/CriminalInvestigations/ucm217302.htm on 8/10/15.
[xxii] Borrell. (2009). A medical Madoff.
[xxiii] Every-Palmer & Howick. (2014). How evidence-based medicine is failing.
[xxiv] Researching Integrative Medicine: Challenges and Innovations available at http://exploreim.ucla.edu/research/researching-integrative-medicine-challenges-and-innovations/#funding
[xxv] Hartzband, P. & Groopman, J. (2014, November 18). How medical care is being corrupted. New York Times,