A network for students interested in evidence-based health care

The Evidence Based Medicine problem and what to do about it

Posted on 18th September 2014 by

Tutorials and Fundamentals
evidence based medicine problems

Key message:  Evidence Based Medicine is useful for informing healthcare professionals what works, what doesn’t, and helping to determine if the benefits outweigh the harms, but it’s far from perfect. There are valuable lessons learned about research that we can share across disciplines.

What is the Evidence Based Medicine problem?

In 2005, Dr. John Ioannidis, a well-known meta-researcher, published an article in PLoS Medicine called Why Most Published Research Findings Are False. This article caused a splash and has been making waves in the medical research community ever since.  His paper is a bit technical, go for it if you can, but I recommend everyone at least read the less technical, narrative write-up about his research here in The Atlantic.   He raises a number of very serious issues that have plagued the medical research community. I’ll try to summarize some of his concerns.

They include:

  • Publication bias, which means that if researchers find that an intervention had little or no effect, then those studies are less likely to get published. In other words, not publishing negative findings.
  • Overconfidence of studies that rely on statistical p-values of 0.05. This could lead to false positives as well as false negatives.
  • Lack of replication since many important studies are not repeated, or even ‘repeatable’, by other independent researchers to verify the results. Some major studies that have been replicated have found surprisingly different and even contradictory results.
  • Small study size and underpowered methodologies. The lower the number of subjects in the study, the less likely the results are true.
  • Selective outcome reporting, manipulation of data/fraud and financial conflicts of interest. The greater the financial interest, the less likely the study will be true.
  • The greater the flexibility in designing studies and in definitions, the less likely the research findings are to be true.
  • The reward systems within medical research, particularly in academia, incentivizes quantity of publications over quality of research.

These factors collectively have led Dr. Ioannidis to conclude that a large part of the evidence that doctors and healthcare providers have come to rely on, including major foundational studies used to treat patients, are frequently misleading, exaggerated, and often flat-out wrong.

When I first read his article, I was a bit shocked and in disbelief that Evidence Based Medicine could possibly be this wrong. The article left me begging the question, what now?

What can we do about this Evidence Based Medicine problem? How do we produce better research?

Thankfully, since his original article was published, many research and academic groups have made progress in several of these areas. In a 2014 JAMA article, Dr. Ioannidis returns to add suggestions for additional solutions, particularly in regards to changing the reward system from prioritizing quantity of research over quality of design.  He recommends a list of reward criteria, or principles, to help appraise and identify desirable research methods.  He calls it the “PQRST” which stands for Productive, Quality, Reproducible, Shareable and Translatable.

Let’s break this down.

Productive: This basically means setting a fixed definition as to what it means to be ‘productive’ in research. For example, the number of publications in top tier journals, % of citations for each scientific field per year. Not just simply to publish something somewhere for the sake of getting published.

Quality: This means setting high publication standards as appropriate in each field for research methods and study designs. This is important to ensure increased reliability and credibility of results. These standards should also be easily verifiable.

Reproducible: This means making sure the raw data and methods are clear, so other independent researchers can (and should) reproduce the study.

Sharable: This means registering and sharing of data materials and protocols of all trials.

Translatable: This means ensuring the research is relevant and can be applied in real-life settings.

The challenge in reproducing or reanalyzing previous studies was featured in the latest issue of JAMA and discussed in Richard Lehman’s BMJ blog. In the article, called “REanalyses of Randomized Clinical Trial Data, Dr. Ebrahim, who works as part of Dr Ioannidis’ team of researchers, found that only:

 “A small number of reanalyses of RCTs have been published to date. Only a few were conducted by entirely independent authors. Thirty five percent of published reanalyses led to changes in findings that implied conclusions different from those of the original article about the types and number of patients who should be treated”

So we still have a long way to go. The good news is that research and efforts on improving Evidence Based Medicine is ongoing. It seems as though we are making progress by identifying weaknesses and addressing them.

Conclusions & comment

I’d like to point out that Alice Buchan, an S4BE Pioneer, wrote a wonderful piece earlier this year based on a series of Lancet articles about increasing value and reducing waste in research here. Waste in research is, of course, a related topic and there have been some great ideas on how to improve priorities as well as reduce waste in research.

As Students 4 Best Evidence, we represent a variety of different medical disciplines. We all value research and evidence as part of the clinical decision making process. Perhaps these articles should give us pause about the state of our evidence and help us think about possible solutions for the so called “Evidence Based Medicine problem”.

What do these challenges mean for the state of research and evidence in our own respective disciplines? Have you seen any of these issues raised or solutions implemented in your field? How can we as students implement these ‘lessons learned’ into our respective fields and influence others to do so?

Links/References:

David H. Freedman (November, 2010) Lies, Damned Lies, and Medical ScienceThe Atlantic

Ebrahim S, Sohani ZN, Montoya L, et al. “REanalyses of Randomized Clinical Trial Data.” JAMA 312, no. 10 (September 10, 2014): 1024–32. doi:10.1001/jama.2014.9646.

Ioannidis, John P. A. “Why Most Published Research Findings Are False.” PLoS Med 2, no. 8 (August 30, 2005): e124. doi:10.1371/journal.pmed.0020124.

Ioannidis JA, and Khoury MJ. “Assessing Value in Biomedical Research: The Pqrst of Appraisal and Reward.” JAMA 312, no. 5 (August 6, 2014): 483–84. doi:10.1001/jama.2014.6932.

Young, Neal S, John P. A Ioannidis, and Omar Al-Ubaydli. “Why Current Publication Practices May Distort Science.” PLoS Med 5, no. 10 (October 7, 2008): e201. doi:10.1371/journal.pmed.0050201.

Tags:

Danny Minkow

Danny Minkow has a History degree from the University of Florida. He then served with the US Air Force for 7 years and is an Iraq War veteran. He is currently pursuing a Master's of Science in Occupational Therapy at Brenau University near Atlanta, Georgia. Danny has great interest in The Cochrane Collaboration and how medical evidence is used and communicated. Particularly the differences between how the UK and US view and apply the best available evidence. He is currently working on a group thesis for stroke rehabilitation. View more posts from Danny

Leave a Reply

Your email address will not be published. Required fields are marked *

No Comments on The Evidence Based Medicine problem and what to do about it

  • Padmavathy S Menon

    Very well written. I wonder sometimes it is evidence based or evidence driven medicine.

    28th July 2019 at 4:44 pm
    Reply to Padmavathy
  • Emma Carter

    Hello – you can subscribe to the website by following this link and providing your details. You will then receive through the monthly newsletter to keep you updated. Many thanks, Emma

    18th April 2018 at 9:55 am
    Reply to Emma
  • panacea

    Hello, I would like to subscribe for this blog to get newest updates, thus where can i do it please
    assist.

    13th April 2018 at 6:39 pm
    Reply to panacea

Subscribe to our newsletter

You will receive our monthly newsletter and free access to Trip Premium.