Statistical significance vs. clinical significance

Posted on March 23, 2017

Tags: , , , ,

What if I told you that I conducted a study which shows that a single pill can significantly reduce tiredness without any adverse effects?

Would you try it? Or recommend it? Or would you want more information to decide? Maybe you’re a little skeptical. I will show you the results, so don’t make a decision just yet.

From now on let’s imagine this scenario…

Before I tell you the results of my study, you need to know how it was carried out.

So does that mean the treatment is effective? Should you take “energylina”? Should every doctor prescribe it?

Not necessarily…Let’s make a couple of things clear first. At this point, you might be wondering why the title of this blog is “statistical significance vs. clinical significance”.

Well, I will explain it right now… the results I gave you are there to help you make a decision. You want to know whether “energylina” is effective enough to recommend to individuals who suffer fatigue. Did the results convince you?

Before you answer, first let me clarify something: Clinical significance is the practical importance of the treatment effect, whether it has a real, palpable, noticeable effect on daily life. For example, imagine a safe treatment that could reduce the number of hours you suffered with flu-like symptoms from 72 hours to 10 hours. Would you buy it? Yes, probably! When we catch a cold, we want to feel better, as quickly as possible. So, in simple terms, if a treatment makes a positive and noticeable improvement to a patient, we can call this ‘clinically significant’ (or clinically important).

In contrast, statistical significance is ruled by the p-value (and confidence intervals). When we find a difference where p < 0.05, we call this ‘statistically significant’. Just like our results from the above hypothetical trial. If a difference is statistically significant, it simply means it was unlikely to have occurred by chance. It doesn’t necessarily tell us about the importance of this difference or how meaningful it is for patients.

So it’s important to consider that trial results could be…

Going back to our hypothetical study, what have we got: statistical significance? clinical significance, or both?

Remember we had 2 groups, with 1000 participants in each. In the intervention group, 90% of the participants improved by 2 points on the tiredness scale whereas 80% of the participants in the placebo group improved by 1 point on the tiredness scale.

Is the difference between both groups remarkable? Would you buy my product to have a slightly higher probability of achieving 1 point less on a tiredness scale, compared with taking nothing? Perhaps not. You might only be willing to take this new pill if it were to lead to a bigger, more noticeable benefit for you. For such a small improvement, it might not be worth the cost of the pill. So although the results may be statistically significant, they may not be clinically important.

To avoid falling in the trap of thinking that because a result is statistically significant it must also be clinically important, you can look out for a few things…

  1.  Look to see if the authors have specifically mentioned whether the differences they have observed are clinically important or not.
  2. Take into account sample size: be particularly aware that with very large sample sizes even small, unimportant differences may become statistically significant.
  3. Take into account effect size. In general, the larger the effect size you have, the more likely it is that difference will be meaningful to patients.

So to conclude, just because a treatment has been shown to lead to statistically significant improvements in symptoms does not necessarily mean that these improvements will be clinically significant (i.e. meaningful or relevant to patients). That’s for patients and clinicians to decide.

Cindy Denisse Leyva De Los Rios

Cindy Denisse Leyva De Los Rios

I am 20 years old and I'm a 3rd grade medical student in the Autonomous University of Sinaloa, in Culiacán Sinaloa, México. I love science in all fields, classical music, impressionism art, mystery books, EBM and learning about everything! I want to be a researcher in the oncological, psychiatric or neurological field, but mostly in the oncological field; be part of Cochrane; learn a lot, discover new things and teach everything I know.

More Posts - Website

Related Post

creative commons license
Statistical significance vs. clinical significance by Cindy Denisse Leyva De Los Rios is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. Unless otherwise stated, all images used within the blog are not available for reuse or republication as they are purchased for Students 4 Best Evidence from shutterstock.com.

9 thoughts on “Statistical significance vs. clinical significance

  1. Pradnya Kakodkar

    Very well explained. This is the same topic on which i am working. we all get carried away by the p value least realizing the importance of clinical or practical significance.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *