saturday science session – is it likely that alternative medicine works? the importance of prior probability

The Logic of Science

Presents a series of grounded science based articles covering Evolution, Global Warming, Genetic Modified Organisms (GM’s), Logic, Nature of Science, Vaccines/Alt.Medicines.

saturday science session – is it likely that alternative medicine works? the importance of prior probabilityPrior probability is basically just the plausibility of a result, given everything else we know about the universe.

In other words, when evaluating a hypothesis or a study, you should weight your confidence in the result not only on the characteristics of the study in question, but also on how likely it is that a given result is true given our prior knowledge.

To put that another way, science always builds on previous research, and the results of previous studies can often give us a good idea of whether or not a given result is plausible, before we actually conduct the study. People often act as if ideas and hypotheses exist in a vacuum and should be considered in isolation from everything else, but that isn’t how science works.

It always builds on existing knowledge, and new ideas and results have to be evaluated in light of what we already know. Thus, it is often the case that we know enough about the system in which a hypothesis exists to have a really good idea of whether or not that hypothesis is correct before directly testing it.

In some cases, our knowledge of a system is comprehensive enough that it is not even necessary to directly test the hypothesis.

To give a silly example, imagine someone tells me that pouring Mountain Dew into my car’s engine will improve its performance. Even without actually testing that hypothesis, I can reject it with a very high degree of confidence.

I know enough about how cars work to know that it is almost certainly wrong. In other words, this idea conflicts with what we already know about how cars work, therefore it has a very low prior probability, and there’s really no good reason to bother testing it.

Further, ignoring this suggestion that Mt Dew is good for cars isn’t being “close-minded,” “ideological,” “dismissive,” etc. There’s simply no good reason to think that it is right, and plenty of good reasons to think that it is wrong. Science inherently has to use what we already know as a starting point, and throwing everything that we know out the window to chase absurd suggestions is a substantial waste of time and money.

saturday science session – is it likely that alternative medicine works? the importance of prior probabilityAt this point, I can hear people screaming at their computers that science has to be based on evidence and experimental results, and you’re absolutely correct, but nothing that I have said disagrees with that.

Prior probabilities have to be based on previous research, or else they are going to be very inaccurate probabilities. I’m not suggesting that we arbitrarily decide what is and is not true.

Rather, I am simply pointing out that, thanks to decades of scientific research, we know enough about how the universe works to know that some ideas are patently absurd without directly testing those ideas. Further, keep in mind that science itself is an inherently probabilistic endeavor.

Even after we directly test a hypothesis, we can’t say with 100% confidence whether the hypothesis is true or false. Rather, we can simply say that it is likely true or false based on the existing evidence.

Let me give another trivial example to illustrate this further. As a herpetologist, people often send me blurry photos of reptiles and amphibians that they want me to identify.

Now, imagine someone sends me some blurry pictures that were taken in northern Michigan, and they tell me that they think that these photos show a population of broad-headed skinks (a lizard species).

Without even looking at those pictures, I could say that they almost certainly are not broad-heads, and I could say that, because northern Michigan is well outside the known range of broad-heads. In other words, everything that we know about broad-heads says that they aren’t in Michigan.

Therefore, the prior probability that these photos are of broad-heads is essentially zero. We’d have to be very wrong about our understanding of that species for those photos to be broad-heads.

Nevertheless, it is, of course, always technically possible that our knowledge is in fact wrong, but we’d need far more evidence than some blurry photos before we could reach that conclusion. I’d want to see the lizards themselves and, ideally, test their DNA.

With all of that in mind, let’s talk about “alternative medicine” (aka complementary medicine, aka CAM).

There are countless alternative treatments out there, and while many of them haven’t been studied (or only have a few studies), others have been extremely well-studied and have hundreds of publications (e.g., homeopathy and acupuncture).

Looking at the evidence for these well-studied treatments can, however, be confusing, because while there are tons of studies saying they don’t work, there are also some studies saying that they do, sometimes even including systematic reviews.

There are really good ways to evaluate the studies themselves (details herehereherehere, and here), but prior probability is also quite useful.

Further, evaluating the studies themselves obviously doesn’t help for the treatments that lack studies, but prior probability is still helpful.

that's not how this works memeLet’s take homeopathy as an example. As I’ve written about before (here and here) homeopathy relies on some pretty strange assumptions.

First, it is based on the concept that “like cures like.” In other words, it treats a condition with something else that should cause the symptoms of that condition.

For example, according to homeopathy, since coffee can cause people to have difficulty sleeping, coffee should also be useful for treating people with insomnia, because coffee causes the same symptoms as insomnia (I’m not making this up; homeopaths literally make supposed sleep aids from coffee beans). If that sounds crazy, good. It is crazy. We know a lot about biochemistry and how the human body works, and we know that it doesn’t work that way.

Second, homeopathic treatments are made by doing numerous serial dilutions, with each step making the active ingredient increasingly dilute. According to homeopathy, this works because diluting something actually makes it stronger.

Again, that’s now how things work. One of, if not the, most fundamental concepts in toxicology and pharmacology is that the dose makes the poison. Everything is safe at a low enough dose (i.e., if it is diluted enough) and everything is toxic at a high enough dose. We know this.

So, this concept that diluting something makes it stronger flies in the face of basic chemistry. As I’ve argued before, if you think that diluting something makes it stronger, try diluting some beer and let me know if it takes more or less of it for you to become drunk.

Third, because homeopathic solutions are often so dilute that they literally no longer contain a single molecule of the active ingredient (i.e., they are nothing but water). Homeopathy also claims that water has memory and somehow retains the properties of the active ingredient even though the active ingredient is no longer there.

Again, that’s not how chemistry works. Water doesn’t retain the properties of things it previously came in contact with.

So, when you add it all up, homeopathy is extremely implausible because it requires three different assumptions, each of which disregards basic facts about the universe.

The odds that we are so fundamentally wrong on all three of those topics are very low. Therefore, homeopathy has an extremely, extremely low prior plausibility.

A huge chunk of modern science would have to be wrong for it to be right. This means that we have really good grounds for dismissing it without further investigation.

It also means that we would need some truly extraordinary evidence before we could conclude that it actually works. A few small studies simply won’t cut it. We would need many, massive studies with exquisite experimental designs and a very consistent pattern of positive results among them before we could say that homeopathy works.

Homeopathy has actually been well studied, but when you look at the results of those studies, you don’t find anything even approaching extraordinary evidence.

There are lots of studies that found negative results (i.e., that it doesn’t work) and the studies that found positive results usually had small sample sizes and only found moderate effects. That is simply not sufficient evidence for a topic with such a low prior probability.

Remember, false positives do occur even when a study was conducted correctly. That’s why we need to look for consistent patterns of evidence and consider the prior probability of a given outcome. The less plausible the conclusion, the more consistent and powerful the evidence needs to be.

saturday science session – is it likely that alternative medicine works? the importance of prior probabilityMoving beyond homeopathy, we find the same type of implausible conclusions throughout alternative medicine.

Acupuncture is based on mythical meridians and the pseudoscientific concept that there is good and bad energy.

Similarly, “treatments” like reiki and healing crystals rely on unsubstantiated nonsense about energy and frequencies. Detoxes and cleanses ignore how your liver works, and the acid alkaline diet ignores how homeostasis works.

I can keep going here. Ear candling, magnet therapy, earthing, reflexology, cupping, using chiropracty to treat disease, etc. All of these ignore basic facts about how the human body functions, while making absurd assumptions for which there is no evidence.

Thus, they all have a very, very low prior probability, and we would need some extraordinary evidence before we could conclude that they work, and in the absence of that evidence, we can confidently move them into the rubbish-bin of failed ideas, because they conflict with everything that we already know about physiology, physics, chemistry, medicine, etc.

To be fair, some alternative treatments do have a higher prior plausibility than the treatments I’ve talked about thus far.

Herbs, for example, are kind of a wash when it comes to prior probability. We know that many plants produce chemicals that have effects on our bodies, and we know that sometimes those effects are beneficial.

So, the basic concept behind herbs makes sense, and there is a good prior probability that some of them work. However, that does not mean that there is a high prior probability for any one particular herb.

Basically, what this means is that we need studies before we can conclude that an herb works, but the studies don’t need to be anything extraordinary. A few reasonably large, well-constructed studies are sufficient (assuming that there are consistent results among those studies).

In the absence of those studies, the rational position is to say that we don’t know if a given treatment works, rather than saying that a given treatment probably doesn’t work. Thus, this is a very different situation from something like homeopathy where we would have to be fundamentally wrong about the universe in order for it to work.

Now, at this point, you may be wondering why I am picking on alternative medicine instead of talking about conventional, or so-called “western” medicine (aka medicine).

The answer is simply that our system for developing new drugs and treatments is specifically designed to maximize prior probability before we ever get to human trials.

First, drugs are designed based on a detailed knowledge of biochemistry, which is far better than alternative medicine’s strategy of making a treatment based on anecdotestradition, and, often, superstition. Second, before being tested on humans, new drugs go through in vitro and/or animal trials.

Drugs that fail those tests are deemed to have a low prior probability of working and are discarded, whereas drugs that work safely and effectively in those trials have a higher prior probability of being safe and effective in humans.

To be clear, we still need good studies before we can conclude that they do actually work, but, as I tried to explain earlier, we don’t need the type of monumental, paradigm-shattering evidence that we need for many alternative treatments.

Finally, I can foresee those who believe in alternative medicines responding to this with the comment, “but [insert anecdotes].”

So let me forestall that.

Anecdotes simply are not good evidence of causation and, as such, don’t help you much with prior probability. As I’ve explained at length before, saying, “I took X, then got better, therefore X made me better” is a logical fallacy known as post hoc ergo propter hoc.

It is an invalid line of reasoning.

You could have improved because of regression to the mean, something else you took, chance, another type of placebo effect, etc. Further, you can find anecdotes for literally any treatment.

There are, for example, many people who use anecdotes to argue that you should be drinking bleach and/or turpentine, but, somehow, I doubt that you place much weight on those anecdotes.

Nevertheless, you might try to argue that anecdotes aren’t good evidence of causation, but they do shift the prior probability.

In the case of something like herbs, where no fundamental scientific concepts are being violated, I will grant you that they slightly improve the prior probability, but only to the extent that, if a scientist is going to test some herbs, it makes sense to start with the ones with lots of anecdotes behind them.

I would not agree that anecdotes shift the prior probability enough to be useful for interpreting the results of the subsequent studies.

Further, for things like homeopathy, acupuncture, etc. the prior probability is already so insanely low that anecdotes don’t make any difference.


In short, prior probability is simply the concept of using previous scientific knowledge to assess how likely it is that a given hypothesis or result is actually true.

From that, we can  determine the strength of evidence that is necessary before we reach the conclusion that a given result is correct. If a result is consistent with what we know about the universe and makes sense based on previous research, then it has a high prior probability, and only needs moderately strong evidence before we can conclude that it is likely correct.

In contrast, a result that flies in the face of basic scientific concepts would have a very low prior probability and would require extraordinary evidence before we could accept it as likely being true.

Much of alternative medicine falls into that later category. It often ignores basic facts about science, makes absurd assumptions, and invokes fictitious concepts about energy, frequencies, etc. Indeed, many alternative treatments fly in the face of what we already know about the universe and have such a low prior probability that we can be reasonably confident that they don’t work without actually testing them.

Additionally, although I focused on alternative medicine for this post, the concept of prior probability is widely applicable.

Countless topics like astrology, psychics, tarot cards, etc. have a very low prior probability based on everything we know about the universe.

Therefore, they can be confidently dismissed as nonsense until such time as extraordinary evidence arises in their favor.

Again, to be clear, if that evidence arises, you have to consider it, but you are not required to take the positions seriously or treat them as plausible until extraordinary evidence in their favor is found.

To put that another way, we know enough about the universe to know that something like astrology is almost certainly wrong without actually testing astrology.

This article is written by “Fallacy Man”


The article has been reproduced by kind permission of the publication – “The Logic of Science”