Being a new blogger, I’ve been struggling to develop an approach for handling comments in a consistent and sustainable way. Not that I have had many trolls on this blog or the Thinking Person’s Guide to Autism, but I find myself wanting to quickly respond to each and every criticism. If this blog gains the readership that I hope it does someday, that is not a sustainable approach. Further, it is not even efficient as most blog criticisms focus around one or two common points, which could be addressed in one place. So my approach is, at least for now, to take such common criticisms and address them as a new blog entry when I think there is enough meat there to do so and the topic is interesting enough as well. This is the first such post.
The most common criticism I saw on my listening therapy post is one that I see on many skeptical blogs: “but it works for me”. Often, when I express skepticism of treatments with family and friends, I get a similar response. So I think this is an important criticism to address, and it highlights a critical difference between how skeptics and believers think.
At the heart of the matter is epistemology, or more simply, the rules for how we decide what we believe. The rules of evidence used by skeptics are a critical difference between skeptics and non-skeptics, if not “the” critical difference. So when someone says to me that “ listening therapy works for me”, I think “how do you know”. Or even better, I should ask “how could you know”. This is not a criticism of their intelligence, but of the data available to them and the ability to draw conclusions from it. Further, I am nothing if not ecumenical. I recognize that if I try something like Airborne® when I have a cold, that I cannot afterwards make any conclusions about whether it has helped me or not. All I can say is that I got better from the cold, not a very astonishing fact in and of itself. This is also why I find such demands that I “try” some treatment before criticizing it to be a pointless endeavor (unless it is just a question about a matter of taste). My single patient non-experiment could not tell me anything. In fact, self experimenting introduces all sorts of additional bias if anything.
So is all hope lost? Am I advocating classical Pyrrhic skepticism reflected in current postmodernism, where all hope of knowledge eludes us. Hardly, but that is often the simple caricature of skeptics used in strawman arguments against their position. No, there is solution to this dilemma, and its name is science. If I were forced to give a one sentence definition of science, it would be that science is the collection of tools to address human biases and investigate the natural world as objectively and systematically as possible. And before the other common criticism of skeptics is posted, I am not arguing for scientism or the position that science can answer all questions. I am simply saying that reason, using the tools of science and logic, are the best we’ve got when it comes to understanding the natural world.
So what are my thoughts when someone comes to me and says “treatment X worked for me or some of my patients”? What else were they doing? What would have happened if they did nothing; could this just be coincidental? How did their expectations color their perception? How susceptible was the outcome to personal perception? Did the person giving the treatment have expectations that could affect their analysis?
None of these issues can really be addressed in a single person, uncontrolled, self-experiment. In fact, such a method violates every single principle of a randomized, controlled and double-blinded experiment, the gold standard, which was created to address precisely the questions above.
Being only one person, there is no statistical significance and chance could sufficiently explain it. Of course, it also means there is no control group which gets to the question of what would happen if you simply had let nature run its course? To answer that question, we need a treatment and non-treatment group of statistically significant size. If we want to get at causation and not simply correlation, we also need to randomly assign people to the two groups. This gets at the question of what else were they doing? Perhaps it was another treatment or simple consequence of the treatment (like resting more) that really caused the recovery. And when you experiment on yourself neither the experimenter nor experimentee is blinded. Our perception, especially of pain, is colored by expectations. Similarly, the experimenter’s expectations can subtly affect their analysis or influence the one being experimented upon. In the end, you could hardly create a worse experiment for effectiveness of a treatment than the “just try it yourself” approach.
Does this mean that everything in life needs rigorous scientific experimentation to establish its validity? No. For immediate, dramatic, repeatable and objectively definable results, we can often infer causality quite easily. For example, a few self experiments with a severe food allergy would give pretty concrete conclusions. Unfortunately, it may also kill you. I also don’t need an experiment to establish the fact that cuts and scrapes of the skin cause pain. You usually have a enough data by the time you are a toddler to securely make that conclusion. The problem is that most treatments don’t have such a direct and clear effect as a few shots of whiskey, and so we need more than personal anecdotes to evaluate most interesting questions.