Motus Mentis

In the December 13th edition of The New Yorker is a feature article by Jonah Lehrer titled The Truth Wears Off, about what’s known as the “Decline Effect” in experimental research.

If you have a subscription to the New Yorker, you can read the article here. The magazine’s website does not allow selecting and pasting of text, however, and I can’t find it anywhere else online, so not having the patience tonight for much transcribing, I’ll just sum up, rather than copying extensive quotes as I normally would.

The idea — and it is a troubling one — is that a great many experimental results just don’t seem to last. The author introduces us to psychologist Jonathan Schooler, who twenty years ago published a paper called Verbal overshadowing of visual memories: some things are better left unsaid.

At the time, it was widely believed that the act of describing our memories improved them. But. in a series of clever experiments, Schooler demonstrated that subjects shown a face and asked to describe it were much less likely to recognize the face when shown it later than those who had simply looked at it. Schooler called the phenomenon “verbal overshadowing”.

The study turned him into an academic star. Since its publication, in 1990, it has been cited more than four hundred times. Before long, Schooler had extended the model to a variety of other tasks, such as remembering the taste of a wine, identifying the best strawberry jam, and solving difficult creatine puzzles. In each instance, asking people to put their perceptions into words led to dramatic decreases in performance.

The problem was that it was getting harder and harder, as time went on, to demonstrate the effect, and harder for others to replicate it. It was just as, if in some baffling sense, the “novelty” was wearing off, or as if the world was developing a “tolerance” to being investigated this way. Schooler started calling it “cosmic habituation”.

And it turns out that the effect is widespread. The article describes, for example, a marked decline over time in the effectiveness of once-powerful antidepressant drugs. It also talks about the pioneering ESP experiments of J.B. Rhine, who found that his subject Adam Linzmayer’s impressive ability to predict cards fell off over time to no better than chance.

We also read about an observational study of sexual selection in barn swallows. A 1991 project by the Danish researcher Anders MÁ¸ller found that swallows with symmetrical tail-feathers had a far better chance of mating than their less-symmetrical fellows. This concept — that symmetry is a proxy for genetic health — soon became a widely accepted view, and studies strongly confirmed the effect repeatedly in other species, from fruit flies to humans. After a while, though, the effect, at first extremely robust, became harder and harder to detect.

What’s going on here? One might imagine that the results of psychological studies, once they make their way into the public awareness, begin to affect the system that’s being measured. But barn swallows?

Various points have been raised to try to account for the Decline Effect. The first to come to mind is as described by Mr. Lehrer:

The most likely explanation for the decline is an obvious one: regression to the mean. As the experiment is repeated, that is, an early statistical fluke gets canceled out. The extrasensory powers [seen by Schooler in an experiment to test for precognition, which then disappeared in later trials] didn’t decline — they were simply an illusion that vanished over time.

But this is not statistically satisfying; many of these initial results are simply too strong to have a significant likelihood of being anomalies. Other possible causes put forward refer to common biases, such as confirmation bias and publication bias. Lehrer points out that statistician Theodore Sterling, for example, noticed in 1957 that 97% of all psychological papers with statistically significant data confirmed the expected hypothesis. And it is clear that scientists often “shoehorn” their data, ever so slightly, to fit their hypotheses: after all, scientists are human, and you don’t rise to academic fame by proposing hypotheses only to disconfirm them. As more, and more disinterested, researchers repeat the experiments, the shoehorning effect falls away.

Mr. Lehrer gives us the example of acupuncture tests done in the East and the West. Between 1966 and 1995, of 47 studies done in the Far East, avery one of them found significant therapeutic benefit, while in the West, only 56% of 94 trials did.

We read about Stanford epidemiologist John Ioannidis, who reviewed 432 studies on genetic difference in disease risk between the sexes, and found that only one was consistently replicable. Dr. Ioannadis, convinced that much of medical fashion was based on completely untrustworthy results, published a 2005 paper in PLoS Medicine called Why Most Published Research Findings Are False. (The abstract is here, and you can read an article about Dr. Ioannadis here.) Most research, indeed, is never even subjected to attempts at replication.

What are we to make of this? Nobody is really sure; I’m certainly not. I will say this, though: although this sort of thing is catnip to postmodernists and various New Age sorts (who yearn, respectively, to undermine the notion of objective truth or to refute the idea that cold-hearted, left-brain-y Science is a useful approach to it), this is most certainly not a death-knell for Western rationality, or the scientific method. Look around you: airplanes still stay aloft; gasoline still combusts in the usual way in the engine of your car; helium is still less dense than air; our space probes still achieve their trajectories with pinpoint accuracy; the transmission of a digital signal still excites the screen of your computer in a precisely predictable pattern as you read this post. Mr. Lehrer’s title seems to argue against the existence in nature of objective, or at least stable, truths; not so fast, say I.

But clearly there is something odd going on. One thing that stands out to me is that the results that seem the most compromised by the Decline Effect are almost exclusively to do with studies of living systems. What might that mean?