Science In Decline?

In the December 13th edition of The New Yorker is a feature article by Jonah Lehrer titled The Truth Wears Off, about what’s known as the “Decline Effect” in experimental research.

If you have a subscription to the New Yorker, you can read the article here. The magazine’s website does not allow selecting and pasting of text, however, and I can’t find it anywhere else online, so not having the patience tonight for much transcribing, I’ll just sum up, rather than copying extensive quotes as I normally would.

The idea — and it is a troubling one — is that a great many experimental results just don’t seem to last. The author introduces us to psychologist Jonathan Schooler, who twenty years ago published a paper called Verbal overshadowing of visual memories: some things are better left unsaid.

At the time, it was widely believed that the act of describing our memories improved them. But. in a series of clever experiments, Schooler demonstrated that subjects shown a face and asked to describe it were much less likely to recognize the face when shown it later than those who had simply looked at it. Schooler called the phenomenon “verbal overshadowing”.

The study turned him into an academic star. Since its publication, in 1990, it has been cited more than four hundred times. Before long, Schooler had extended the model to a variety of other tasks, such as remembering the taste of a wine, identifying the best strawberry jam, and solving difficult creatine puzzles. In each instance, asking people to put their perceptions into words led to dramatic decreases in performance.

The problem was that it was getting harder and harder, as time went on, to demonstrate the effect, and harder for others to replicate it. It was just as, if in some baffling sense, the “novelty” was wearing off, or as if the world was developing a “tolerance” to being investigated this way. Schooler started calling it “cosmic habituation”.

And it turns out that the effect is widespread. The article describes, for example, a marked decline over time in the effectiveness of once-powerful antidepressant drugs. It also talks about the pioneering ESP experiments of J.B. Rhine, who found that his subject Adam Linzmayer’s impressive ability to predict cards fell off over time to no better than chance.

We also read about an observational study of sexual selection in barn swallows. A 1991 project by the Danish researcher Anders Møller found that swallows with symmetrical tail-feathers had a far better chance of mating than their less-symmetrical fellows. This concept — that symmetry is a proxy for genetic health — soon became a widely accepted view, and studies strongly confirmed the effect repeatedly in other species, from fruit flies to humans. After a while, though, the effect, at first extremely robust, became harder and harder to detect.

What’s going on here? One might imagine that the results of psychological studies, once they make their way into the public awareness, begin to affect the system that’s being measured. But barn swallows?

Various points have been raised to try to account for the Decline Effect. The first to come to mind is as described by Mr. Lehrer:

The most likely explanation for the decline is an obvious one: regression to the mean. As the experiment is repeated, that is, an early statistical fluke gets canceled out. The extrasensory powers [seen by Schooler in an experiment to test for precognition, which then disappeared in later trials] didn’t decline — they were simply an illusion that vanished over time.

But this is not statistically satisfying; many of these initial results are simply too strong to have a significant likelihood of being anomalies. Other possible causes put forward refer to common biases, such as confirmation bias and publication bias. Lehrer points out that statistician Theodore Sterling, for example, noticed in 1957 that 97% of all psychological papers with statistically significant data confirmed the expected hypothesis. And it is clear that scientists often “shoehorn” their data, ever so slightly, to fit their hypotheses: after all, scientists are human, and you don’t rise to academic fame by proposing hypotheses only to disconfirm them. As more, and more disinterested, researchers repeat the experiments, the shoehorning effect falls away.

Mr. Lehrer gives us the example of acupuncture tests done in the East and the West. Between 1966 and 1995, of 47 studies done in the Far East, avery one of them found significant therapeutic benefit, while in the West, only 56% of 94 trials did.

We read about Stanford epidemiologist John Ioannidis, who reviewed 432 studies on genetic difference in disease risk between the sexes, and found that only one was consistently replicable. Dr. Ioannadis, convinced that much of medical fashion was based on completely untrustworthy results, published a 2005 paper in PLoS Medicine called Why Most Published Research Findings Are False. (The abstract is here, and you can read an article about Dr. Ioannadis here.) Most research, indeed, is never even subjected to attempts at replication.

What are we to make of this? Nobody is really sure; I’m certainly not. I will say this, though: although this sort of thing is catnip to postmodernists and various New Age sorts (who yearn, respectively, to undermine the notion of objective truth or to refute the idea that cold-hearted, left-brain-y Science is a useful approach to it), this is most certainly not a death-knell for Western rationality, or the scientific method. Look around you: airplanes still stay aloft; gasoline still combusts in the usual way in the engine of your car; helium is still less dense than air; our space probes still achieve their trajectories with pinpoint accuracy; the transmission of a digital signal still excites the screen of your computer in a precisely predictable pattern as you read this post. Mr. Lehrer’s title seems to argue against the existence in nature of objective, or at least stable, truths; not so fast, say I.

But clearly there is something odd going on. One thing that stands out to me is that the results that seem the most compromised by the Decline Effect are almost exclusively to do with studies of living systems. What might that mean?

Related content from Sphere

13 Comments

  1. JK says

    Increased use[s] of endocrine disruptors.

    The [mentioned] studies have almost exclusively taken place in, and the test subjects were nearly exclusively, Western.

    One noteable exception, the immediate environs of – Bhopal, India.

    Posted December 14, 2010 at 2:10 am | Permalink
  2. Malcolm says

    What about the barn swallows?

    Posted December 14, 2010 at 2:22 am | Permalink
  3. Fascinating stuff. As for your observation that living systems are more likely to be involved, Malcolm, perhaps the numbers are easier to fudge, even inadvertently, in biology, but that’s just a guess.

    Posted December 14, 2010 at 7:54 am | Permalink
  4. JK says

    Great googly moogly Malcolm, I had no idea of the amount of research done on barn swallows. I did turn up one study though which, given my recent reading on something related, tends to have me down in the chops.

    The swallows of Chernobyl:

    “…First, they found that birds in the Chernobyl nuclear disaster study area have an increased number of genetic mutations. Second, they found that birds with red, yellow or orange plumage based on organic pigments called carotenoids, such as Barn Swallows, were much less abundant in contaminated areas than were less colorful species.”

    I don’t suppose Russian barn swallows can properly be termed Western.

    http://scientopia.org/blogs/thisscientificlife/2007/07/17/the-birds-of-chernobyl-better-off-drab-and-lazy/

    Posted December 14, 2010 at 9:51 am | Permalink
  5. bob koepp says

    Since the observation of the decline effect is itself a piece of social science, I imagine that if we just wait a bit, all manner of odd results will become increasingly robust. Either that or the decline effect is scientific rubbish.

    Posted December 14, 2010 at 10:25 am | Permalink
  6. Better Response says

    Here’s the excellent response to the article – it is very flawed.

    http://www.sciencebasedmedicine.org/?p=8987

    Posted December 14, 2010 at 3:54 pm | Permalink
  7. @Posted December 14, 2010 at 3:54 pm:

    “Here’s the excellent response to the article – it is very flawed.”

    Which begs the question, “What does it refer to?”

    Posted December 14, 2010 at 6:15 pm | Permalink
  8. Are Jonathan Schooler’s findings still valid . . . or are they also no longer being replicated?

    Jeffery Hodges

    * * *

    Posted December 14, 2010 at 7:34 pm | Permalink
  9. Better Response says

    @TheBigHenry – you read the whole response and your only question is about the word ‘it’?

    Posted December 14, 2010 at 7:52 pm | Permalink
  10. Malcolm says

    Thanks, Kimo, for posting that link, which I think does get it about right, though I would say that the case is perhaps not quite as neatly closed as David Gorski suggests.

    I certainly do agree in spades, as I hope I made clear in my penultimate paragraph, that none of this should be taken as any sort of validation of mush-mouthed PoMo relativism, or as a brief against the scientific method; it is certainly no secret that science is done by fallible humans, and that all scientific understanding of the natural world is provisional, and subject to continuous revision and remodeling.

    A scientific community that lays claim to “truth” is a popular straw man among postmodernists, but honest science makes no such boast; even at its best, all it can manage is the whittling away of falsehood.

    Posted December 14, 2010 at 10:34 pm | Permalink
  11. @December 14, 2010 at 7:52 pm

    Where did you get the idea that I read the response?

    Before reading the response, I wanted to know which you believed to be flawed, the article or the response.

    Posted December 15, 2010 at 7:11 pm | Permalink
  12. Malcolm says

    Hi Jeffery,

    Sorry for the delayed reply. It did appear that Dr. Schooler’s own work appeared to be harder and harder to replicate, also.

    Posted December 16, 2010 at 11:56 am | Permalink
  13. JK says

    Man, this has (and in deference to waka’s theme) taken me many places. I wonder – might the “Deline” be at least partially explained by the many emerging methodologies by which scientific research is conducted?

    Not being trained (a rank amateur myself, admittedly) in science, I have trouble explaining precisely what I mean to say, so:

    http://thonyc.wordpress.com/2010/12/14/the-2nd-scientific-revolution/

    Posted December 17, 2010 at 12:55 pm | Permalink