(* with apologies to Oscar Wilde)
Replication is never far from the news in psychology these days. Pick up a newspaper or journal, or browse the blogosphere, and chances are you’ll encounter yet another piece on the importance of repeating experiments. Like those people who dress up in period costume and reenact old battles, researchers are battening down and busying themselves repeating each and every tedious behavioral experiment from the past 100 years or so. Just to make sure. Meanwhile, the bright young things are immersing themselves in metascience, writing treatises on statistics and worthy lists of good research practices that will probably, eventually help us figure out how minds and brains work… one day, when someone actually gets around to looking.
Were Onan alive today, he’d be a replicator. Or second author on a methods paper, anyway. Which is not so say that I think replication is a bad idea in itself (as I made clear in an earlier post, I’m all for it). It’s rather that I feel a sense of mounting horror as I realize that most of the things I think of as ‘standard scientific practice,’ seem to be news to many of my colleagues. And I feel queasy as I contemplate what seems to be the replication movement’s big idea, which is that somehow a few new statistical techniques and some registered replications will suffice to fix the many parts of psychology and neuroscience that are misguided, doltish, or simply wrong.
In fact, to the extent that I occasionally find myself feeling some support for the replication movement, it is because of this last point. In the same way that every minute that the managing committee of a soviet car factory spent debating their workers’ political education saved many a poor oppressed people from having to actually travel in lethally incompetent cars, so much of the output of the replication movement is, thankfully, irrelevant. Usually, at least, it avoids being actively intellectually harmful.
The problem, as I see it, is the bottom-line: There is an expectation that those of us that draw a salary for pursuing research in the brain and cognitive sciences will actually try to contribute to advancing research in the brain and cognitive sciences; that the people who are paying us to pipe would prefer some tunes to more methods papers. They want us to pump out as many cars as we can, and they would like them not to be deathtraps, if we can possibly avoid it.
We are promising this:
yet delivering this:
Which raises a question: how can I convince you that I’m right about this problem without sounding like a soviet car committee pondering a new five year plan?