Slate recently had an interesting article discussing the issue of replication of findings from experiments in social psychology. While I am an academic, the issue of replication does not come up much in the legal world. Most legal academic articles make normative arguments, although it is true that in some areas, such as legal history, issues similar to replication do arise.
Perhaps I would feel differently if I had more experience with the issue of replication, but from my perspective replication seems like an essential aspect of science, even (or especially) soft social science. If one is relying on an experiment, then that experiment better be replicable or else the findings do not constitute reliable information and should not be considered science. The point is so basic that it is hard to know how anyone could argue with it.
Yet, some social scientists disagree. There appear to be two counter arguments: that the academics doing the replication are not skilled enough to perform it competently or that they incorrectly conducting the experiment because subtle aspects are not being followed (in part because they were not stated in the original article announcing the experiment).
These counters seem weak. Starting with the second one, if the original publication omits pertinent information, then that information should have been included – either in the original publication or online. After all, it is relevant to evaluating the study (even if one is not attempting a replication). As to the first counter, the skills of the scientist will be evaluated by the journal in deciding whether to publish the experiment. After all, the journal that published the original article also had to evaluate the skills of the author.
I am sure that it is annoying for one’s study to be criticized if the criticisms are wrong. But there is, of course, a check on incorrect replications – other replications that support the original study. It might be thought that all of these replications are a waste, but of course they are not – if we are to increase our knowledge it is essential that we determine whether experiments can stand up to scrutiny.
It is sad that scientists who attempt to replicate other findings seem to be held in less esteem than the scientists who do the studies in the first place. In law, I am not sure that is the case. If a significant piece comes out which relies on historical assertions, there will be substantial interest in a piece that disputes these assertions. My co-blogger, John McGinnis, once published a piece that took down two very prominent academics who had made sloppy and incorrect assertions about the Appointments Clause. The piece was not treated as unimportant because it merely critiqued others. Quite the contrary.
As an outsider, it is hard to know whether, and if so, how corrupt – yes that word is appropriate – social psychology is. If important people are resisting replications – and if journals are refusing to publish replications that fail – then the field is corrupt. But that may not be the case. Perhaps it is only a small minority who oppose this replication movement. Let’s hope.