Thoughts on Replication with a Case Study

Sociology lags neighbouring disciplines like psychology and economics in recognizing the urgency of replication. Replication is straightforward for experiments that claim to elicit universal human characteristics. It is more complicated when we are interested in social phenomena that depend on an unspecified array of contextual factors. Drawing lessons from literatures on field experiments and on civil wars, I draw a distinction between robustness and repeatability. I then test the robustness of a recent article from political science. I show how the result disappears when we add a few obvious additional variables and utilize a more suitable method of estimation. My conclusions are pessimistic. The proliferation of datasets facilitates replication but also encourages data mining. Because the profession rewards novelty rather than falsification, we can expect no diminution in the publication of results that are neither robust nor repeatable.