Document Type
Article
Publication Date
8-16-2017
Abstract
Over the last thirty years, legal scholars have increasingly deployed experimental studies, particularly hypothetical scenarios, to test intuitions about legal reasoning and behavior. That movement has accelerated in the last decade, facilitated in large part by cheap and convenient Internet participant recruiting platforms like Amazon Mechanical Turk. The widespread use of MTurk subjects, a practice that dramatically lowers the barriers to entry for experimental research, has been controversial. At the same time, law and psychology’s home discipline is experiencing a public crisis of confidence widely discussed in terms of the “replication crisis.” At present, law and psychology research is arguably in a new era, in which it is both an accepted feature of the legal landscape and also a target of fresh skepticism. The moment is ripe for taking stock.
In this paper, we bring an empirical approach to these problems. Using three canonical law and psychology findings, we document the challenges and the feasibility of reproducing results across platforms. We evaluate the extent to which we are able to reproduce the original findings with contemporary subject pools (MTurk, other national online platforms, and in-person labs). We partially replicate the results, and show marked similarities in subject responses across platforms. In the context of the experiments here, we conclude that meaningful replication requires active intervention in order to keep the materials relevant and sensible. The second aim is to compare Turk subjects to the original samples and to the replication samples. We find, consistent with the weight of recent evidence, that MTurk samples are highly reliable and useful. Subjects are highly similar to subjects on other online platforms an in-person samples, but they differ in their high level of attentiveness. Finally, we review the growing replication literature across disciplines, as well as our firsthand experience, to propose a set of standard practices for the publication of results in law and psychology.
Keywords
replication crisis, reproducibility, experimental legal studies, law and psychology
Publication Title
Journal of Empirical Legal Studies
Repository Citation
Irvine, Krin; Hoffman, David A.; and Wilkinson-Ryan, Tess, "Law and Psychology Grows Up, Goes Online, and Replicates" (2017). All Faculty Scholarship. 1884.
https://doi.org/10.1111/jels.12180
Included in
Cognition and Perception Commons, Experimental Analysis of Behavior Commons, Law and Psychology Commons
Publication Citation
15 J. Empirical Leg. Stud. 320 (2018)