Pre-registration
9 November 2017
Open science essentials in two minutes, part one
The problem
As a scholarly community we allowed ourselves to forget the distinction between exploratory vs confirmatory research, presenting exploratory results as confirmatory, presenting post-hoc rationales as predictions. As well as being dishonest, this makes for unreliable science.
Flexibility in how you analyse your data (“researcher degrees of freedom”) can invalidate statistical inferences.
Importantly, you can employ questionable research practices like this (“p-hacking”) without knowing you are doing it. Decide to stop an analysis because the results are significant? Measure three dependent variables and use the one that “works”? Exclude participants who don’t respond to your manipulation?
All justified in exploratory research, but mean you are exploring a garden of forking paths in the space of possible analysis - when you arrive at a significant result, you won’t be sure you got there because of the data, or your choices.
The solution
There is a solution - pre-registration. Declare in advance the details of your method and your analysis: sample size, exclusion conditions, dependent variables, directional predictions.
You can do this
Pre-registration is easy. There is no single, universally accepted, way to do it.
You could write your data collection and analysis plan down and post it on your blog.
You can use the Open Science Framework to timestamp and archive a pre-registration, so you can prove you made a prediction ahead of time.
You can visit aspredicted.org which provides a form to complete. This will help you structure your pre-registration, making sure you include all relevant information.
Registered Reports: More and more journals are committing to published pre-registered studies. They review the method and analysis plan before data collection and agree to publish once the results are in (however they turn out).
You should do this
Why do this?
Credibility - other researchers and journals will know you predicted the results before you got them.
You can still do exploratory analysis, it just makes it clear which is which.
Forces you to think about the analysis before collecting the data (a great benefit).
More confidence in your results.
Further reading
A pre-registration primer by Tom Hardwicke - see also for background references on the state of psychology and p-hacking.
Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P., ... and Ioannidis, J. P. (2017) A manifesto for reproducible science, Nature Human Behaviour, 1, 0021.
View articlePsychology’s ‘registration revolution’, Chris Chambers in The Guardian.
View article
Addendum 14 November 2017
As luck would have it, I stumbled across a bunch of useful extra resources in the days after publishing this post.
Another template for pre-registration, via @openstatslab.
A pre-registration coaching network from the Center for Open Science.
How to properly pre-register a study from the people who brought you aspredicted.org.
Two examples of pre-registration:
1. (by us) via OSF
2. via Sheffield ORDA
Slides from SfN talk on pre-registration from COS’s David Mellor.
A consortium pre-registration for undergraduate projects:
Notes to support lighting talk as part of Open Science seminar in the Department of Psychology, University of Sheffield on 14/11/17.