9 November 2017

Open science essentials in two minutes, part one

The problem

As a scholarly community we allowed ourselves to forget the distinction between exploratory vs confirmatory research, presenting exploratory results as confirmatory, presenting post-hoc rationales as predictions. As well as being dishonest, this makes for unreliable science.

Flexibility in how you analyse your data (“researcher degrees of freedom”) can invalidate statistical inferences.

Importantly, you can employ questionable research practices like this (“p-hacking”) without knowing you are doing it. Decide to stop an analysis because the results are significant? Measure three dependent variables and use the one that “works”? Exclude participants who don’t respond to your manipulation?

All justified in exploratory research, but mean you are exploring a garden of forking paths in the space of possible analysis - when you arrive at a significant result, you won’t be sure you got there because of the data, or your choices.

The solution

There is a solution - pre-registration. Declare in advance the details of your method and your analysis: sample size, exclusion conditions, dependent variables, directional predictions.

You can do this

Pre-registration is easy. There is no single, universally accepted, way to do it.

  • You could write your data collection and analysis plan down and post it on your blog.

  • You can use the Open Science Framework to timestamp and archive a pre-registration, so you can prove you made a prediction ahead of time.

  • You can visit which provides a form to complete. This will help you structure your pre-registration, making sure you include all relevant information.

  • Registered Reports: More and more journals are committing to published pre-registered studies. They review the method and analysis plan before data collection and agree to publish once the results are in (however they turn out).

You should do this

Why do this?

  • Credibility - other researchers and journals will know you predicted the results before you got them.

  • You can still do exploratory analysis, it just makes it clear which is which.

  • Forces you to think about the analysis before collecting the data (a great benefit).

  • More confidence in your results.

Further reading

  • A pre-registration primer by Tom Hardwicke - see also for background references on the state of psychology and p-hacking.

  • Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P., ... and Ioannidis, J. P. (2017) A manifesto for reproducible science, Nature Human Behaviour, 1, 0021.
    View article

  • Psychology’s ‘registration revolution’, Chris Chambers in The Guardian.
    View article

Addendum 14 November 2017

As luck would have it, I stumbled across a bunch of useful extra resources in the days after publishing this post.

Notes to support lighting talk as part of Open Science seminar in the Department of Psychology, University of Sheffield on 14/11/17.

Other parts in the series

The Open Science Framework