20. May 2021
inferences we can draw from research syntheses
depend on the decisions in the research process
(Ioannidis, 2016)
e.g. “garbage in garbage out”
(Egger et al., 2001)
degrees of freedom, e.g. in
- literature search
- eligibility criteria
- strategy for data synthesis
transparency allow for critical evaluation & enhance reproducibility
(Wilson, 2019)
Open Science Practices help to make research transparent and reproducible
(Page et al., 2021)
preregistration makes research syntheses as a process transparent
(Atkinson et al., 2015; Stewart et al., 2012)
preregistration of research syntheses is not standard procedure yet (outside of health)
(Ioannidis, 2016; Møller et al., 2018)
PROSPERO
(Booth et al., 2012)
PRISMA-P
(Moher et al., 2015)
MARS
(Appelbaum et al., 2018)
PROSPERO
(Booth et al., 2012)
PRISMA-P
(Moher et al., 2015)
MARS
(Appelbaum et al., 2018)
discipline-specific frameworks
“Preregistration is hard”
e.g. including power analysis, coding manual, data analysis strategy
(Nosek et al., 2019)
If we want culture change, we need to “make it easy”
(Nosek, 2019)
TüDiLB
www.tuedilb-tuebingen.de
Joint project of
University of Tübingen and
Leibniz-Institut für Wissensmedien (IWM)
Research group “Transformation & Translation”
research syntheses on the topic of digitization
in teaching and teacher education
• utilize existing frameworks and standards
• make them accessible & applicable
- for meta-analyses and systematic reviews
- for all disciplines
• enable seamless integration of data analyses, files, figures, text, …
• ensure compatibility with markup languages
produce standalone HTML/PDF
preregRS = R Markdown template (install via R-package)
preregRS = R Markdown template (install via R-package)
preregRS = R Markdown template (install via R-package)
preregRS = R Markdown template (install via R-package)
include R code and its output
encode files (e.g. separate coding system)
include figures, iframes, latex, …
include interactive content (e.g. plotly)
preregRS = R Markdown template (install via R-package)
include R code and its output
encode files (e.g. separate coding system)
include figures, iframes, latex, …
include interactive content (e.g. plotly)
compile to standalone file
Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73(1), 3–25. https://doi.org/10.1037/amp0000191
Atkinson, K. M., Koenka, A. C., Sanchez, C. E., Moshontz, H., & Cooper, H. (2015). Reporting standards for literature searches and report inclusion criteria: Making research syntheses more transparent and easy to replicate: Reporting Standards for Literature Searches. Research Synthesis Methods, 6(1), 87–95. https://doi.org/10.1002/jrsm.1127
Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012). The nuts and bolts of PROSPERO: An international prospective register of systematic reviews. Systematic Reviews, 1(1), 2. https://doi.org/10.1186/2046-4053-1-2
Egger, M., Smith, G. D., & Sterne, J. A. C. (2001). Uses and abuses of meta-analysis. Clinical Medicine, 1(6), 478–484. https://doi.org/10.7861/clinmedicine.1-6-478
Ioannidis, J. P. A. (2016). The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. The Milbank Quarterly, 94(3), 485–514. https://doi.org/10.1111/1468-0009.12210
Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4(1), 1. https://doi.org/10.1186/2046-4053-4-1
Møller, M. H., Ioannidis, J. P. A., & Darmon, M. (2018). Are systematic reviews and meta-analyses still useful research? We are not sure. Intensive Care Medicine, 44(4), 518–520. https://doi.org/10.1007/s00134-017-5039-y
Nosek, B. A. (2019). Culture change toward more open, rigorous, and reproducible research.
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van ’t Veer, A. E., & Vazire, S. (2019). Preregistration Is Hard, And Worthwhile. Trends in Cognitive Sciences, 23(10), 815–818. https://doi.org/10.1016/j.tics.2019.07.009
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
Stewart, L., Moher, D., & Shekelle, P. (2012). Why prospective registration of systematic reviews makes sense. Systematic Reviews, 1(1), 7. https://doi.org/10.1186/2046-4053-1-7
Wilson, D. B. (2019). Systematic Coding for Research Synthesis. In H. M. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), Handbook of research synthesis and meta-analysis (3rd edition, pp. 153–172). Russell Sage Foundation.
Images
(in order of appearance)
title page icons | fontawesome.com CC-BY 4.0 (incons used throughout the slides)
title page background | Bogdan Karlenko on Unsplash Javier Allegue Barros on Unsplash
Center for Open Science osf.io/sxn73/
marianne bos on Unsplash
Bogdan Karlenko on Unsplash
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.