A meta-perspective on reproducibility and replication rates

José Rolando Villaseñor Agúndez^1*, Heinrich Nax²

¹ University of Zurich, Zurich, Switzerland.
² University of Zurich/ETH, Zurich, Switzerland.

Originally published on June 1, 2026 at:

https://doi.org/10.31235/osf.io/smfqn_v2

Abstract

Large-scale replication projects have become central to debates about reproducibility in the social and behavioral sciences and related fields. In one of the most recent replication projects (Tyner et al. 2026), 274 positive results from 164 published papers between 2009 and 2018 were subjected to replication attempts. Depending on the criterion applied to evaluate replication success, between 28.6% and 74.8% of results replicated successfully. A key conclusion of this project was that the conditions that accept or reject replicability need further investigation. Building on that insight, the present study takes a meta-perspective on reproducibility by examining replication projects themselves. In particular, it focuses on how the choice of replication criterion may shape the conclusions that are reported. Two widely used criteria are the p-value-based replication criterion (PVRC), which assesses whether an originally significant effect remains statistically significant in the same direction in the replication, and the confidence-interval-based replication criterion (CIRC), which assesses whether the confidence intervals of the original and replication studies overlap. Using a sample of 31 replication projects across psychology, behavioral and social sciences, psychiatry, and behavioral ecology, this study shows substantial heterogeneity in reported replication rates both across projects and within projects depending on the criterion applied. Meta-analytic evidence suggests no overall time trend toward improved replicability and no robust association of replication rates with field, authorship patterns, or journal impact. At the same time, funnel-plot patterns indicate that reported CIRC estimates may be selectively biased toward more extreme values. These findings raise the possibility of publication bias and reporting in favor of “extreme results” operating not only in original studies, but also in the meta-literature on replication itself.

Full text

1. Introduction

Replication projects have played a major role in bringing concerns about reproducibility to the forefront of scientific debate. Studies across several disciplines have attempted to estimate the reproducibility of scientific findings through systematic replication efforts, including well-known projects in psychology and related fields (e.g., OSC 2015, Camerer et al. 2016, 2018, Nosek et al. 2022). These efforts have helped establish what is now widely described as a replication or reproducibility crisis.

At the same time, the increased attention to open science practices, preregistration, and methodological reform has led some observers to argue that scientific practice may be improving (e.g., Korbmacher et al. 2023). However, broad empirical support for such optimism remains limited. Despite roughly a decade separating early landmark replication projects and more recent large-scale efforts, the available evidence continues to show similar levels of variability and only moderate robustness, especially in psychology and across the social and behavioral sciences more broadly.

2. Replication Criteria and Investigator Discretion

A central concern motivating this study is that the choice of replication criterion may lead to qualitatively different conclusions. Two criteria are especially common in the literature.

The first is the p-value-based replication criterion (PVRC). For positive original findings, PVRC classifies a result as replicated when the replication study also finds a statistically significant effect in the same direction.

The second is the confidence-interval-based replication criterion (CIRC). CIRC classifies a result as replicated when the confidence intervals of the original study and the replication overlap.

These criteria do not measure identical properties of the data, and their use can yield substantially different replication rates. In OSC (2025), for example, the weighted PVRC was reported as 49%, whereas the CIRC was reported as 31%. This discrepancy highlights that the definition of replication success is not merely technical; it can materially affect the resulting narrative.

The possibility of ex post selection of replication criteria is especially concerning. If authors can emphasize whichever criterion produces the more striking result, reporting may become biased toward more extreme conclusions. Such concerns have already been raised in the methodological literature. In their critique of Protzko et al. (2024),

Bak-Coleman and Devezer (2024) argued that the study’s emphasized replicability claims relied on non-preregistered outcome measures and analyses, despite claims to the contrary, and that the published article shifted attention away from the project’s originally preregistered focus. These concerns were later reflected in the journal’s retraction note, which cited misstatement of hypotheses, lack of preregistration for analyses supporting the titular claim, selection of outcome measures and analyses with knowledge of the data, and incomplete reporting. More broadly, this suggests that replication projects themselves may be vulnerable to the same selective reporting incentives long criticized in original research (Bak-Coleman C Devezer, 2024; Protzko et al., 2024 Retraction Note).

3. Conceptual Distinctions

To avoid terminological confusion, the present analysis distinguishes among three types of papers.

Original papers: studies whose results are being “checked” via replication; e.g. “Self-interest and fairness: self-serving choices of justice principles” by Rodriguez-Lara and Moreno-Garrido (Experimental Economics, 2012).
Replications: replicators’ studies “checking” original papers (or claims)
Replication projects: projects that assess how many findings from some set of original papers are confirmed by corresponding replications; g. “Investigating the replicability of the social and behavioral sciences” (2026).

4. Data and Scope

To provide a broader historical and cross-disciplinary perspective, this study assembles a sample of replication projects in fields including psychology, behavioral and social sciences, psychiatry, and behavioral ecology. The goal is to go more “meta” on reproducibility than individual replication projects by synthesizing projects that themselves estimate reproducibility.

At present, the sample includes 31 relevant replication projects. These projects differ in field, time period, journal context, authorship structure, and analytic choices. Table 1 summarizes the sample.

**Table 1.** Summary statistics for original papers and replication projects.
	Mean	Min	Max	N
Features of replication projects				31
PVRC	0.50	0.05	0.88	25
CIRC	0.54	0.09	0.87	18
Impact factor	3.9	0.0	13.7
Publication year	2019	2014	2024
# Authors	36.4	1.0	270.0
PVRC	0.52	0.05	0.88	23
CIRC	0.54	0.09	0.87	18
Average features of originals per replication project				1182
Impact factor	5.3	0.0	17.4
Publication year	2008	1988	2024
# Authors	9.1	1.9	80.7

5. Descriptive Patterns

As illustrated in Figure 1, replication rates vary substantially across projects. Across the sampled projects, the average reported replication rate is 0.52 under PVRC and 0.54 under CIRC. However, these averages conceal substantial dispersion. The lowest observed rates are 0.05 for PVRC and 0.0G for CIRC, whereas the highest are 0.88 for PVRC and 0.87 for CIRC.

Figure 1. Studies ordered by the average value of PVCR and CIRC.

This heterogeneity appears both across and within replication projects. Not only do projects differ markedly from one another, but PVRC and CIRC also often diverge within the same project, and there is no clear general pattern indicating that one criterion systematically yields higher values than the other. In some cases PVRC exceeds CIRC; in others the reverse holds. For example, Kvarven et al. (2020) reports PVRC greater than CIRC, whereas Sobkow et al. (2022) reports CIRC greater than PVRC.

This absence of a consistent directional relationship is precisely what makes criterion choice consequential. Where the criteria do not move together predictably, post hoc emphasis on one or the other can support different interpretations of the same evidential base.

6. Meta-Analytic Findings

Meta-analytic examination of the replication-project data, including funnel plots and meta-regression, yields two main findings.

First, despite the large heterogeneity in reported estimates, there is no clear evidence that replication rates are associated with the choice of criterion itself when considered across the available sample, nor with the field of study, including psychology, which has received especially intense scrutiny. Likewise, there is no evidence that replication rates are systematically related to the number of authors involved in the original papers or in the replication projects, nor to publication in higher-impact journals. In addition, there is no detectable time trend suggesting that science overall has become either more or less replicable over time.

Second, the funnel plots suggest an important asymmetry in the reporting of CIRC estimates relative to PVRC estimates. Figure 2 shows PVRC estimates distributed on both sides of the sample average, including several observations near the center of the distribution. In particular, reported CIRC values appear to be disproportionately concentrated at more extreme levels, as seen on Figure 3 while values closer to the sample average are comparatively underrepresented. Replication rates near the center of the distribution appear to be largely missing for CIRC.

Figure 2. Funnel plots for PVRC

Figure 3. Funnel plots for CIRC

One plausible interpretation is that the meta-literature on replication is itself affected by selective reporting or publication bias favoring stronger, more dramatic messages. If so, the same incentive structures that distort original research may also shape the way replication outcomes are summarized and published.

7. Discussion

These findings challenge simple narratives about the reproducibility crisis. The current evidence does not support the view that most published findings are false, but neither does it support the claim that most findings are robust. Rather, the literature points to a more ambiguous picture in which conclusions can shift depending on methodological and reporting choices, especially the choice of replication criterion.

This has important implications. If authors investigating similar literature can support either a pessimistic or an optimistic message by selecting one criterion over another, then the replication literature may itself become vulnerable to narrative inflation. Under such conditions, criterion choice is not a neutral analytic detail; it becomes a potential source of bias.

A further observation is that many replication projects appear not to have been formally preregistered at a level that would constrain later discretion over analytic emphasis.

This is striking given that replication projects often explicitly aim to improve the credibility of science and to reduce bias. The present findings therefore support the view that replication projects should be held to at least the same standards of methodological transparency and rigor that they demand of original research.

Overstating irreproducibility in one field or overstating robustness in another is unlikely to advance the broader goal of improving scientific practice. What is needed instead is greater consistency, transparency, and justification in how replication success is defined, analyzed, and reported.

8. Conclusion

A meta-perspective on replication projects suggests that the literature on reproducibility may itself be affected by selective emphasis and publication bias.

Across 31 replication projects, replication rates show substantial heterogeneity, and the choice between PVRC and CIRC can materially affect the conclusions drawn within a given project. Although there is no clear evidence that replicability differs systematically by field, authorship, journal impact, or time period, there are indications that reported CIRC values may be selectively skewed toward more extreme messages.

These results reinforce the importance of continued investigation into the conditions that accept or reject replicability. They also suggest that future replication projects should place greater emphasis on preregistration, transparent analytic decision-making, and comprehensive reporting of alternative replication criteria. If replication research is to help de-bias science, it must also scrutinize its own methods and incentives.

Data Availability Statement

All data and analysis code used in this study are publicly available. The compiled dataset of 31 replication projects, together with the analysis scripts and accompanying documentation, is archived at “A Meta perspective on Reproducibility and Replication Rates on Zenodo (and Github), DOI: 10.5281/zenodo.20479721, under MIT license and CC-BY-4.0. The replication-rate estimates analysed here were drawn from the previously published replication projects listed in the References and in the List of Replication Projects below. This study did not generate new primary experimental data.

Software

All analyses, including the meta-analysis, meta-regression, and funnel plots, were conducted in R, using RStudio. The packages used are described in the code. Exact version numbers can be obtained by running sessionInfo() in R, and a reference-list entry for each package can be generated with citation(“packagename”).

Author Contributions

Rolando Villaseñor: Conceptualization, Methodology, Formal analysis, Software, Visualization, final draft. Heinrich Nax: Conceptualization, Data curation, Writing – review and editing, Supervision.

Competing Interests

The authors declare no competing interests.

Funding

Heinrich Nax: funded by the SNSF under the Eccellenza Grant “Markets and Norms” Rolando Villasenor: None

References

Bak-Coleman, J., Devezer, B. (2024). Claims about scientific rigour require rigour. Nature Human Behaviour, 8, 1890–1891.

Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., … C Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637-644.

Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., … C Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433-1436.

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.

Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., … C Evans, T. (2023). The replication crisis has led to positive structural, procedural, and community changes. Communications Psychology, 1(1), 3.

Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., … C Vazire, S. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73(1), 719-748.

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

Open Science Collaboration (2025). Investigating the replicability of the social and behavioral sciences. pre-print.

Protzko, J., Krosnick, J., Nelson, L. et al. (2024). RETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable. Nature Human Behaviour 8, 311–319.

Protzko, J., Krosnick, J., Nelson, L. et al. (2024). Retraction Note: High replicability of newly discovered social-behavioural findings is achievable. Nature Human Behaviour 8, 2067.

Rodriguez-Lara, I., C Moreno-Garrido, L. (2012). Self-interest and fairness: self-serving choices of justice principles. Experimental Economics, 15(1), 158-175.

R Core Team (2024). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/

Tyner, A. H., Abatayo, A. L., Daley, M., Field, S., Fox, N., Haber, N. A., … C McHugh, C. (2026). Investigating the replicability of the social and behavioural sciences. Nature, c52(8108), 143-150.

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03

Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag, New York. https://doi.org/10.1007/978-3-319-24277-4

List of Replication Projects

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124.

Boyce, V., Mathur, M., C Frank, M. C. (2023). Eleven years of student replication projects provide evidence on the correlates of replicability in psychology. Royal Society Open Science, 10(11), 231240.

da Costa, G. G., Neves, K., C Amaral, O. (2024). Estimating the replicability of highly cited clinical research (2004–2018). Plos one, 1S(8), e0307145.

Sobkow, A., Surowski, M., Olszewska, A., Antoniewska, N., Barcik, K., Bartkiewicz, U., Brzeska, A., Brzozowska, A., Budrewicz, O., Choja, J., Choma, K., Chorbotowicz, P.,

Filimoniak, M., Filip, Ł., Gambuś, P., Gierlik, W., Gonczar, T., Goryczka, K., Góra, M.,

Haczek, M., Hetmańczuk, W., Holka, Z., Janosz, A., Kikowska, N., Kołcun, J., Kozłowska, Z., Kujawińska, M., Kuleszczyk, M., Lach-Galińska, A., Latacz, K., Ławniczak, A.,

Majewska, K., Makowska, K., Mamzer, M., Marciniszyn, I., Masternak, A., Matuszek, M., Mehr, J., Miela, E., Mleczko, M., Morga, P., Niemczyk, M., Ostrowski, D., Pełdiak, J., Piotrowicz, K., Płuciennik, A., Ryśkiewicz, O., Sekuła, W., Sikora, M., Sikora, N., Sitko, D., Sobczak, A., Sosenko, J., Stando, S., Starek, K., Ślak, Ł., Świtała, J., Świtniewska, N., Tyc, A., Urban, O., Wcisło, N., Wiśniewska, K., Wodzińska, J., Zabiełło, A., Żygadło, M.,

Zaleskiewicz, T., C Traczyk, J. (2022). Conceptual replication study of fifteen JDM effects: Insights from the Polish sample. Polish Psychological Bulletin, 53(3), 138–151. https://doi.org/10.24425/ppb.2022.141862

Davis, A. M., Flicker, B., Hyndman, K., Katok, E., Keppler, S., Leider, S., … C Tong, J. D. (2023). A replication study of operations management experiments in management science. Management Science, cS(9), 4977-4991.

Avelino, G., Desposato, S., C Mardegan, I. (2021). Transparency and Replication in Brazilian Political Science: A First Look. Dados, c4, e20190304.

Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., C Nosek, B. (2021). Investigating the replicability of preclinical cancer biology.

Maassen, E., Van Assen, M. A., Nuijten, M. B., Olsson-Collentine, A., C Wicherts, J. M. (2020). Reproducibility of individual effect sizes in meta-analyses in psychology. PloS one, 15(5), e0233107.

Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D. J., Buttrick, N. R., Chartier, C. R., … C Szecsi, P. (2020). Many Labs 5: Testing pre-data-collection peer review as an intervention to increase replicability. Advances in Methods and Practices in Psychological Science, 3(3), 309-331.

Kvarven, A., Strømland, E., C Johannesson, M. (2020). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour, 4(4), 423-434.

Soto, C. J. (2019). How replicable are links between personality traits and consequential life outcomes? The life outcomes of personality replication project. Psychological Science, 30(5), 711-727.

Kelly, C. D. (2019). Rate and success of study replication in ecology and evolution. PeerJ, 7, e7654.

Naudet, F., Sakarovitch, C., Janiaud, P., Cristea, I., Fanelli, D., Moher, D., C Ioannidis, J. (2018). Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS Medicine.

Niven, D. J., McCormick, T. J., Straus, S. E., Hemmelgarn, B. R., Jeffs, L., Barnes, T. R., C Stelfox, H. T. (2018). Reproducibility of clinical research in critical care: a scoping review. BMC medicine, 1c, 1-12.

Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams Jr, R. B., Alper, S., … C Sowden, W. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443-490.

Tajika, A., Ogawa, Y., Takeshima, N., Hayasaka, Y., C Furukawa, T. A. (2015). Replication and contradiction of highly cited research papers in psychiatry: 10-year follow-up. The British Journal of Psychiatry, 207(4), 357-362.

Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., … C Zhou, X. (2021). Estimating the reproducibility of experimental philosophy. Review of Philosophy and Psychology, 12, 9-44.

Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., … C Uhlmann, E. L. (2016). The pipeline project: Pre-publication independent replications of a single laboratory’s research pipeline. Journal of Experimental Social Psychology, cc, 55-67.

Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., … C Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, c7, 68-82.

Patil, P., Peng, R. D., C Leek, J. T. (2016). What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspectives on Psychological Science, 11(4), 539-544.

Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., … C Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433-1436.

Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., … C Johannesson, M. (2015). Using prediction markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sciences, 112(50), 15343-15347.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 34S(6251), aac4716.

Klein, R., Ratliff, K., Vianello, M., Adams Jr, R. B., Bahník, S., Bernstein, M. J., … C Nosek, A. (2014). Data from investigating variation in replicability: A” many labs” replication project.

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124.

Editors

Kathryn Zeiler
Editor-in-Chief

Olmo R. van den Akker
Handling Editor

Editorial assessment

by Olmo R. van den Akker

DOI: 10.70744/MetaROR.435.1.ea

Both reviewers agree that the manuscript addresses an important and worthwhile question but identify fundamental problems that currently prevent confidence in the study results. Most importantly, the manuscript does not provide a reproducible procedure for identifying, selecting, and coding the 31 projects, and the resulting sample appears to combine replication projects with reproducibility studies, reviews, and methodological or theoretical papers. The reviewers also identify an insufficiently described analytical strategy. In addition, both reviewers question whether the analyses support the manuscript’s central claims: comparisons of project-level replication rates and funnel plots do not directly test post hoc criterion selection or selective reporting, and the manuscript does not provide the within-project comparisons needed to establish how strongly PVRC and CIRC alter conclusions. The conceptual justification for focusing on these two criteria and the connection between the manuscript’s theoretical argument and empirical analyses also require substantial development. There reviewers are consistent in their concerns that these issues require substantial reconstruction of the dataset, methods, analyses, and arguments of the paper. I agree with this assessment and think the paper requires a lot of work to meet the threshold of valid and informative science.

Recommendations from the editor

Clearly define the target population of replication projects and reconstruct the dataset using a transparent and reproducible search strategy, explicit inclusion and exclusion criteria, and a documented data-extraction and coding procedure. In particular, distinguish replication studies from reproducibility/reanalysis studies, reviews, and methodological or theoretical contributions.
Provide a substantially fuller description and justification of the statistical approach.
Strengthen the conceptual framework. Explain why PVRC and CIRC were selected from the broader set of replication-success criteria, what distinct statistical properties they capture, and under what circumstances differences between them should lead to meaningfully different scientific conclusions. The terminology distinguishing replication, replicability, reproduction, and reproducibility should be used consistently throughout.

Recommendations for enhanced transparency

All data should be published with proper documentation to ensure maximal reusability.
Analytical code files should be properly annotated. Consider including a codebook.

For more information on these recommendations, please refer to our author guidelines.

Competing interests: None.

Peer review 1

Marin Dujmović

DOI: 10.70744/MetaROR.435.1.rv1

The authors start from a premise that the inference about whether a replication attempt was successful or not is complicated one and can heavily depend on the analytical approach and influence conclusions. They pit two different criteria for making the decision against each other in a meta-analytical approach.

On its face this is an important issue, different choices lead to different conclusions about replicability. This, and the growing number of replication projects, provides fertile ground for rigorous assessment. First, from a theoretical and statistical point of view – one can characterise the various approaches to assessing replicability. Second, one could assess whether there are systematic differences in assessed replicability depending on the measure used. Third, one could assess which factors affect those differences.

This manuscript seems like it is on that path, but unfortunately does none of these things at an acceptable level. The first clue (which would turn out to be prophetic) was the mixing of replicability and reproducibility as interchangeable terms. In this field, they are not. Replicability refers to whether a replication attempt – collecting new data by running a new study – is successful (by whatever criteria one wants to adopt). The second refers to reproducing analyses from existing data. The fact that they are used interchangeably becomes important because upon inspection of an outlier result – the Avelino study managing to replicate only 5% of findings and being unknown to me even though it seems to be a large-scale project – I realized that the Avelino paper was not associated with a replication project, but a reproducibility one. And I quote from page 7 of the paper

“Our results show that transparency and reproduction are still in a development stage in Brazilian Political Science. Of the 650 articles reviewed, we asked for data to 197 quantitative articles. From those, only 28% agreed to share datasets and computed codes. We were able to attempt a replication for only 14%, and successfully reproduce the results of less than 5%”

Therefore, among the other issues, this manuscript and its analyses cannot be used to draw any inferences. Indeed, the discussion section of Tyner et al. (2026) has a sub-section on this issue which provides more value than the current manuscript.

Even if every aspect of the paper was at a high level otherwise, this very basic selection issue makes all of the analyses and inferences basically meaningless.

However, this same idea can be executed better

A more thorough statistical and theoretical overview of the various criteria in the introduction
A clear methodology (Cochrane review style) outlining how replication projects were searched for, what were the inclusion and exclusion criteria, how was quality and risk assessed etc.
Clear and precise description of the meta-analytic approach
A thorough analysis, quantitatively demonstrating rather than descriptively inferring
A thorough and careful discussion of the findings

There are many other issues apart from the manuscript not achieving any of the steps above (and it is not a full list). For starters, the shared data and code are not truly shared with FAIR principles in mind. There is no codebook explaining the data formatting or variables, and the code is not annotated. Second, if one re-runs the analysis, one gets a figure which does not match Figure 1 of the manuscript (notice it does not include all 31 studies). Table 1 has duplicate rows. And very few if any of the claims are backed up by stats (as basically none are reported), the Tyner et al. (2026) paper is sometimes cited as a pre-print and sometimes as the final publication etc.

Competing interests: None.

Peer review 2

Merle-Marie Pittelkow

DOI: 10.70744/MetaROR.435.1.rv2

Large-scale replication projects are central in discussions about the credibility of research fields. It is therefore crucial to be critical of their work and ask careful questions. It is therefore essential to scrutinise their methods, assumptions, and conclusions carefully.

This manuscript addresses an interesting and important question: whether conclusions drawn from large-scale replication projects depend on the criterion used to define replication success. The authors argue that their distinctive contribution is the attempt to take a “meta-perspective” on the replication literature by comparing reported replication rates across 31 replication projects. They focus on two criteria in particular: (1) the p-value-based replication criterion (PVRC) and the confidence-interval-based replication criterion (CIRC). The authors further argue that they investigate whether replication rates vary with project characteristics and whether the reporting of replication rates may itself be affected by selective reporting or publication bias.

The premise and research question are worthwhile. The observation that different replication success criteria can lead to different assessments of the same body of evidence deserves systematic investigation. However, the manuscript in its current form contains substantial conceptual, methodological, and reporting flaws, which I detail below. In my opinion, these concerns raise serious doubts about the findings.

Major concerns.

1. I do not have confidence in the data construction and analysis.

My most serious concern is the construction of the dataset and the data analysis.

The manuscript states that the sample contains 31 replication projects, but it does not explain how these projects were identified, selected, or coded. There is no (reproducible) search strategy, no clear inclusion and exclusion criteria, and no description of how information was extracted from the individual publications.
The authors do not report how the data was extracted. For example, how did they determine the overall replication rate for each project (e.g., Klein et al report both 11/13 or 10/13 studies successfully replicated. How did the authors decide for 11/13).
I also could not reconcile the studies in the dataset with the manuscript’s “List of Replication Projects.” Some studies appear in the dataset but not in the list, while some publications listed as replication projects do not appear in the analysis-ready data (see Appendix 1). I was left to wonder whether my assumption that the replication projects would list the projects that the data was extracted from was valid. If not, I wonder why the authors separated these references from the reference list.
Related, while reviewing the data, I noticed that some of the projects listed do not follow the definition of “Replication projects: projects that assess how many findings from some set of original papers are confirmed by corresponding replications” (see Appendix 1). For example, Ioannidis (2005) is primarily a theoretical and statistical argument about the reliability of published findings, while Nosek et al. (2022) is a review article on replicability, robustness, and reproducibility. Neither is itself a large-scale replication project. Nevertheless, both appear in the manuscript’s list and Nosek et al. (2022) also appears in the data analysis. Combining replication projects, reviews, methodological papers, and theoretical contributions produces a conceptually incoherent sample and undermine the data quality of this project. Other researchers have compiled comprehensive lists of reproducibility projects (including replication projects). For example, https://pmc.ncbi.nlm.nih.gov/articles/PMC12395386/ developed a comprehensive search strategy and eligibility criteria. I would strongly encourage the authors to use these projects either as a data source or as a methodological framework.
The data analysis strategy is not described. The manuscript does not provide enough information to assess the meta-analysis and meta-regression.
Table 1 further reduced my confidence in the data. The same variables appear multiple times with different values and sample sizes. For example, PVRC is reported once with N= 25 and again with N=35, while CIRC appears twice with N=18. Also, the number of features of replication projects does not sum to the total N of 31, which is likely since some articles report both. Still, a note to the readers would be useful in this case. These may be formatting errors, but in combination with the other discrepancies they reduce my confidence in the analyses.
I appreciate the authors openly sharing their data and code, but had difficulties going through the code since notes explaining the purpose of the code sections were absent. For the future, I recommend including explanations for the readers to improve the usefulness of openly sharing your code.

2. I am sceptical whether the analyses support the manuscript’s claims

Even if the dataset and statistical analysis were fully reliable, I am sceptical that the analyses support the authors’ conclusions.

The manuscript’s central argument concerns the possibility that researchers select or emphasise replication criteria after seeing the results, thereby producing more extreme or favourable narratives. However, the analyses do not examine whether:
- criteria were selected after the results were known;
- multiple criteria were calculated, but only some were reported;
- reported criteria differed from preregistered criteria;
- titles or abstracts emphasised one result over another;
- or authors changed their interpretation after observing the data.
  Instead, the manuscript compares reported project-level replication rates and examines funnel plots. This cannot establish post hoc criterion selection or selective reporting. To test the stated research question, the authors would need to compare preregistrations, protocols, available analyses, reported outcomes, and the claims emphasised in titles, abstracts, and discussions.
The interpretation of the funnel plots is particularly problematic to me. The manuscript argues that CIRC estimates are concentrated at more extreme values and that values near the average may be “missing,” suggesting publication or reporting bias. However, these plots contain highly heterogeneous project-level replication rates rather than comparable study-level effect estimates. Differences in field, project design, outcome selection, denominator, success criterion, and sampling frame could all produce asymmetry and are not discussed by the authors.
The conclusion that “PVRC and CIRC can materially affect the conclusions” is not supported by the data. I see great overlap for the two criteria for the studies reporting both in Figure 1. Only a comparison of the conclusions on the same data based on these two criteria would be sufficient to support this statement. The authors do not provide such comparisons. They only state “For example, Kvarven et al. (2020) reports PVRC greater than CIRC, whereas Sobkow et al. (2022) reports CIRC greater than PVRC.” However, they seem to ignore that the CIs overlap and the cases where PVRC and CIRC are close (e.g. Goncalves et al. (2022); Camerer et al. (2018), Schweisberg et al. (2016). I do not want to accuse the authors of knowingly omitting these cases, but it does seem like selective reporting to me.

3. The conceptual argument contains substantial logical gaps

The manuscript lacks clarity and has logical gaps. Here are examples,

“At the same time, the increased attention to open science practices, preregistration, and methodological reform has led some observers to argue that scientific practice may be improving (e.g., Korbmacher et al. 2023). However, broad empirical support for such optimism remains limited. Despite roughly a decade separating early landmark replication projects and more recent large-scale efforts, the available evidence continues to show similar levels of variability and only moderate robustness, especially“ (p.2)
Do the authors mean to imply that they believe that improvements in scientific practice would be linked to greater replication success, and how do they think this link would look? I am sceptical that the link is as straightforward as may be implied. I believe the relationship is not straightforward and requires a clear theoretical argument. Greater methodological rigour could reduce failed replications caused by questionable research practices or selective reporting. However, greater rigour could also reveal that effects are smaller, more uncertain, or more context-dependent than previously assumed. Improved scientific practice therefore does not necessarily imply higher replication rates.
“These criteria do not measure identical properties of the data, and their use can yield substantially different replication rates.“
Imo, more detail needed. What are the different properties of the data that the authors imply here?
“These criteria do not measure identical properties of the data, and their use can yield substantially different replication rates. In OSC (2025), for example, the weighted PVRC was reported as 49%, whereas the CIRC was reported as 31%. This discrepancy highlights that the definition of replication success is not merely technical; it can materially affect the resulting narrative.
No specification of the different properties of data measured by the two criteria. How does the prevalence of the criteria relate to the narrative? It would be the conclusions drawn based on the criteria that would shape the narrative not the mere presence of one criterion over another. I am missing a convincing argument here.
“The possibility of ex post selection of replication criteria is especially concerning.”
The sentence does not seem grammatically sound, what is ex post selection? The authors did not show any arguments to support this statement.
The scope also changes throughout the paper. The abstract emphasizes social and behavioral sciences, the dataset includes several disciplines, and the results make claims about “science overall.” The included sample cannot support general conclusions about science as a whole.
The introduction does not clearly connect the broader discussion of the Replication Criteria and Investigator Discretion.
My impression is that over the past years, replication projects now use multiple replication criteria to triangulate their results and see how the choice of different crtieria impact their results. The authors do not

4. The choice of replication-success criteria is insufficiently justified

The authors focus on PVRC and CIRC and describe them as widely used, but they do not provide evidence that these are the most frequent, influential, or theoretically relevant criteria. Many other replication-success measures exist, including effect-direction agreement, effect-size consistency, prediction intervals, equivalence tests, meta-analytic criteria, and Bayesian measures and could have been potential targets. The manuscript should explain why these two criteria were selected specifically and provide evidence to support their selection.
It should also engage more thoroughly with existing literature directly comparing or discussing replication-success measures. For example,
- https://doi.org/10.1098/rsos.201697 (potential CoI since we were colleagues at the time that this was written, but I have formal connection to this paper and no gain from a citation).
- https://doi.org/10.1098/rsos.252263 (seems like an interesting starting point when considering replication criteria)
- or this FORRT resource https://forrt.org/fred/articles/success_criteria.html
Several replication projects report multiple success criteria to triangulate their findings and assess whether conclusions depend on the chosen criterion (e.g., https://doi.org/10.1038/s41562-024-02062-9). The manuscript does not sufficiently consider this practice and risks treating differences between criteria as evidence of selective reporting rather than as transparent robustness checks.

In summary,

Are the ideas clearly communicated? No

Is the relevant existing literature described and discussed? No
Are the references appropriate? Yes (except the replication projects that are not replication projects).
Are all research methods appropriate, rigorous, and clearly described? No
Are the results clearly presented? Are graphs easily readable and clearly labelled? Do they contain descriptive captions? Figures yes, table no
Are the results discussed in the context of previous findings? Are differences from previously reported findings clearly described? No
Are interpretations and conclusions consistent with the empirical materials and data? No
Are the limitations of the study and their implications for interpreting the results clearly described where applicable? No

5. Conclusion

In a traditional journal format, I would recommend rejection of this paper. This recommendation is based primarily on the inability to verify and trust the analytical dataset and on the mismatch between the research aim and the analyses performed. These problems would require more than clarification or additional robustness checks. The study would need a clearly defined sample, a fully reproducible extraction procedure, a reconciled dataset, and a substantially different design capable of examining criterion selection and selective reporting directly.

Competing interests: None.

Author response

DOI: 10.70744/MetaROR.435.1.ar

Thank you for two unusually thorough and constructive reviews. We will be direct: the reviewers are right on the central point, and we are grateful the problems were caught at this stage and in this format.

On the main criticism

Our sample conflated direct replication projects with computational reproducibility studies, literature follow-ups, secondary analyses of existing replication data, and review articles. Reviewer 1’s identification of Avelino et al. (2021) as a reproducibility study is correct, and Reviewer 2’s identification of Ioannidis (2005) and Nosek et al. (2022) as, respectively, a theoretical contribution and a review is likewise correct. There is a particular irony in a paper about criterion hygiene failing to apply the field’s most basic definitional distinction — replication (new data) versus reproduction (same data) — to its own sample. We accept this without reservation.

What we have done since receiving the reviews

We have completed a study-by-study re-audit of all 31 projects (plus Ioannidis 2005 from our project list), classifying each as: (A) direct replication project with new data collection; (B) computational reproducibility/reanalysis; (C) literature follow-up; (D) secondary analysis of existing replication-project data; (E) review/theory. The audit — with citations, DOIs, and a supporting quote for every non-obvious classification — yields 19 A, 4 B, 4 C, 3 D, and 2 E entries. It also surfaced additional extraction errors that the reviewers had no way to see, which we disclose proactively: most notably, our value for Marcoci et al. (2024) recorded the number of claims taken to replication (35/100) rather than a success rate (the correct figure is 19 of 29 analyzed replications), and our value for Niven et al. (2018) recorded the share of practices with any reproduction attempt (42%) rather than a success rate — moot for the main analysis, as the study is a literature follow-up, but indicative of the extraction problems the reviewers inferred. We also note that Protzko et al. (2023), whose retraction we discuss in the manuscript, nonetheless appeared with its originally reported rates in our dataset; the revision will handle it consistently (exclusion, with a sensitivity analysis).

How we will revise

Following the editor’s recommendations and Reviewer 2’s pointer to the systematic registry of replication and reproducibility projects, the revision will (i) reconstruct the sample from a documented search with explicit inclusion/exclusion criteria and a two-coder extraction protocol recording a quote and page for every data cell; (ii) restrict the main analysis to direct replication projects, with reproducibility studies analyzed separately and clearly labelled — or dropped; (iii) replace the funnel-plot argument, whose limitations both reviewers correctly identified, with the within-project paired comparison of criteria that Reviewer 2 rightly notes is the only design that can support our central claim; and (iv) either drop the post-hoc criterion-selection claim or test it properly, by comparing the preregistrations and registered protocols of the replication projects against their reported criteria.

We have since verified every extracted value against the primary sources. This verification found further errors that we disclose here: two values coded as confidence-interval-based criteria were in fact nothing of the kind (Sobków et al.’s “80%” is the share of successful plus partial replications under a significance criterion; Davis et al. report no CI-based rate at all), one PVRC value was actually a subjective-assessment rate (Cova et al.: 77.5% subjective vs. 78.4% significance-based), one value matched no reported statistic (Boyce et al.: our 35% turns out to be the share of cognitive-psychology studies in their Table 1), one project’s rate was another project’s (our Many Labs 5 value was the RP:P result for that study subset), and two denominators were wrong (Holzmeister et al.: 26, not 41; Errington et al.: our N of 53 was the number of papers selected, not analyzed).

On the verified data, the within-project comparison (n = 8 pairs; 7 excluding the retracted Protzko et al.) shows a mean absolute divergence between significance-based and interval-based criteria of 13 percentage points (maximum 26), with a non-significant tendency for interval-based criteria to be stricter in high-precision projects — and it additionally reveals that “CI-based” rates across projects mix distinct operationalizations (original-in-replication-CI, prediction intervals, effect-size compatibility) that must not be pooled unlabelled. We believe this reframing — criterion choice and criterion operationalization materially affect headline rates, without evidence of directional bias — is what the data can actually support. A first pass over the projects’ preregistrations and registered protocols additionally indicates that documented divergences between pre-specified and emphasized criteria exist but are case-specific (the post hoc aggregate scoring acknowledged by the Reproducibility Project: Cancer Biology; the preregistration misstatements cited in the Protzko et al. retraction) rather than general — which is, we think, a fairer and better-supported version of our original concern.

We will also bring our own materials up to the standard the paper preaches: a codebook, annotated analysis code, and figures that regenerate exactly from the released data.

Cite