Published at MetaROR

March 4, 2026

Table of contents

Cite this article as:

Hardwicke, T. E. (2025, November 14). Risk of bias in robustness reports. https://doi.org/10.31222/osf.io/wj26e_v2

Risk of bias in robustness reports

Tom Hardwicke1Email

1. University of Sydney

Originally published on November 17, 2025 at: 

https://doi.org/10.31222/osf.io/wj26e_v2

Abstract

Bartoš et al. propose a new article format — Synchronous Robustness Reports (SRRs) — in which journals invite researchers to conduct alternative analyses of recently accepted studies. The goal is to assess the robustness of the original results by exploring how they vary across different analysis choices. However, the proposal does not distinguish between SRRs planned before versus after analysts have seen the data. This distinction is crucial because data-dependent analyses introduce a risk of bias from selective reporting. In this commentary, I argue that without safeguards, SRRs could inadvertently create more confusion than clarity about robustness.

Bartoš et al. (2025) propose a new article format — Synchronous Robustness Reports (SRRs) — in which journals invite researchers to conduct alternative analyses of recently accepted studies. The goal is to assess the robustness of the original results by exploring how they vary across different analysis choices. However, the proposal does not distinguish between SRRs planned before versus after analysts have seen the data. This distinction is crucial because data- dependent analyses introduce a risk of bias from selective reporting (Hardwicke & Wagenmakers, 2023). In this commentary, I argue that without safeguards, robustness reports could inadvertently create more confusion than clarity about robustness.

On the pathway from raw data to results, researchers must make a variety of decisions, each of which has multiple defensible options; for example, which covariates to include, how to handle missing data, or whether to exclude outliers. Together, these decisions create a multiplicity of potential analyses and results, sometimes referred to as a “garden of forking paths” (Gelman & Loken, 2014).

Academic incentives and cognitive biases can lead researchers to exploit this analytic flexibility (perhaps unintentionally), selectively reporting results that align with their expectations or desired conclusions — a practice sometimes referred to as p-hacking or data-dredging (Stefan & Schönbrodt, 2023). Even when selective reporting is absent, Many Analyst studies have shown that different researchers make different analysis choices when using the same dataset to answer the same question, often leading to substantial variation in results (Aczel et al., 2021). Thus, multiplicity creates uncertainty about the extent to which the results of a single analytic approach are robust in the context of a wider distribution of plausible analysis approaches and results.

SRRs aim to reduce uncertainty about robustness by inviting independent researchers to run alternative analyses of recently accepted studies (Bartoš et al., 2025). When results converge across different analysis specifications, this indicates robustness and should increase confidence in the original result. By contrast, diverging results indicate fragility, which should decrease confidence. However, SRRs — as currently described — do not differentiate between robustness analyses planned before versus after analysts have seen the data, which introduces a risk of bias from data-dependent analyses and selective reporting (Hardwicke & Wagenmakers, 2023). The same incentives and biases that drive selective reporting of original research could also affect robustness analyses. Specifically, proponents of the original claim may be motivated to seek evidence for robustness, while skeptics might seek to highlight fragility. This concern is speculative in the context of the novel SRR format, but not unfounded: robustness analyses by authors in the social sciences often frame original results as robust and rarely reveal fragility (Leamer, 1983; Young & Holsteen, 2017). In sum, SRRs risk showcasing biased snapshots of the garden of forking paths, potentially giving a misleading impression of robustness or fragility.

How should we handle the risk of bias in robustness reports? Bartoš et al. (2025) briefly suggest that SRRs could be combined with preregistration (Hardwicke & Wagenmakers, 2023) or Registered Reports (Chambers & Tzavella, 2022), but do not elaborate. By contrast, I argue that integrating these approaches will be crucial to the success of SRRs. Preregistration involves publicly specifying an analysis plan before accessing the data, allowing readers to distinguish between data-dependent and data-independent analyses (Hardwicke & Wagenmakers, 2023). Typically, preregistration occurs before data collection, ensuring that analysis plans are not influenced by the data. In the case of preexisting data, this guarantee is harder to enforce (Baldwin et al., 2022). However, SRRs present a unique opportunity: because SRRs are submitted before the target article is published, the data could be temporarily embargoed (kept private) until analysts preregister their analysis plans (Scott & Kline, 2019; Thibault et al., 2023). Once analyses are preregistered, the data are released to the analysts (akin to a library checkout system) and the analyses are run. After the embargo period, the data can be released publicly, to maximise ongoing access to the scientific community; however, any subsequent robustness analyses are no longer guaranteed to be data-independent.

Expanding the preregister-checkout model into a full Registered Reports framework (Chambers & Tzavella, 2022) — Registered Robustness Reports — could offer additional benefits. Under this model, analysts would submit analysis plans for peer review before accessing the data and the journal would commit to publishing SRRs based solely on methodological quality, regardless of the outcome. This would reduce both the risk of bias from data-dependent analyses and the risk of publication bias (i.e., analysts/editors preferentially submitting/publishing SRRs based on their results).

Registered Robustness Reports might also improve SRR quality by incorporating peer review feedback before analyses are conducted. Bartoš et al. (2025) propose that quality is maintained by journal editors inviting analysts to submit SRRs; however, such editorial discretion may introduce selection bias — for example, if editors preferentially invite proponents or skeptics, or attempt to engineer balance. Registered Robustness Reports could mitigate these issues by issuing an open call for analysis proposals and selecting SRRs based solely on methodological merit.

A potential concern with withholding data access until analysis plans are submitted is that partial inspection of data can benefit statistical validity (Sarafoglou et al., 2023). This can be addressed through an Explore and Confirm Workflow (Thibault et al., 2023) in which a subset of the data, synthetic data, or masked data are released prior to preregistration for the purposes of exploratory (data-dependent) analysis. The main dataset is reserved for confirmatory (data-independent) analyses and only released after preregistration. Another concern is delayed publication of the original study and demand on journal resources. While some delay is inevitable, it could be reduced by scheduling SRR reviewers in advance, a system that has been used in the standard Registered Reports format (Chambers & Tzavella, 2022). Ultimately, journals considering introducing SRRs will need to weigh these costs against the benefits of reducing bias.

Conclusion

Analytic flexibility introduces uncertainty about the robustness of results derived from any single analytic approach (Aczel et al., 2021). SRRs aim to illuminate this uncertainty by publishing independent robustness analyses alongside target articles (Bartoš et al., 2025). However, because SRRs do not distinguish between data-independent and data-dependent robustness analyses, they are vulnerable to bias from selective reporting and may therefore provide a misleading impression of robustness or fragility. Combining SRRs with preregistration or Registered Reports would identify when robustness checks are data-independent, thereby clarifying or minimizing the risk of bias. When the risk of bias is reduced, SRRs can support a more transparent and informed debate about why different analytic approaches yield different results.

Competing interests

The author declares no competing interests.

Acknowledgements

I’m grateful to Simine Vazire and Rose O’Dea for feedback on an earlier version of this manuscript. The views expressed are my own.

References

Aczel, B., Szaszi, B., Nilsonne, G., Van Den Akker, O. R., Albers, C. J., Van Assen, M. A., Bastiaansen, J. A., Benjamin, D., Boehm, U., Botvinik-Nezer, R., Bringmann, L. F., Busch, N. A., Caruyer, E., Cataldo, A. M., Cowan, N., Delios, A., Van Dongen, N. N., Donkin, C., Van Doorn, J. B., … Wagenmakers, E.-J. (2021). Consensus-based guidance for conducting and reporting multi-analyst studies. eLife, 10, e72185. https://doi.org/10.7554/eLife.72185

Baldwin, J. R., Pingault, J.-B., Schoeler, T., Sallis, H. M., & Munafò, M. R. (2022). Protecting against researcher bias in secondary data analysis: Challenges and potential solutions. European Journal of Epidemiology, 37(1), 1–10. https://doi.org/10.1007/s10654-021-00839-0

Bartoš, F., Sarafoglou, A., Aczel, B., Hoogeveen, S., Chambers, C. D., & Wagenmakers, E.-J. (2025). Introducing synchronous robustness reports. Nature Human Behaviour, 9(4), 635–637. https://doi.org/10.1038/s41562-025-02129-1

Chambers, C. D., & Tzavella, L. (2022). The past, present and future of Registered Reports. Nature Human Behaviour, 6, 1–14. https://doi.org/10.1038/s41562-021-01193-7

Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460–465. https://doi.org/10.1511/2014.111.460

Hardwicke, T. E., & Wagenmakers, E.-J. (2023). Reducing bias, increasing transparency and calibrating confidence with preregistration. Nature Human Behaviour, 7(1), 15–26. https://doi.org/10.1038/s41562-022-01497-2

Leamer, E. E. (1983). Let’s take the con out of econometrics. The American Economic Review, 73(1), 31–43. https://www.jstor.org/stable/1803924

Sarafoglou, A., Hoogeveen, S., & Wagenmakers, E.-J. (2023). Comparing analysis blinding with preregistration in the many-analysts religion project. Advances in Methods and Practices in Psychological Science, 6(1), 1–19. https://doi.org/10.1177/2515245922112831

Scott, K. M., & Kline, M. (2019). Enabling confirmatory secondary data analysis by logging data checkout. Advances in Methods and Practices in Psychological Science, 2(1), 45–54. https://doi.org/10.1177/2515245918815849

Stefan, A. M., & Schönbrodt, F. D. (2023). Big little lies: A compendium and simulation of p-hacking strategies. Royal Society Open Science, 10(2), 220346. https://doi.org/10.1098/rsos.220346

Thibault, R. T., Kovacs, M., Hardwicke, T. E., Sarafoglou, A., Ioannidis, J. P. A., & Munafò, M. R. (2023). Reducing bias in secondary data analysis via an Explore and Confirm Analysis Workflow (ECAW): A proposal and survey of observational researchers. Royal Society Open Science, 10(10), 230568. https://doi.org/10.1098/rsos.230568

Young, C., & Holsteen, K. (2017). Model uncertainty and robustness: A computational framework for multimodel analysis. Sociological Methods & Research, 46(1), 3–40. https://doi.org/10.1177/0049124115610347

Editors

Kathryn Zeiler
Editor-in-Chief

Kathryn Zeiler
Handling Editor

Editorial assessment

by Kathryn Zeiler

DOI: 10.70744/MetaROR.287.1.ea

The author provides rationale to support the author’s suggestion that journal editors require Synchronous Robustness Reports (SRRs) to be preregistered. The reviewer suggests distinguishing between robustness checks, which are generally included in the original article and performed by the original study’s author to defend the main results and reanalyses, which are performed by others after a paper is published. This distinction likely arises given different uses of the terms across fields (e.g., economists use “robustness checks” to refer to author analyses that are included in the original study and designed to increase confidence in the main results). The reviewer is skeptical about requiring preregistration given that much can be learned from exploring the data and reanalysis options.

Recommendations from the editor

The author cautions against the usefulness of SRRs in cases where SRR authors observe the data prior to deciding on their analysis plan, which stands as an alternative to the plan executed in the original study. Given existing evidence of high rates of irreproducibility in the absence of preregistration, the suggestion to require preregistration of SRR analysis plans seems important. In addition, the claim suggesting that liking or disliking the original results can bias SRR results seems plausible considering existing evidence on researcher biases the author relies on. The idea that editors issue open calls for SRRs is a great one. Editor bias is as potentially harmful as analyst bias.

Suggestions:

  1. The author notes that “SRRs are submitted before the target article is published” (2), but Bartoš et al. suggest that a “journal can…choose either to publish the target article first or to delay the publication of the target article until the SRRs are ready to be published alongside it” and that “the gap between the potential publication of the target article and the accompanying SRRs is, at most, six weeks.” (2) Thus, the author might want to propose a fuller alternative workflow.
  1. The author suggests embargoing the data used in the original study until SRRs are submitted, but what about the fact that an increasing number of authors are posting not only preprints prior to publication but also data and analysis scripts to invite scrutiny as soon as possible. Might a policy embargoing data used to produce the original results lead to a delay in reliability checks? This begs the question whether we need to adjust how we perform original research. Specifically, should researchers (or editors of Registered Reports) invite SRR research plans after submission of the original research plan but prior to the original author’s performance of the analysis? In other words, imagine a journal that publishes only Registered Reports. Accepted plans for original research would trigger a call for plans for SRRs. All accepted SRR plans, just like the original plan, would be conditionally accepted before any analyses are performed by either the original author or SRR authors. Could we imagine an alternative method for conducting research that includes authors themselves issuing open calls for SRRs that get added to the original paper prior to submission (with authors including all SRRs along with author comments on each)?

Recommendations for Enhanced Transparency

  • Add a Data Availability Statement to report that no data are used in the article.
  • Add author ORCID iD.
  • Add a funding source statement. Authors should report all funding in support of the research presented in the article. Grant reference numbers should be included. If no funding sources exist, explicitly state this in the article.

For more information on these recommendations, please refer to our author guidelines.

Competing interests: None.

Peer review 1

Andrew Gelman

DOI: 10.70744/MetaROR.287.1.rv1

This paper crisply argues in favor of preregistration of reanalysis of published studies. The author uses the term “robustness reports,” which I gather is standard terminology, but I have to admit that this term bothers me, in that I think it inappropriately pushes people toward an attitude that the goal is to find out whether the original published finding was “robust” or not. Rather than framing this in terms of “robustness,” I’d prefer to just think of these as reanalyses. This complaint of mine is not a criticism of this paper; it’s just more of a general concern which perhaps the authors could think about and mention in a revision.

What is the distinction between “robustness study” and “reanalysis”? I’m not quite sure. In one sense, a reanalysis is literally a check of robustness to an alternative specification. I guess the difference to me is that “reanalysis” sounds like it can stand on its own, whereas “robustness” is typically defined relative to the original choice of analysis. To put it another way, “reanalysis” sounds like it’s coming from an outsider, whereas a “robustness study” is typically done defensively, so as to provide a result with some buffer against criticism.

Here’s how Uri Simonsohn [1] put it: “Robustness checks involve reporting alternative specifications that test the same hypothesis. Because the problem is with the hypothesis, the problem is not addressed with robustness checks.”  I added some discussion of my own [2].  So maybe my problem is with the cultural baggage surrounding the idea of robustness.

To return to the main point, the proposal to have a formal system combining preregistration with reanalysis seems fine to me—as long as this were not taken to rule out non-preregistered reanalyses. Over the years, I’ve done many many reanalyses of studies published by others, and none of these were preregistered. I wouldn’t want them to be preregistered. When I reanalyze data, I need to try out different things and see how they work, and I learn a lot after looking at the data.

I think it’s great to have preregistered reanalyses. I just don’t think this should be any kind of requirement.

References

[1] Uri Simonsohn (2016).  P-hacked hypotheses are deceivingly robust.  http://datacolada.org/48/

[2] Andrew Gelman (2018).  Robustness checks are a joke.  https://statmodeling.stat.columbia.edu/2018/11/14/robustness-checks-joke/

Competing interests: None.

Leave a comment