Evaluated works
Research Articles
Article
Adrian Barnett, Matt Spick
PLOS journals allow authors of accepted articles to choose whether the peer review will be openly published alongside the article. This creates an observational study to examine the characteristics of authors and articles that more often choose open peer review, and whether open review is associated with measures of article quality. We examined over 115,000 PLOS articles and estimated what characteristics of the articles were associated with open peer review. We also examined if open peer review was associated with the subsequent retraction of the article and the number of citations.
Forty percent of articles chose open peer review. Authors from the UK, France, the Netherlands, and Ethiopia were more likely to choose open peer review. In contrast, authors from Saudi Arabia, the Republic of Korea, Pakistan, Poland, and China were less likely to choose open review. Authors with an edu email were less likely to choose open review, whilst authors with a gmail were more likely. Articles with open peer review were less likely to be retracted (adjusted hazard ratio = 0.75, 95% CI 0.60 to 0.93) and had more citations on average (adjusted rate ratio = 1.07, 95% CI 1.05 to 1.09).
In this exploratory study, we found clear differences in participation in open reviews with strong differences between countries. Authors who have confidence in their article and who engage in other open science practices may be more likely to chose open peer review, making this choice an indicator of article quality.
November 19, 2026
Article
Robbie Clark, Lorna Duncan, Pen-Yuan Hsing, Neil Jacobs, Rosalind Strang
Meta-research has investigated various concerns around ‘research culture’ in Higher Education Institutions (HEIs). Despite sector-wide efforts to improve institutional cultures, evidence suggests that such initiatives are seldom evaluated. This study explored how meta-research informs decisions affecting research culture within English HEIs, along with the mechanisms which enable or hinder meta-research from impacting decisions. Interviews were conducted with 14 institution-level ‘decision-makers’ from 11 diverse English HEIs, plus 7 meta-researchers in the UK. Data were analysed using deductive content analysis.
Participants discussed how research has greatly informed HEI strategic directions to improve research culture. This primarily happened indirectly, brought into decisions by advocates and extrinsic forces. Some institutions have also conducted local research into their own cultures. Nevertheless, innovations to improve research culture are less evidence-based and infrequently evaluated. Participants identified several mechanisms by which meta-research could impact on such decisions. Engagement, consultation, and co-creation could make meta-research more impactful, whereas career stage, prestige, and personal connections could either enable or hinder researchers from achieving impact. Generating sufficiently persuasive evidence to inform impactful decisions was an additional barrier to meta-research impact. Lastly, findings which reflected poorly on institutions or individuals could reduce uptake or sharing of results.
This research highlights the need to more rigorously evaluate the impact of research culture reforms. Given the scale of institutional changes, the research encourages sector-wide collaboration and sharing of findings, suggesting ways in which researchers and institutions can seek to maximise the impact of meta-research on research culture.
November 16, 2026
Article
Anna Butters, Melanie Benson Marshall, Stephen Pinfield, and Tom Stafford
Background - Distributed peer review (DPR) of grant proposals, where applicants review the proposals of others applying to the same funding call, offers advantages, including increased feedback to applicants. With the reviewer pool being determined by the applicants to a particular call, there are, however, concerns that reviewers may not possess the necessary expertise and experience to provide quality reviews and feedback. We analysed review comments from recent trials of DPR for funding evaluation at the Volkswagen Foundation to obtain insights into reviewing and the impact of reviewer characteristics on review content. Method - Review comments across two years (N=2950) were classified (content and sentiment) using existing, fine-tuned machine learning classifier models. Mean prevalence of content categories and sentiment was calculated, both overall and stratified by reviewer/proposal characteristics. Results - Review comments were consistent with review scores and overall ranking of proposals. Minor differences in content and sentiment according to reviewer gender and research experience (years working in research) were detected. There was minimal variation according to funding peer review experience and disciplinary alignment between reviewer and proposal. Conclusion - Existing classifier models can be used to analyse grant review comments and to obtain insights into reviewer consistency and adherence to funding criteria. Concerns about the influence of reviewer experience and expertise on reviews may be unfounded. Results from analyses of review comments can be used to audit and refine review processes.
November 10, 2026
Article
Lee Jones, Adrian Barnett, Gunter Hartel, Dimitrios Vagenas
Background: Reproducibility concerns in health research have grown, as many published results fail to be independently reproduced. Achieving computational reproducibility, where others can replicate the same results using the same methods, requires transparent reporting of statistical tests, models, and software use. While data-sharing initiatives have improved accessibility, the actual usability of shared data for reproducing research findings remains underexplored. Addressing this gap is crucial for advancing open science and ensuring that shared data meaningfully support reproducibility and enable collaboration, thereby strengthening evidence-based policy and practice.
Methods: A random sample of 95 PLOS ONE health research papers from 2019 reporting linear regression was assessed for data-sharing practices and computational reproducibility. Data were accessible for 43 papers. From the randomly selected sample, the first 20 papers with available data were assessed for computational reproducibility. Three regression models per paper were reanalysed.
Results: Of the 95 papers, 68 reported having data available, but 25 of these lacked the data required to reproduce the linear regression models. Only eight of 20 papers we analysed were computationally reproducible. A major barrier to reproducing the analyses was the great difficulty in matching the variables described in the paper to those in the data. Papers sometimes failed to be reproduced because the methods were not adequately described, including variable adjustments and data exclusions.
Conclusion: More than half (60%) of analysed studies were not computationally reproducible, raising concerns about the credibility of the reported results and highlighting the need for greater transparency and rigour in research reporting. When data are made available, authors should provide a corresponding data dictionary with variable labels that match those used in the paper. Analysis code, model specifications, and any supporting materials detailing the steps required to reproduce the results should be deposited in a publicly accessible repository or included as supplementary files. To increase the reproducibility of statistical results, we propose a Model Location and Specification Table (MLast), which tracks where and what analyses were performed. In conjunction with a data dictionary, MLast enables the mapping of analyses, greatly aiding computational reproducibility.
October 15, 2026
Article
André Brasil, Soraia de Queiroz Costa, Gabriela da Rocha Barbosa, Roberta Giraldi Romano, Cláudia Terezinha Kniess, José Carmino Gomes Junior, Henyo Trindade Barretto Filho, Flaviane de Magalhães Barros Bolzan de Morais, Oswaldo E. do Amaral, Evanilde Benedito, Fernando Luiz Westphal, Edna Ursulino Alves, Maria Teresa Gomes Lopes, Tania Mari Bellé Bresolin, Valdir Florêncio da Veiga Junior, Rômulo Dante Orrico Filho, Paulo Ricardo Merísio, Terezinha Valim Oliver Gonçalves, and Carlos Alberto Cioce Sampaio
Mapping research contributions to the Sustainable Development Goals (SDG) remains a global challenge, particularly in the Global South, where language barriers, data gaps, and limited database coverage undermine comprehensive analysis. This article presents a mixed-methods framework developed by a national committee appointed by the Brazilian Agency for Support and Evaluation of Graduate Education (CAPES) to assess the contributions of Brazil’s graduate research system to the 2030 Agenda. Drawing on the STRINGS methodology, the project developed bilingual descriptors and applied them to the Sucupira Platform, which records data from all accredited master’s and doctoral programmes in the country. This approach enabled a large-scale and reproducible mapping of SDG-related theses and dissertations, complemented by three nationally defined goals—Ethnic-Racial Equality (SDG 18), Art, Culture, and Communication (SDG 19), and Rights of Indigenous Peoples and Traditional Communities (SDG 20). Benchmarking against OpenAlex provided an international reference point, while qualitative components—including disciplinary surveys, consultations with state departments of education, and programme-level case studies—added interpretive depth and contextual specificity. The Brazilian experience demonstrates how transparent, adaptable, and policy relevant methods can support more inclusive and responsible approaches to mapping research contributions to sustainable development.
October 2, 2026
Article
Hans de Jonge, Katharina Rieck
The importance of having funding metadata openly available is widely acknowledged for research transparency and tracking research funding outcomes. Since 2013, Crossref has provided members the opportunity to deposit funding information when registering DOIs for publications. However, earlier research shows this information is far from complete, with coverage varying significantly across publishers despite funding information often being available in article acknowledgement sections.
This paper quantifies this metadata gap at approximately 30%. Using publications from two national funding councils (the Dutch research council NWO and the Austrian Science Fund FWF), we demonstrate that 30% of funding acknowledgements readily available to publishers in full-text articles are not transferred as metadata to Crossref. We also observe considerable differences between publishers.
This work provides, for the first time, a concrete baseline for improving funding metadata quality and coverage in Crossref. It may also inform publishers about their performance, many of whom may be unaware of these gaps due to outsourced metadata extraction processes.
September 10, 2026
Article
Heidi J. Imker
For decades, life science researchers have had cost-free, unrestricted access to data through online databases. However, the sustainability of even well-established resources was already tenuous, and abrupt changes in science funding in the United States seems poised to exacerbate these challenges. This study employed a multiple mini case study approach, triangulating semi-structured interviews with supplemental documentation to investigate 9 diverse, long-standing databases that had previously received support from U.S. federal agencies. The research explored each database’s purpose and use, examined current and emerging funding strategies, and considered the potential consequences if any of these databases were forced to shut down. Participating databases include: 1) BHL: Biodiversity Heritage Library, 2) MorphoBank, 3) OMIM: Online Mendelian Inheritance in Man, 4) ORDB: Olfactory Receptor Database, 5) rrnDB: ribosomal RNA operon copy number database, 6) VEuPathDB: Eukaryotic Pathogen, Vector, and Host Informatics Resources, 7) WormAtlas, and 8-9) two databases that wished to remain anonymous. Findings revealed anticipation of higher barriers to data access and reuse, loss of subject matter expertise now and into the future, and lost or interrupted opportunities. Additionally, real impacts have already begun through redirection of energy, abrupt reductions in support, and increased competition for funding. Sustainability models in light of the current funding outlook in the US are discussed.
September 9, 2026
Article
Yu Zhu, Jiyuan Ye
The lack of a macro-level, systematic evaluation theory to guide the implementation of evaluation practices has become a key bottleneck in the reform of global research evaluation systems. By reviewing the historical development of research evaluation, this paper highlights the current binary opposition between qualitative and quantitative methods in evaluation practices. This paper introduces the System of All-round Evaluation of Research (SAER), a framework that integrates form, content, and utility evaluations with six key elements. SAER offers a theoretical breakthrough by transcending the binary, providing a comprehensive foundation for global evaluation reforms. The comprehensive system proposes a trinity of three evaluation dimensions, combined with six evaluation elements, which would help academic evaluators and researchers reconcile binary oppositions in evaluation methods. The system highlights the dialectical wisdom and experience embedded in Chinese research evaluation theory, offering valuable insights and references for the reform and advancement of global research evaluation systems.
September 4, 2026
Article
José Rolando Villaseñor Agúndez, Heinrich Nax
Large-scale replication projects have become central to debates about reproducibility in the social and behavioral sciences and related fields. In one of the most recent replication projects (Tyner et al. 2026), 274 positive results from 164 published papers between 2009 and 2018 were subjected to replication attempts. Depending on the criterion applied to evaluate replication success, between 28.6% and 74.8% of results replicated successfully. A key conclusion of this project was that the conditions that accept or reject replicability need further investigation. Building on that insight, the present study takes a meta-perspective on reproducibility by examining replication projects themselves. In particular, it focuses on how the choice of replication criterion may shape the conclusions that are reported. Two widely used criteria are the p-value-based replication criterion (PVRC), which assesses whether an originally significant effect remains statistically significant in the same direction in the replication, and the confidence-interval-based replication criterion (CIRC), which assesses whether the confidence intervals of the original and replication studies overlap. Using a sample of 31 replication projects across psychology, behavioral and social sciences, psychiatry, and behavioral ecology, this study shows substantial heterogeneity in reported replication rates both across projects and within projects depending on the criterion applied. Meta-analytic evidence suggests no overall time trend toward improved replicability and no robust association of replication rates with field, authorship patterns, or journal impact. At the same time, funnel-plot patterns indicate that reported CIRC estimates may be selectively biased toward more extreme values. These findings raise the possibility of publication bias and reporting in favor of “extreme results” operating not only in original studies, but also in the meta-literature on replication itself.
September 3, 2026
Article
Dorothea Strecker, Heinz Pampel, Jonas Höfting
This article presents the results of a survey conducted in 2024 among research performing organizations (RPOs) in Germany on how they collect data on publication costs. Of the 583 invitees, 258 (44.3%) completed the questionnaire. This survey is the first comprehensive study on the recording of publication costs at RPOs in Germany. The results show that the majority of surveyed RPOs recorded publication costs at least in part. However, procedures in this regard were often non-binding. Respondents' ratings of the reliability of the collection of data on publication costs varied by the source of publication funding. Eighty percent of respondents rated the contribution of collecting data on publication costs to shaping the open access transformation as "very important" or "important." Yet, these data were used as a basis for strategic decisions in only 59% of the surveyed RPOs. Moreover, most respondents considered the implementation of an information budget at their institutions by 2025 unlikely. We discuss the implications of these findings for the open access transformation.
August 13, 2026











