Evaluated works
Research Articles
Article
Hans de Jonge, Katharina Rieck
The importance of having funding metadata openly available is widely acknowledged for research transparency and tracking research funding outcomes. Since 2013, Crossref has provided members the opportunity to deposit funding information when registering DOIs for publications. However, earlier research shows this information is far from complete, with coverage varying significantly across publishers despite funding information often being available in article acknowledgement sections.
This paper quantifies this metadata gap at approximately 30%. Using publications from two national funding councils (the Dutch research council NWO and the Austrian Science Fund FWF), we demonstrate that 30% of funding acknowledgements readily available to publishers in full-text articles are not transferred as metadata to Crossref. We also observe considerable differences between publishers.
This work provides, for the first time, a concrete baseline for improving funding metadata quality and coverage in Crossref. It may also inform publishers about their performance, many of whom may be unaware of these gaps due to outsourced metadata extraction processes.
September 10, 2026
Article
Heidi J. Imker
For decades, life science researchers have had cost-free, unrestricted access to data through online databases. However, the sustainability of even well-established resources was already tenuous, and abrupt changes in science funding in the United States seems poised to exacerbate these challenges. This study employed a multiple mini case study approach, triangulating semi-structured interviews with supplemental documentation to investigate 9 diverse, long-standing databases that had previously received support from U.S. federal agencies. The research explored each database’s purpose and use, examined current and emerging funding strategies, and considered the potential consequences if any of these databases were forced to shut down. Participating databases include: 1) BHL: Biodiversity Heritage Library, 2) MorphoBank, 3) OMIM: Online Mendelian Inheritance in Man, 4) ORDB: Olfactory Receptor Database, 5) rrnDB: ribosomal RNA operon copy number database, 6) VEuPathDB: Eukaryotic Pathogen, Vector, and Host Informatics Resources, 7) WormAtlas, and 8-9) two databases that wished to remain anonymous. Findings revealed anticipation of higher barriers to data access and reuse, loss of subject matter expertise now and into the future, and lost or interrupted opportunities. Additionally, real impacts have already begun through redirection of energy, abrupt reductions in support, and increased competition for funding. Sustainability models in light of the current funding outlook in the US are discussed.
September 9, 2026
Article
Yu Zhu, Jiyuan Ye
The lack of a macro-level, systematic evaluation theory to guide the implementation of evaluation practices has become a key bottleneck in the reform of global research evaluation systems. By reviewing the historical development of research evaluation, this paper highlights the current binary opposition between qualitative and quantitative methods in evaluation practices. This paper introduces the System of All-round Evaluation of Research (SAER), a framework that integrates form, content, and utility evaluations with six key elements. SAER offers a theoretical breakthrough by transcending the binary, providing a comprehensive foundation for global evaluation reforms. The comprehensive system proposes a trinity of three evaluation dimensions, combined with six evaluation elements, which would help academic evaluators and researchers reconcile binary oppositions in evaluation methods. The system highlights the dialectical wisdom and experience embedded in Chinese research evaluation theory, offering valuable insights and references for the reform and advancement of global research evaluation systems.
September 4, 2026
Article
José Rolando Villaseñor Agúndez, Heinrich Nax
Large-scale replication projects have become central to debates about reproducibility in the social and behavioral sciences and related fields. In one of the most recent replication projects (Tyner et al. 2026), 274 positive results from 164 published papers between 2009 and 2018 were subjected to replication attempts. Depending on the criterion applied to evaluate replication success, between 28.6% and 74.8% of results replicated successfully. A key conclusion of this project was that the conditions that accept or reject replicability need further investigation. Building on that insight, the present study takes a meta-perspective on reproducibility by examining replication projects themselves. In particular, it focuses on how the choice of replication criterion may shape the conclusions that are reported. Two widely used criteria are the p-value-based replication criterion (PVRC), which assesses whether an originally significant effect remains statistically significant in the same direction in the replication, and the confidence-interval-based replication criterion (CIRC), which assesses whether the confidence intervals of the original and replication studies overlap. Using a sample of 31 replication projects across psychology, behavioral and social sciences, psychiatry, and behavioral ecology, this study shows substantial heterogeneity in reported replication rates both across projects and within projects depending on the criterion applied. Meta-analytic evidence suggests no overall time trend toward improved replicability and no robust association of replication rates with field, authorship patterns, or journal impact. At the same time, funnel-plot patterns indicate that reported CIRC estimates may be selectively biased toward more extreme values. These findings raise the possibility of publication bias and reporting in favor of “extreme results” operating not only in original studies, but also in the meta-literature on replication itself.
September 3, 2026
Article
Dorothea Strecker, Heinz Pampel, Jonas Höfting
This article presents the results of a survey conducted in 2024 among research performing organizations (RPOs) in Germany on how they collect data on publication costs. Of the 583 invitees, 258 (44.3%) completed the questionnaire. This survey is the first comprehensive study on the recording of publication costs at RPOs in Germany. The results show that the majority of surveyed RPOs recorded publication costs at least in part. However, procedures in this regard were often non-binding. Respondents' ratings of the reliability of the collection of data on publication costs varied by the source of publication funding. Eighty percent of respondents rated the contribution of collecting data on publication costs to shaping the open access transformation as "very important" or "important." Yet, these data were used as a basis for strategic decisions in only 59% of the surveyed RPOs. Moreover, most respondents considered the implementation of an information budget at their institutions by 2025 unlikely. We discuss the implications of these findings for the open access transformation.
August 13, 2026
Article
Daniel W. Hook
The debate about scholarly knowledge infrastructure has long been framed as a contest between openness and commercial enclosure. This framing distorts both policy and practice. The real tension lies between the persistent cost of producing and refining structured metadata under deep technological friction, and the differentiated demands distinct communities place on data quality, focus and granularity. We introduce the innovation annulus: the zone between freely available structured data and the advancing frontier of commercially refined knowledge products. This zone is a permanent, functional feature of the ecosystem -- not a pathology to eliminate. By analogy with the efficient market hypothesis, its width measures production inefficiency, set by the interplay of friction and demand. Artificial intelligence reshapes the annulus, lowering barriers to basic structuring, raising the threshold at which refinement adds value, and introducing systemic risks through unprovenanced AI-derived metadata. CRediT contributions, funding acknowledgements and AI disclosure statements illustrate the annulus lifecycle. Governance should calibrate the annulus, not abolish it: thin enough to serve research efficiently, wide enough to sustain innovation. A formal welfare framework, analogous to the Nordhaus optimal patent life, characterises the trade-offs and yields testable predictions. The Barcelona Declaration offers a promising forum for boundary governance.
August 12, 2026
Article
Lee Jones, Adrian Barnett, Gunter Hartel, Dimitrios Vagenas
Background: In health research, variability in modelling decisions can lead to different conclusions even when the same data are analysed, a challenge known as inferential reproducibility. In linear regression analyses, incorrect handling of key assumptions, such as normality of the residuals and linearity, can undermine reproducibility. This study examines how violations of these assumptions influence inferential conclusions when the same data are reanalysed.
Methods: We randomly sampled 95 health-related PLOS ONE papers from 2019 that reported linear regression in their methods. Data were available for 43 papers, and 20 were assessed for computational reproducibility, with three models per paper evaluated. The 14 papers that included a model at least partially computationally reproduced were then examined for inferential reproducibility. To assess the impact of assumption violations, differences in coefficients, 95% confidence intervals, and model fit were compared.
Results: Of the fourteen papers assessed, only three were inferentially reproducible. The most frequently violated assumptions were normality and independence, each occurring in eight papers. Violations of independence were particularly consequential and were commonly associated with inferential failure. Although reproduced analyses often retained the same binary statistical significance classification as the original studies, confidence intervals were frequently wider, indicating greater uncertainty and reduced precision. Such uncertainty may affect the interpretation of results and, in turn, influence treatment decisions and clinical practice.
Conclusion: Our findings demonstrate that substantial violations of key modelling assumptions often went undetected by authors and peer reviewers and, in many cases, were associated with inferential reproducibility failure. This highlights the need for stronger statistical education and greater transparency in modelling decisions. Rather than applying rigid or misinformed rules, such as incorrectly testing the normality of the outcome variable, researchers should adopt modelling frameworks guided by the research question and the study design. When assumptions are violated, appropriate alternatives, such as robust methods, bootstrapping, generalized linear models, or mixed-effects models, should be considered. Given that assumption violations were common even in relatively simple regression models, early and sustained collaboration with statisticians is critical for supporting robust, defensible, and clinically meaningful conclusions.
August 10, 2026
Article
Simon Porter, Daniel Hook
Bibliographic data is a rich source of information that goes beyond the use cases of location and citation -- it also encodes both cultural and technological context. For most of its existence, the scholarly record has changed slowly and hence provides an opportunity to gain insight through its reflection of the cultural norms of the research community over the last four centuries. While it is often difficult to distinguish the originating driver of change, it is still valuable to consider the motivating influences that have led to changes in the structure of the scholarly record. An "initial era" is identified during which initials were used in preference to full names by authors on scholarly communications. Causes of the emergence and demise of this era are considered as well as the implications of this era on research culture and practice.
August 5, 2026
Article
Zhicheng Lin
Scientific communication faces a dual crisis: exponential publication growth—now accelerated by AI-assisted writing—overwhelms human readers and reviewers, while fragmented research practices block automated synthesis. The behavioral and social sciences in particular suffer from incomparable stimulus databases, jingle–jangle measurement fallacies (same label, distinct constructs; different labels, same construct), and contextual blindness that conceals effect heterogeneity. Current AI tools can summarize papers but cannot synthesize findings across incommensurable studies; they also risk amplifying biases when trained on unstructured, unverified text. I propose restructuring scientific papers for dual audiences: front-loaded narratives for time-pressed human readers, paired with research-object packages containing executable code, semantic annotations, and tidy trial-level data. This design makes papers queryable research environments: readers can interrogate data and probe analytic choices in real time, while research-object packages enable automated verification and AI-assisted peer review grounded in executable evidence rather than narrative claims. Such papers become nodes in continuously updated evidence networks: each publication automatically contributes effect sizes to living, versioned meta-analyses, with corrections and retractions propagating through dependent analyses. Widespread adoption will require institutional recognition of structured documentation as essential scholarly output and computational infrastructure that serves both human comprehension and machine analysis.
July 28, 2026
Article
Soohong Eum
Comprehensive and reliable funding information is essential for evaluating public research investment, yet funding metadata remain fragmented across heterogeneous data sources maintained by funders and bibliographic platforms. This study examines the interoperability between a national funder database, the National Science and Technology Information Service (NTIS) of South Korea, and two bibliographic sources, Web of Science (WoS) and OpenAlex, in order to assess the quality and coverage of funded publication data. Using a multi-step metadata matching procedure, more than 99% of NTIS records were successfully linked to corresponding documents in WoS and OpenAlex, demonstrating that bibliographic metadata in NTIS records are generally complete and accurate despite incomplete coverage of persistent identifiers such as DOIs. However, limitations are identified in funding-specific metadata, particularly in the centrally assigned contribution rate, which does not account for non-Korean or non-governmental funding and may bias national statistics. Comparison across data sources reveals substantial overlap but also notable differences in coverage and granularity. These differences reflect the distinct collection mechanisms of funders’ databases and bibliographic sources and give rise to complementarities that can be exploited to obtain a more comprehensive picture of funded research outputs and their associated funding sources. The findings carry broader implications for funding agencies seeking to improve interoperability, metadata standardisation, and the analytical utility of research information systems.
July 23, 2026












