Background

Peer review is central to research funding evaluation and in deciding what, and who, is funded. With existing systems under strain and recruitment of reviewers increasingly challenging (1), funders have begun to innovate, trialling alternative methods of evaluation and of allocating funding (2,3). Distributed Peer Review (DPR), where applicants also serve as reviewers for other proposals in the same funding call, is one such example. Building on existing principles of peer review, it addresses some of the challenges of conventional funding evaluation systems (2,4) without departing too far from the familiar.

First conceived as a mechanism for allocating telescope time (5), DPR has since been trialled by several research funders (e.g. 6–8). Benefits include improved efficiency, potential for faster decision times, and democratisation of funding evaluation (6,7,9,10). A further advantage is that, with each proposal receiving more reviews, applicants receive more feedback (6,9,11).

Feedback on a proposal, a secondary function of grant peer review (12), may be an important, albeit also secondary, incentive for applicants, with DPR applicant-reviewers reporting appreciating extensive, diverse feedback from peers (6). Whilst applicants are by no means guaranteed to find grant feedback useful (e.g. 12), relevant and constructive feedback can help strengthen applications and, particularly for ECRs, motivate persistence with a research idea (13).

Review reports are also a rich source of data which have the potential to provide insight into (distributed) peer review, including inter-reviewer variability (6) and concerns around reviewer expertise and experience (9,14). More broadly, analysis of reports might enhance understanding of differential application and emphasis of funding criteria (15–17) and contribute to assessments of review quality (18). Despite this promise, few studies have comprehensively examined grant review reports.

Assessment of DPR reviewer comments specifically has typically focussed on comment length and perceived usefulness. Evaluation of DPR for the allocation of telescope time at the ALMA observatory found almost three-quarters of review comments were considered helpful with lower scoring proposals receiving longer comments (9). Whilst Meyer and colleagues observed no differences in review quality according to reviewer career stage (2022), at the European Southern Observatory (ESO) research found earlier career DPR reviewers had limited ability to provide very helpful comments (11). Additionally, comment helpfulness was observed to differ according to both self-rated and inferred reviewer expertise: experts rarely gave very unhelpful comments; non-experts rarely provided very helpful ones (11). This aligns with our own evaluation of DPR at the Volkswagen Foundation (VWS); whilst most applicants who provided feedback found reviews to be polite and helpful, only around half considered reviewer expertise to be good (6).

Beyond DPR, research on grant peer review reports is still relatively limited; however, examination of the sentiment and content of reports from conventional funding evaluation processes has yielded some insights into reviewing and the influence of reviewer characteristics. Machine learning-based cluster analysis of review comments found content to be consistent with evaluation criteria (19), suggesting reviewers were adhering to review instructions. Grant report sentiment has also been shown to predict both review scores and funding outcomes (19–22). A ‘negativity bias’, where negative information has more weight on a decision than positive information (23,24), has been observed in grant review (19,21,22). Linguistic analysis of European Research Council (ERC) reviews found negative words had the strongest association with reviewer scores (22). Elsewhere, the number of weaknesses, but not strengths, in a report has been shown to predict the score awarded (21) and to have the greatest bearing on whether a proposal is funded (19).

Evidence regarding disciplinary differences in grant review reports is mixed. Whilst Hren and colleagues (2022) found no disciplinary differences in reviews, analysis of review comments at the Swiss National Science Foundation (SNSF) (20) identified differences in length, tone and content of comments. Reviews for social science and humanities (SSH) proposals were longer and offered more criticism whilst maths, informatics, natural science, and technology reviews were more concise and positive. Content also differed: for example, in life sciences the suitability of methods was emphasised, whilst reviews from SSH emphasised relevance and originality (20). It remains unclear as to how reviewer-proposal disciplinary alignment might influence review content and sentiment.

Gender differences in grant funding are well-studied, with consideration given to both reviewer and applicant gender (25,26). Variation in review reports according to reviewer gender have been noted, though research appears extremely limited. Analysis of almost 40,000 reviews found that, across all disciplines, reviews from female reviewers were longer, more positive and more comprehensive (20). However, given the scarcity of evidence, further investigation is required to support any generalisation.

The Volkswagen Foundation, a German non-profit funder, recently trialled DPR in parallel with their conventional panel peer review process (6). In the present study, we examine DPR review comments, using pretrained, fine-tuned transformer-based machine learning models (18) to assess content and sentiment. In doing so, we aimed to confirm the suitability of existing models on reviews from another funding organisation, to examine (DPR) reviewer consistency and alignment with funder priorities, and to investigate the influence of reviewer characteristics on reviews. Finally, we considered how, beyond providing applicants with valuable feedback, insights from review reports can be harnessed to enhance the review process.

Method

Setting

This study uses data from the 2024 and 2025 VWS experiments trialling DPR for the interdisciplinary ‘Open Up’ funding program. A comprehensive evaluation of the 2024 experiment provides a detailed account of the DPR procedure (6) shown in brief in Figure 1. Submissions (140 in 2024, 161 in 2025) underwent both DPR and panel review. As only DPR reviewers provided review comments, only data from the DPR arm of the experiment is used here. All applicants and co-applicants were assigned up to five proposals to review; each proposal received nine or ten reviews. Proposals were ranked according to their ‘trimmed’ mean (highest and lowest score removed) and the top ten recommended for funding. The same general procedure was used in both years. In 2025 there were minor modifications to the applicant-proposal matching algorithm, funding criteria, rating scale, and review survey (see Supplementary Materials for details).

Figure 1. DPR process in the Volkswagen Foundation experiment

Participants

All applicant-reviewers (317 in 2024, 378 in 2025) providing one or more reviews in English were included in the analysis; six reviewers were excluded in 2024 and one in 2025 as they did not provide any reviews in English. Participant characteristics, similar in both years, are shown in Table 1.

Data collection

This analysis uses data collected during the application and review stage of the 2024 and 2025 funding calls. Ethical approval was granted by the School of Psychology at the University of Sheffield (ethics no: 057926) and informed consent obtained on application. Sociodemographic characteristics were provided on submission. For each review, reviewers provided an overall score, a score for each of four funding criteria (Supplementary Tables 1 & 2), and a text comment. Additional measures were included at both timepoints but, as they are not relevant to the present study, these measures, described elsewhere (Butters et al., 2026), are not reported here.

Measures

Criteria scores. Scores awarded to each of four funding criteria (see Supplementary Tables 1 & 2 for full descriptions): ‘Exploratory nature and novelty’, ‘Added value of the constellation’ (in 2025, ‘Added value of the multiperspective approach’), ‘Originality’, ‘Scientific quality’ (2024: 9 pt scale, A+(outstanding) to C-(unsuitable); 2025: 6pt scale, ‘outstanding’ to ‘not eligible’).

Proposal score. Overall score awarded (scales as above).

Review comment. Feedback on the proposal. Reviewers were instructed to summarise the strengths and weaknesses of proposals, being objective, concise, professional, and constructive.

Applicant characteristics. Gender, whether they were a first-generation university student, and number of years working in research.

Review experience. Funding and non-funding related peer review experience. Included in 2025 only.

Disciplinary affiliation (of applicant). One to six subject areas (according to German Research Foundation (DFG) classifications) applicants felt able to review.

Disciplinary affiliation (of proposal). One to six subject areas associated with the proposal.

Variables

Length of comment. Length in characters of the review comment.

Prevalence of content or sentiment. The percentage of statements within each review comment classified as mentioning a content category or being positive or negative (sentiment).

Working years quantile. Years working in research are partitioned into four groups according to the number of participants (2024: 0-8 years, 9-12 years, 13-19 years, 20-52 years; 2025: 2-8 years, 9-12 years, 13-19 years, 20-55 years).

Disciplinary overlap. In 2024, where the primary reported discipline of applicant and proposal under review matched, applicants were considered ‘within discipline’; where they did not, they were ‘outside discipline’.

Disciplinary closeness. In 2025, where applicants and the proposal under review shared at least one DFG subject area they were considered ‘close’, where they shared at least two DFG review boards they were ‘closest’, otherwise they were classed as ‘less close’.

Analysis

Data from 2024 (27) and 2025 were analysed separately to ensure between-year changes in procedures did not confound results. All English language review comments were classified (2024: N=1352, 2025: N=1598) using publicly available classifier models (https://huggingface.co/snsf-data) in Visual Studio Code/Python version 3.14.0. Analysis code is available at https://osf.io/mavz8. Non-English comments were excluded (35 in 2024, nine in 2025). The models were originally developed using a corpus of English language grant reports from the SNSF ‘Project Funding’ scheme (18), which funds research from all disciplines including interdisciplinary projects. Development and validation of these models (detailed in Okasa et al., 2025) included human annotation of 3,000 sentences and testing of six different transformer models/classification approaches. The final, best-performing models, SPECTER2 binary models, classified comments on a sentence by sentence basis (e.g. “Does the sentence address the feasibility of the proposed research project?” present/not present) according to 11 categories (Supplementary Materials); nine related to content (Track record, Relevance, originality & topicality, Applicant, Applicant quantity, Feasibility, Method, Proposal, Suggestion, Suitability) and two to sentiment (Positive, Negative). Some categories were less relevant to data from the VWS DPR experiment. For example, DPR reviewers did not receive applicant CVs so ‘Track record’, ‘Applicant’, and ‘Applicant quantity’ were less applicable and expected to be of low prevalence. Prevalence was calculated at the review level and denotes the proportion of sentences in each review comment classified as mentioning a content category or being of positive or negative sentiment. One author (AB) manually validated the binary classifications of a small amount of data, annotating 220 randomly selected sentences (20 per model); mean agreement with model classification was 85%.

All subsequent analyses were exploratory and conducted in R Studio/R version 4.5.1. Mean and median prevalence of each category (category prevalence) were calculated for each year. Criteria and overall scores were recoded to a numeric scale (2024, 1= C-, 9=A+; 2025, 6= ‘outstanding’, 1= ‘not eligible’). For review experience, two response options, ‘no experience’ and ‘Peer review experience but not reviewing funding applications’, were combined into a ‘no funding peer review experience’ category. Spearman’s correlation was calculated between funding criteria scores and overall scores and category prevalence.

To examine how comments might differ according to reviewer and proposal characteristics, mean prevalence was recalculated according to group for: gender (male/female only), working years quantile, reviewing experience, disciplinary overlap/closeness, and the top and bottom 25% of proposals according to final rankings. Prevalence was visualised using radar plots and density plots (see Supplementary Materials). We agree with the previously described limitations of radar plots, namely that the shape/area of the polygon is subject to the position of variables on the axes and thus has the potential to be misleading (28,29). However, here, where no calculation of area or attempt to create a composite measure is made, we believe they effectively illustrate differences, or lack thereof, in content and sentiment prevalence between groups. To evaluate the sensitivity of the models for our data we visualised prevalence for each review (Supplementary Figure 1). This indicated sufficient variability to be confident that lack of between-group differences were not attributable to model insensitivity. Statistical analysis of the differences between category prevalence for each variable was conducted and effect sizes calculated. For variables with two groups Mann-Whitney U tests were conducted. For variables with three or more groups Kruskal-Wallis tests were used. Where significant differences were observed, post-hoc Bonferroni-adjusted Dunn tests were conducted.

Results

Statements discussing the proposal and the relevance, originality, and topicality (‘relevance…’) of the proposed project were most prevalent, present in, on average, over two-thirds and over a quarter of comments respectively (Table 2). Prevalence of ‘relevance…’ was higher than in other analyses using the same classifier models (18,20) (mean prevalence ~ 18%), suggesting reviewer-applicants were prioritising originality and innovation as per the scope of the funding call.

Content about applicants was limited (Track record, Applicant, Applicant quantity), as expected, given the double-blind nature of the experiment. Further investigation of the four comments mentioning ‘Applicant quantity’ revealed that these were mistakenly categorised as such and instead referred to topic quantity, e.g. publications on a topic. This illustrates the potential for machine error and need for caution in interpretations. Discussion of project feasibility was also minimal, in accordance with instructions to reviewers. Positive and negative sentiment were similarly prevalent, though in 2025, compared to 2024, there were slightly more positive and slightly fewer negative comments. This may be attributable to the introduction of an option to provide an additional, confidential comment to the Volkswagen Foundation. Overall however, the relatively equal prevalence of positive and negative sentiment differed from other analyses using these models, which identified more positive comments and substantially fewer negative ones (20).

Based on the prevalence of content of existing reviews we can also estimate how many reviews would be required to ensure at least one statement mentioning a specific category. For example, having three reviews per proposal would likely provide at least one comment pertaining to ‘relevance….’.

Criteria and overall review scores were moderately positively correlated with prevalence of positive comments and moderately negatively correlated with negative comments (Tables 3 & 4). Higher scores were associated with more positivity in comments whilst lower scores were associated with more negativity, indicating consistency between scoring and written reports.

Of the content categories, ‘relevance…’ was most aligned with any funding criteria. This had the strongest, though notably still fairly weak, association with criteria scores of any content category. Correlation with the most closely aligned funding criteria, ‘Originality’, was strongest. There were no notable differences in association between 2024 and 2025.

There were minor differences in review comments between male and female reviewers (Figure 2). In both years, relative to men, comments by female reviewers had a higher prevalence of statements regarding ‘relevance…’, method, feasibility, and more positive sentences and suggestions (Supplementary Table 3). In contrast, comments by male reviewers contained more negative sentences. In 2024 only, female reviewers discussed the suitability of the method and the applicant more and in 2025 only their comments contained more sentences about the proposal. All effect sizes were small, practically speaking; the largest differences were with respect to comment sentiment.

Figure 2. Prevalence of content and sentiment in comments according to gender* in 2024 (N=1335) and 2025 (N= 1499)

*Due to only a small number of applicants reporting diverse gender or preferring not to say, statistical analysis only included reviews from those reporting female or male gender.

There was some variation according to reviewer career stage (Figure 3), with statistically significant differences in prevalence of statements about the proposal, positive sentences (2024 only), and suggestions (Supplementary Table 4). Again, the magnitudes of these effects were small.

Post-hoc tests revealed ‘early-mid’ career researchers had lower prevalence of proposal related statements than their ‘later-mid’ career colleagues. On average, ‘late’ career researchers mentioned proposals less than ‘later-mid’ and early career researchers. Results from 2025 were partially consistent; ‘early’ career researchers had a higher prevalence of proposal related content than ‘late’ career researchers.

Comments from ‘late’ career researchers were also less positive than ‘early-mid’ career reviewers. Both ‘later-mid’ and ‘late’ career researchers made fewer suggestions for improvement than their ‘early’ and ‘early-mid’ counterparts. ‘Early’ career researchers continued to make more suggestions for improvements than researchers at all other career stages in 2025.

Figure 3. Prevalence of content and sentiment in comments according to years working in research

There were minimal differences according to grant funding review experience (Figure 4). Prevalence of content regarding track records was slightly higher in comments by reviewers with no experience relative to those with funding peer review and panel experience (Supplementary Figure 4, Supplementary Table 5). Reviewers with previous review and panel experience discussed the method slightly more than those who only had funding review experience. Reviewers with no prior experience made more suggestions and gave longer comments than their colleagues with funding peer review experience. All effects were small.

Figure 4. Prevalence of content and sentiment in comments according to reviewing experience

Minimal differences were observed with respect to disciplinary overlap (2024) and disciplinary closeness (2025; Figure 5, Supplementary Table 6). Prevalence of statements regarding track record, very low in both groups, was slightly lower across within-discipline reviewers. In 2025, reviewers who were disciplinarily ‘closest’ to the proposal had a slightly higher prevalence of statements regarding the applicant than ‘close’ reviewers. ‘Less close’ reviewers made more negative statements than ‘closest’ reviewers and included fewer suggestions than ‘close’ reviewers.

Figure 5. Prevalence of content and sentiment according to disciplinary overlap (2024) or disciplinary ‘closeness’ (2025) between DPR reviewer and proposal

Reviews of proposals ultimately ranked in the top 25% of submissions (2024=417; 2025=429) contained more comments about ‘relevance…’, suitability of the method (2024 only), and applicants relative to proposals ranked in the bottom 25% (2024= 278; 2025=378) (Figure 6, Supplementary Table 7). They also contained a higher prevalence of positive statements and a lower prevalence of negative statements, consistent with their final ranking. Again, all effect sizes were small, though were largest for sentiment.

Figure 6. Prevalence of content and sentiment of proposals ranked in the top and bottom 25% in 2024 (N=695) and 2025 (N= 807)

Discussion

Application of Okasa et al’s (18) classifier models to data from the Volkswagen Foundation DPR experiment demonstrates their suitability for proposal data from other funding organisations. Our analysis revealed consistency between content and sentiment of reviewer reports and the scores awarded. Minor differences in review comments were observed according to reviewer gender and career stage. Minimal differences were apparent with respect to prior experience of reviewing funding applications and disciplinary overlap/closeness between reviewer and proposal. Comparison of reviews for the top and bottom 25% of proposals revealed reports were consistent not only with review-level scoring but also with overall funding decisions.

Analysis of review comments provided insight into DPR reviewer behaviour, including within-review consistency and alignment with funder priorities. Review sentiment was relatively consistent with both individual scores and final rankings. In agreement with previous research (19–22), this indicates that, on average, reviewers were consistent within their reviews. Comparison with research employing the same classifier models demonstrated that, in the current study, prevalence of positive sentiment was lower and negative sentiment substantially higher (20). Instructions to discuss both strengths and weaknesses of a proposal may have contributed to this.

Review sentiment was associated with the score awarded. Incongruency between score and sentiment may indicate inauthentic behaviour. ‘Gaming’, strategic scoring to try and improve one’s own standing, is a concern in DPR (10,14). Disparities should be considered, alongside other indicators, in any auditing process. Ostensibly, inclusion of criteria not relevant to the current funding call is a limitation of using pre-trained classifier models; however, it may prove valuable in auditing. (High) prevalence for content not aligned with funding criteria may indicate misalignment between reviewer and funder priorities whereby reviewers value, and so emphasise, alternative criteria more strongly. It may also signal inauthentic or low effort reviewing. Whilst not justification for exclusion of an applicant, this could nonetheless signal the need for further investigation.

We identified similar gender differences in review comments to those reported by Müller and colleagues (2025), with reports by women more positive and more comprehensive. Comparable differences have been observed in journal article peer review reports, with comments from female reviewers found to be more informative (30) and to contain more praise (31), though the cause of this is unclear. Female applicants are suggested to interpret feedback more negatively than men (32,33). Female reviewers may therefore be more conscious of how someone else may respond to the feedback they give and thus endeavour to highlight positive elements of a proposal even when, overall, their evaluation is average or negative. In the current study, where reviewers were also applicants and anticipating feedback on their own reviews, awareness of potential responses may have been even more salient.

Reviewers with less experience or relevant disciplinary expertise provided reviews with similar content to those with greater experience and expertise. Disparities according to research experience were minor, suggesting that more and less experienced researchers generally attended to similar factors when reviewing proposals. This is consistent with prior research which found perceived DPR review quality was consistent across career stages (9,34). Earlier career researchers focussed slightly more on the proposal and on providing suggestions for improvement than those at a more advanced career stage. Differences according to funding evaluation experience were also, perhaps surprisingly, very small, suggesting that more and less experienced reviewers also tended to focus on similar aspects of proposed research.

There were negligible differences in the content of comments by reviewers closely aligned to the discipline of a proposal and by reviewers with greater disciplinary distance. This suggests that in the context of an interdisciplinary, innovation-focussed funding call, closely aligned expertise is not necessarily a prerequisite of a quality review. In the case of DPR, where the experience and expertise of the reviewer pool is determined by who applies to the call (10), this is reassuring. These novel findings further suggest that widening reviewer pools in other funding evaluation mechanisms could help alleviate difficulties in reviewer recruitment without unduly diminishing review quality (or our confidence in it).

Evaluation of review comments yields insights which can be used to enhance and refine (distributed) peer review processes. Results can inform decisions about the number of reviews to obtain for each proposal, a key consideration in any planned implementation of DPR (Stafford et al., 2025). To demonstrate, we calculated the number of reviews required to be likely to get at least one comment about an individual category; a funder might equally opt to estimate the number of reviews required to get, for example, at least one comment about several different criteria. Considering this in conjunction with evidence regarding decision stability (e.g. 6) and feedback on the acceptability of the workload (e.g. 7) would be a principled way of determining the number of reviews per proposal to obtain.

Funders might also use results to ascertain whether instructions to reviewers require clarification. Very low average prevalence for a content category which, given the scope and criteria of a funding call, might reasonably be anticipated to be higher, could indicate that instructions are being misunderstood or misinterpreted. Analysis of grant review reports therefore has the potential to provide insights which are both theoretically and practically pertinent.

This study has several limitations. Content categories are based on SNSF evaluative criteria and as such do not directly map to the Volkswagen Foundation’s ‘Open Up’ criteria. Some criteria, e.g. opening of a new research space, are therefore not represented. Additionally, the models only identify whether a particular topic is mentioned, not whether it is discussed in a positive or negative context. Nor is it possible to tell whether the particular category is discussed accurately. Results describe associations only and do not allow for causal inference.

Overall, this study demonstrates the applicability and sensitivity of existing classifier models to reviews from a different funding call at a different funding organisation using a different funding evaluation mechanism. Exploiting the increased number of reviews per proposal in DPR, our analysis of grant review reports builds on earlier work to provide novel insights into peer review. Our findings speak to concerns about reviewer expertise and experience in DPR and suggest there is no evidence of disadvantage from receiving reviews from earlier career or more disciplinarily distant reviewers. Finally, we demonstrate how this rich source of data can provide insights which can be harnessed to enhance and improve (distributed) peer review processes.

Data availability

Review comments cannot be shared to maintain applicant-reviewer anonymity. Anonymised, delinked quantitative data (criteria/overall scores, etc) for 2024 are available to support audit and reanalysis but not full reproducibility at https://doi.org/10.17605/OSF.IO/YF697. Anonymised data from 2025 will be made available by the Volkswagen Foundation following completion of internal processes. Analysis code is available at https://osf.io/mavz8/ along with surveys/measures.

Acknowledgments

We extend our sincere thanks to the Volkswagen Foundation ‘Open Up’ DPR experiment team: Alexander Bondarenko, Barbara Neubauer, Robert Nuske, Pierre Schwidlinski and Hanna Denecke.

Generative AI use

Google Gemini was used to troubleshoot code during analysis. Generative AI was not otherwise used in the preparation of this manuscript.

References

1. Meadmore K, Fackrell K, Recio-Saucedo A, Bull A, Fraser SDS, Blatch-Jones A. Decision-making approaches used by UK and international health funding organisations for allocating research funds: A survey of current practice. Visram S, editor. PLOS ONE. 2020 Nov 5;15(11):e0239757. doi:10.1371/journal.pone.0239757
2. Bendiscioli S, Garfinkel M. Dealing with the limits of peer review with innovative approaches to allocating research funding [Internet]. EMBO Science Policy Programme; 2021. Available from: embo.org/documents/science_policy/peer_review_report.pdf
3. Kolarz P, Vingre A, Vinnik A, Neto A, Vergara C, Rodriguez CO, et al. Review of peer review final report [Internet]. Technopolis; 2023. Available from: https://technopolis-group.com/report/review-of-peer-review/
4. Bendiscioli S. The troubles with peer review for allocating research funding. EMBO Rep. 2019 Dec 5;20(12):e49472. doi:10.15252/embr.201949472
5. Merrifield MR, Saari DG. Telescope time without tears: a distributed approach to peer review. Astron Geophys. 2009;50(4):4.16-4.20. doi:10.1111/j.1468-4004.2009.50416.x
6. Butters A, Marshall MB, Pinfield S, Stafford T, Bondarenko A, Neubauer B, et al. Applicants as reviewers: A mixed methods evaluation of the risks, benefits, and potential of distributed peer review for grant funding allocations [Internet]. MetaArXiv; 2026 [cited 2026 Jan 7]. Available from: https://osf.io/t2p56_v2 doi:10.31222/osf.io/t2p56_v2
7. DSIT. A year in metascience [Internet]. UK Research and Innovation and the Department for Science, Innovation and Technology; 2025 [cited 2025 Jul 14]. Available from: https://www.gov.uk/government/publications/a-year-in-metascience-2025
8. NWO. Open competition domain science – XS: Round 2024 [Internet]. 2024. Available from: https://www.nwo.nl/sites/nwo/files/media-files/Call%20ENW-XS%202024_EN_1.pdf
9. Meyer JD, Corvillón A, Carpenter JM, Plunkett AL, Kurowski R, Chalevin A, et al. Analysis of the ALMA Cycle 8 Distributed Peer Review Process. Bull AAS. 2022;54(1). doi:10.3847/25c2cfeb.4ece85d4
10. Stafford T, Pinfield S, Butters A, Benson Marshall M. Applicants as Reviewers – a Guide to Distributed Peer Review [Internet]. Research on Research Institute; 2025 [cited 2025 Jul 14]. Available from: https://doi.org/10.6084/m9.figshare.29994841.v2
11. Kerzendorf WE, Patat F, Bordelon D, van de Ven G, Pritchard TA. Distributed peer review enhanced with natural language processing and machine learning. Nat Astron. 2020 Jul 1;4(7):711–7. doi:10.1038/s41550-020-1038-y
12. Gallo SA. The science of peer review: Grant review feedback. BioScience. 2021 May 1;71(5):431–431. doi:10.1093/biosci/biab051
13. Derrick GE, Zimmermann A, Greaves H, Best J, Klavans R. Targeted, actionable and fair: Reviewer reports as feedback and its effect on ECR career choices. Res Eval. 2023 Oct 1;32(4):648–57. doi:10.1093/reseval/rvad034
14. Shah NB. Challenges, experiments, and computational solutions in peer review. Commun ACM. 2022 May;65(6):76–87. doi:10.1145/3528086
15. Abdoul H, Perrey C, Amiel P, Tubach F, Gottot S, Durand-Zaleski I, et al. Peer review of grant applications: Criteria used and qualitative study of reviewer practices. Gagnier JJ, editor. PLoS ONE. 2012 Sep 28;7(9):e46054. doi:10.1371/journal.pone.0046054
16. Hug SE, Ochsner M. Do peers share the same criteria for assessing grant applications? Res Eval. 2022 Jan 1;31(1):104–17. doi:10.1093/reseval/rvab034
17. Lee CJ. Commensuration bias in peer review. Philos Sci. 2015;82(5):1272–83. doi:10.1086/683652
18. Okasa G, de León A, Strinzel M, Jorstad A, Milzow K, Egger M, et al. A supervised machine learning approach for assessing grant peer review reports. Quant Sci Stud. 2025 Nov 8;6:1189–214. doi:10.1162/QSS.a.23
19. Hren D, Pina DG, Norman CR, Marušić A. What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports. J Informetr. 2022 May 1;16(2):101289. doi:10.1016/j.joi.2022.101289
20. Müller S, Okasa G, Strinzel M, Jorstad A, Milzow K, Egger M. Gender and Discipline Shape Length, Content and Tone of Grant Peer Review Reports [Internet]. arXiv; 2025 [cited 2026 Jan 14]. Available from: http://arxiv.org/abs/2507.00103 doi:10.48550/arXiv.2507.00103
21. Pier EL, Brauer M, Filut A, Kaatz A, Raclaw J, Nathan MJ, et al. Low agreement among reviewers evaluating the same NIH grant applications. Proc Natl Acad Sci. 2018;115(12):2952–7. doi:doi:10.1073/pnas.1714379115
22. van den Besselaar P, Sandström U, Schiffbaenker H. Studying grant decision-making: a linguistic analysis of review reports. Scientometrics. 2018 Oct 1;117(1):313–29. doi:10.1007/s11192-018-2848-x
23. Kanouse DE, Hanson L. Negativity in evaluations. In: Attribution: Perceiving the causes of behavior. Morristown, NJ: General Learning Press; 1972. p. 47–62.
24. Rozin P, Royzman EB. Negativity bias, negativity dominance, and contagion. Personal Soc Psychol Rev. 2001 Nov;5(4):296–320. doi:10.1207/S15327957PSPR0504_2
25. Mutz R, Bornmann L, Daniel HD. Does gender matter in grant peer review?: An empirical investigation using the example of the Austrian Science Fund. Z Für Psychol. 2012 Jan;220(2):121–9. doi:10.1027/2151-2604/a000103
26. Severin A, Martins J, Heyard R, Delavy F, Jorstad A, Egger M. Gender and other potential biases in peer review: cross-sectional analysis of 38 250 external peer review reports. BMJ Open. 2020 Aug 1;10(8):e035058. doi:10.1136/bmjopen-2019-035058
27. Nuske R, Denecke H, Schwidlinski P, Bondarenko A, Niemann B. Distributed Peer Review in Research Funding: A Dataset from the Volkswagen Foundation Call ‘Open Up’ (2024) [Internet]. OSF; 2026 [cited 2026 Jul 6]. Available from: https://osf.io/yf697/ doi:10.17605/OSF.IO/YF697
28. Feldman R. Filled radar charts should not be used to compare social indicators. Soc Indic Res. 2013 May 1;111(3):709–12. doi:10.1007/s11205-012-0028-6
29. Heijungs R. Two arguments against the use of radar plots for constructing composite indicators. Braz J Chem Eng. 2022 Sep 1;39(3):885–6. doi:10.1007/s43153-022-00247-1
30. Álvarez-García E, García-Costa D, Squazzoni F, Malički M, Mehmani B, Grimaldo F. Published peer review reports have higher informative content than unpublished reports. J Informetr. 2026 Mar 1;20(1):101760. doi:10.1016/j.joi.2025.101760
31. Zhang G, Wang L, Wang X. Cultural distance, gender and praise in peer review. Account Res. 2025 Nov 17;32(8):1448–73. doi:10.1080/08989621.2024.2409310
32. Biernat M, Carnes M, Filut A, Kaatz A. Gender, race, and grant reviews: Translating and responding to research feedback. Pers Soc Psychol Bull. 2020 Jan 1;46(1):140–54. doi:10.1177/0146167219845921
33. DeCastro R, Sambuco D, Ubel PA, Stewart A, Jagsi R. Batting 300 is good: Perspectives of faculty researchers and their mentors on rejection, resilience, and persistence in academic medical careers. Acad Med. 2013 Apr 1;88(4):497–504. doi:10.1097/ACM.0b013e318285f3c0
34. Carpenter J, Meyer JD, Corvillón A, Impellizzeri V, Kurowski R, Chalevin A. Analysis of the ALMA Cycle 7 Supplemental Call [ALMA Memo] [Internet]. 2020. ALMA Memo no.: 616. Available from: https://library.nrao.edu/public/memos/alma/main/memo616.pdf 

Supplementary Materials

Supplementary methods

Rating scales

2024

A+ outstanding: breakthrough science

A excellent: definitely above average

A- very good: no significant weaknesses

B+ good: minor deficiencies do not detract from strong scientific case

B fair: good scientific case, but with definite weaknesses

B- rather weak: limited science return prospects

C+ weak: little scientific value and/or questionable scientific strategy

C very weak: deficiencies outweigh strengths

C- unsuitable

2025

Outstanding: breakthrough science

Excellent: definitely above average

Very good: no significant weaknesses

Good: minor deficiencies do not detract from strong scientific case

Average: good scientific case, but with definite weaknesses

Not eligible for funding: deficiencies outweigh strengths

Classification categories (taken from Okasa et al., 2025)

Track Record: Does the sentence address the scientific qualifications of the applicant(s) /team?

Relevance, originality, topicality: Does the sentence address the scientific relevance/ impact/ originality of the proposed research project?

Suitability: Does the sentence address the suitability of the methods to be used within the proposed research project?

Feasibility: Does the sentence address the feasibility of the proposed research project?

Applicant: Does the sentence address the applicant(s)/team or their qualifications, without mentioning quantitative indicators?

Applicant quantity: Does the sentence use quantitative indicators to describe the applicant(s) or team?

Proposal: Does the sentence address the proposal or specific parts of it, as opposed to the applicant(s) or context beyond the proposal (such as the research field or the funding scheme’s objectives, etc.)?

Method: Does the sentence address the methods to be used in the proposed research project?

Positive: Is the sentence itself a positive statement or does it contain a positive statement?

Negative: Is the sentence itself a negative statement or does it contain a negative statement?

Suggestion: Does the sentence suggest how to improve the proposal?

Supplementary Results

Supplementary Figure 1. Review-level prevalence for all categories in 2024 (N=1352) and 2025 (N=1598) to assess the model sensitivity to the data

Supplementary Figure 2. Density plots for all categories according to gender in 2024 (N=1335) and 2025 (N=1499)

Applicant quantity excluded due to small number of reviews (0 in 2024, 4 in 2025)

To clearly illustrate differences, or lack thereof, between groups, scales have been allowed to vary

Applicant quantity not included due to limited number of observations (0 in 2024, 4 in 2025)

Supplementary Figure 3. Density plots for all categories according to years working in research

Applicant quantity excluded due to small number of reviews (0 in 2024, 4 in 2025)

To clearly illustrate differences, or lack thereof, between groups, scales have been allowed to vary

Applicant quantity not included due to limited number of observations (0 in 2024, 4 in 2025)      *Bonferroni adjusted p-values

Supplementary Figure 4. Density plots for all categories according to funding peer review experience (2025 only)

Applicant quantity excluded due to small number of reviews (4 in 2025)

To clearly illustrate differences, or lack thereof, between groups, scales have been allowed to vary

*Bonferroni adjusted p-values

Supplementary Figure 5. Density plots for all categories according to disciplinary overlap (2024) or disciplinary closeness (2025)

Applicant quantity excluded due to small number of reviews (0 in 2024, 4 in 2025)

To clearly illustrate differences, or lack thereof, between groups, scales have been allowed to vary

Applicant quantity not included due to limited number of observations (0 in 2024, 4 in 2025)

*Bonferroni adjusted p-values

Supplementary Figure 6. Density plots for all categories for top 25% and bottom 25% of proposals in 2024 and 2025

Applicant quantity excluded due to small number of reviews (0 in 2024, 4 in 2025)

To clearly illustrate differences, or lack thereof, between groups, scales have been allowed to vary