Published at MetaROR
September 4, 2026
Table of contents
Beyond the binary: The system of all-round evaluation of research and its practices in China
Yu Zhu1
,, Jiyuan Ye1
1 School of Information Management, Nanjing University, China
Originally published on September 4, 2026 at:
This article was presented at the STI-ENID 2025 conference.
1. Introduction
With the rapid growth of China’s scientific research output and its increasing share in global research (Tollefson, 2018), the reform of the research evaluation system has become a focal point for both Chinese research management departments and the academic community. As early as 2010, scholars had already identified persistent issues within the academic evaluation system, including excessive quantification, overemphasis on formalities, excessive administrative intervention, weakened evaluation of innovation, favoritism among review experts, and compromised evaluation results (Ye, 2010b). These issues not only contribute to academic misconduct and corruption but also, to some extent, hinder the long-term sustainable and healthy development of academic research.
Since 2020, China have explicitly advocated for moving away from “SCI Supremacy” and shifting the evaluation system from focusing on quantitative and formal indicators to emphasizing academic content and practical contributions, aiming to return academic evaluation to its essence of quality assessment (Ye, 2020b). This reform initiative has also sparked widespread global discussion (Editorial in Nature, 2020; Mallapaty, 2020), highlighting the international academic community’s attention to China’s reforms in its research evaluation system. Meanwhile, despite the launch of international declarations such as the San Francisco Declaration on Research Assessment (DORA, 2012) and the Leiden Manifesto (Hicks et al., 2015), which advocate for reform in academic evaluation, the deep-rooted theoretical issues within the current academic evaluation system remain unresolved. Therefore, constructing a rational evaluation system to clarify ambiguities in the field, define the direction and pathways for evaluation practice, and provide a robust methodological foundation for research evaluation remains a pressing issue for both the global academic community and research management institutions.
In fact, in 2010, the Chinese academic evaluation community has proposed a systematic and forward-looking academic evaluation theory — the System of All-round Evaluation of Research (SAER) (Ye, 2010b, 2021) — to address issues such as weak evaluation theory, unclear evaluation subjects, ambiguous evaluation purposes, vague evaluation standards and indicators, singular evaluation methods, incomplete evaluation systems, and low credibility of evaluation results. As a representative of China’s academic evaluation theory (Information Center for Social Science Renmin University of China, 2025), SAER strategically offers a holistic solution to address evaluation-related issues, designs a phased and systematic approach for implementation, and fosters a constructive interplay between evaluation theory and practice. This theory framework plays a crucial role in tackling issues such as the overemphasis on quantitative metrics, while also fostering academic integrity, creating an inclusive and supportive research environment, and promoting scientific advancement. Given the above context, this paper focuses on the epistemological aspects of research evaluation theory and introduces relevant practical applications.
2. The Binary of Evaluation System
The formation of academic evaluation systems is largely dependent on evaluation methods. Peer review and bibliometric analysis are the two most widely adopted approaches, yet both suffer from inherent limitations, contributing to the current binary landscape of research assessment.
Peer review, as the earliest academic evaluation mechanism, involves independent assessments of scholarly work by experts within the scientific community based on established criteria (Zhang & Sivertsen, 2020). Its institutionalization dates back to the 17th century, marked by the establishment of the Royal Society of London in 1662 and the Académie Royale des Sciences in Paris in 1699, along with their respective journals, Philosophical Transactions and Journal des Sçavans, which signified a transition from private scholarly correspondence to standardized scientific publishing (Lee et al., 2013). With the evolution of scholarly communication, peer review has diversified into newer forms such as open review and preprints. However, its fundamental reliance on subjective expert judgment renders it vulnerable to biases stemming from individual perspectives and social influences (Marsh et al., 2008) and the absence of robust feedback mechanisms further undermines the reliability of the process (Derrick, 2018). Additionally, the substantial financial and time costs associated with peer review make high-quality evaluations a scarce resource (Technopolis Group, n.d.), struggling to meet the demands of large-scale and rapid scholarly assessment (Abramo & D’Angelo, 2011). Despite these challenges, best practices in peer review have been exemplified by initiatives such as the UK’s Research Excellence Framework (REF; Higher Education Funding Council for England, n.d.).
On the other hand, bibliometric analysis, a quantitative approach based on publication and citation frequencies, has gained widespread application in research evaluation. This method dates back to 1955 when Eugene Garfield proposed using citation frequencies and impact factors to assess academic journals (Garfield, 1955). While citation data can partially reflect the outcomes of peer review (Ye, 2020b), the indiscriminate treatment of citation motivations in bibliometric analysis (Brooks, 1986) may lead to criticisms regarding citation inflation and manipulation (MacRoberts & MacRoberts, 1989). In response to these concerns, the San Francisco Declaration on Research Assessment (DORA, 2012) and the Leiden Manifesto (Hicks et al., 2015) both emphasize that research evaluation should be centred on the scientific content of papers rather than journal-based supplementary bibliometric indicators.
Given the strengths and limitations of both mainstream evaluation methods, recent practices have increasingly sought to integrate bibliometric indicators with peer review for a more balanced approach. Examples include the Strategy Evaluation Protocol in the Netherlands (Association of Universities in the Netherlands, 2020) and Italy’s Research Evaluation Exercise (Franceschet & Costantini, 2011), both of which embody the principle of methodological complementarity. This shift signals a growing recognition of the need to move beyond form-driven assessment toward content- and utility-oriented evaluation.
Nevertheless, research on academic evaluation still lacks a systematic theoretical framework, and the persistent dichotomy between qualitative and quantitative approaches remains a dominant paradigm. The SAER offers a theoretical breakthrough by introducing a triadic model that integrates formal evaluation, content evaluation, and utility evaluation. This framework not only provides a holistic perspective for global research evaluation practitioners and scholars but also helps to transcend the binary opposition that has long characterized academic assessment. By offering a more comprehensive theoretical foundation, SAER contributes to a deeper understanding of the historical development, current challenges, and future directions of research evaluation.
3. Theoretical Explanation of SAER
The following subsections present assumptions of premises, theoretical foundations, analytical framework, and originality and innovation of SAER.
3.1 Premise Assumptions and Theories
(1) Assumption of premises
- Universality assumption: In all aspects of human activity, individuals continuously make value judgments about various matters related to themselves in order to make choices and take action.
- Complexity assumption: Like any other subject, academic research—including its outputs, scholars, and institutions—can be evaluated. However, because academic research concerns the exploration of laws governing phenomena, as well as questions of truth, goodness, and beauty, its evaluation is inherently more complex than that of other domains.
- Guidance assumption: Compared to individual assessments, any organized, fair, and reasonable academic evaluation provides more explicit guidance to the evaluated subject, including diagnostic, predictive, selective, and motivational functions.
- Sufficiency assumption: The more comprehensive and profound the understanding of the evaluation object, the more reasonable and reliable the evaluation.
(2) Theoretical Foundations
(a) Value Theory, Epistemology, and Praxis—With a Focus on Marxist Value Theory
Marxist value theory views value as a social-historical relationship, where an object’s value is determined by its ability to meet human needs (Marx & Engels, 1975). It stresses practice as the test for truth and integrate human intent and dialectical objectivity. Therefore, evaluation can reveal objective value and construct a value system, where value judgments go beyond empirical facts to consider the nature of the object, its interaction with the subject, and human needs and ideals. So, unlike truth, value judgments balance plurality and singularity, with “ought” shaped by both objective facts and human aspirations. These perspective lays the foundation for the objectivity standard in SAER.
(b) Systems theory, cybernetics, and information theory—with an emphasis on complete information theory
From a systems theory perspective, objective phenomena can be understood as holistic systems composed of interrelated and interacting elements (Bertalanffy, 1969). Open systems adjust their internal components to achieve new equilibria in response to external environmental changes. As a complex evaluation system, SAER draws directly from systems theory, employing a hierarchical and structured analytical framework to optimize evaluation processes under the principle of system efficiency. Cybernetics conceptualizes system control as an information process (Wiener, 1965), focusing on feedback mechanisms to regulate and control uncertainty, ensuring alignment with predetermined objectives (Shannon, 1948).
SAER’s information-theoretic approach is built upon the complete information theory proposed by Zhong (1991). This theory categorizes information into three levels: syntactic information, semantic information, and pragmatic information, corresponding to the form, content, and utility of information, respectively. Under this framework, academic evaluation subjects—such as scholarly outputs, academic institutions, or individual scholars—can be conceptualized as “information entities”. This allows for the construction of an evaluation index system or a general evaluation framework based on three dimensions: form, content, and utility.
3.2 Analytical Framework
SAER (Ye, 2021) is a conceptual system, a set of application principles, and a framework of rules for academic evaluation as illustrated in Figure 1. The term “All-round” in SAER carries twofold meaning: First, it encompasses all six key elements involved in academic evaluation; second, it considers the entirety of the evaluation object across three dimensions. Specifically, the six key elements of SAER include the evaluation subject, evaluation object, evaluation purpose, evaluation methods, evaluation standards and indicators, and evaluation mechanism. These elements interact to form an integrated evaluation framework. Meanwhile, the three dimensions of evaluation refer to the form, content, and utility of the evaluation object. In essence, SAER is an evaluation system structured around six key elements, three composite evaluation dimensions, and their respective definitions, inferences, explanations, and application principles.
Figure 1. Analytical Framework of SAER
Among the six elements in Figure 1, the evaluation purpose serves as the guiding principle, shaping the selection of the evaluation subject, the classification of evaluation objects, the choice of evaluation methods, the determination of evaluation standards and indicators, and the design of the evaluation mechanism. In SAER, peer experts play a leading role as the evaluation subject, the evaluation object serves as the foundation, evaluation standards and indicators form the core, evaluation methods provide the means, and the evaluation mechanism ensures institutional support.
From this dialectically unified framework, six main theoretical inferences emerge:
- Evaluation Purpose Constraint Theory,
- Peer Expert Dominant and Meta-Evaluation Theory,
- Evaluation Object Classification Theory,
- Bibliometric Auxiliary Verification Theory,
- Evaluation Standards and Indicators Core and Appropriateness Theory, and
- Evaluation Mechanism Guarantee
3.3 Originality and Innovation
SAER demonstrates originality and innovation in the following areas:
(1) Conceptual Innovations
- Introducing the concept of “All-round Evaluation”. The term “All-round” extends beyond the six evaluation elements and three evaluation dimensions in Figure 1, embracing a comprehensive, holistic approach that encompasses the entire process and full life cycle.
- Defining the conceptual integration of formal evaluation, content evaluation, and utility evaluation, incorporating Marxist epistemology, value theory, and praxeology into evaluation theory.
- Differentiating between broad and narrow definitions of academic evaluation, as well as between institutional evaluation and individual spontaneous
- Clarifying the dual function of citation and reprint-based evaluation behaviors, which include both qualitative and quantitative factors, and distinguishing these from references.
(2) Perspective Innovations
- Inferring the Evaluation Purpose Constraint Theory, which identifies evaluation purpose as the most critical factor among the six elements. A lack of clear evaluation purpose or misalignment with other elements is the primary cause of the failure of evaluation. The evaluation purpose dictates the selection of subject, object, method, standard, and mechanism. Under specific evaluation purpose, cross-category comparisons—such as between Shakespeare and Picasso—become feasible. Furthermore, it enables the coexistence of common and individualized indicators, as well as the integration of quantitative and qualitative metrics.
- Proposing a three-dimensional evaluation framework wherein any academic entity can be assessed through formal, content, and utility This framework progresses from superficial to in-depth analysis, increasing in both complexity and explanatory power compared to traditional binary (quantitative vs. qualitative) evaluations.
- Asserting that rigorous, scientific, and authoritative academic evaluation must be grounded in factual However, factual judgment alone is insufficient for value judgment.
- Recognizing that citation-based metrics (e.g., impact factor, h-index) possess both qualitative (cited by peers) and quantitative dimensions (Ye, 2005). These should be utilized appropriately rather than treated as absolute indicators (Ye, 2010a). While bibliometric methods cannot replace peer review, they provide useful references, validations, and benchmarks, particularly for macro-level comparisons or assessing research productivity.
- Distinguishing between academic quality evaluation and academic impact evaluation—with the former mainly focusing on content and utility assessment, using events and cases as primary data sources.
- Advocating for the Peer Expert Dominant Theory in quality and specialized evaluations, emphasizing the importance of meta-evaluation (credibility evaluation of reviewers) and appeal mechanisms to address the inherent limitations and subjectivity of peer
- Arguing that a well-structured evaluation system promotes academic integrity and research ethics.
- Positioning academic evaluation as a critical component of Library and Information Science (LIS), encompassing core journal selection, citation analysis, and web-based resource As an interdisciplinary field, academic evaluation represents a new growth area for LIS research.
(3) Theoretical Innovation
SAER pioneers an all-round analytical framework that integrates all evaluation elements, perspectives, and processes, making it applicable to diverse academic evaluations. SAER extends the traditional binary (qualitative vs. quantitative) evaluation model to a three-dimensional system of formal, content, and utility evaluation framework. This framework not only synthesizes commonalities across output, researcher, and institutional evaluations but also allows for discipline-specific adaptations. It underscores the distinctive features of humanities and social sciences while maintaining connections with evaluation frameworks in natural sciences.
(4) Methodological Innovation
SAER integrates bibliometric analysis with expert surveys, case studies, inductive and deductive reasoning, and both analytical and abstract methodologies. Treating all evaluation objects as “information entities”,” it applies information methods to construct an academic evaluation system grounded in syntax, semantics, and pragmatics—thus forming a “Complete Information Evaluation Method”.
4. Implementation and Application of SAER
4.1 Core Requirements for Implementing
SAER is applicable across various domains, including the evaluation of academic talent, academic outputs, academic institutions, academic media, and disciplines. The usage requirements are as follows:
(1) The Evaluation Purpose Should Be Singular
SAER places significant emphasis on the primacy of the evaluation purpose. It advocates that each evaluation should focus on a single, well-defined purpose based on practical conditions to ensure the accuracy and scientific validity of the conclusions. Empirical evidence has shown that when the evaluation purpose differs, the same evaluator may reach varying conclusions about the same object. Conversely, the same evaluation result may hold different meanings for different subjects being evaluated.
(2) The Evaluation Process Should Be Diachronic
SAER recognizes that both the evaluation subject and object are constrained by time and space. The evaluation of the object’s utility by the evaluator is not only based on historical facts but must also account for long-term practices and future development trends. Thus, evaluation is a diachronic process that combines both the present (ongoing) and the future (incomplete), requiring continuous validation through practical testing.
(3) Evaluation Standards Should Be Targeted
In the SAER, evaluation standards are the value criteria applied to the object during the evaluation process. These standards reflect the evaluator’s value orientation and serve to guide the subject being evaluated. For different objects, the evaluation system should be correspondingly optimized and adjusted to suit their specific characteristics.
(4) Evaluation Methods Should Be Combined
SAER transcends the binary opposition of evaluation methods. It asserts that evaluation subjects and objects should be selected based on the specific evaluation purpose. Evaluation dimensions should be flexibly combined according to the research difficulty to avoid overly formalized, quantitative, or one-sided evaluation processes.
4.2 Exemplary Practices of SAER in China
Over the past decade, SAER has gained significant attention in China’s academic and management sectors. It was featured as a topic of discussion at a national evaluation conference (School of Information Resource Management RUC, 2021) and has seen widespread application. Researchers have applied SAER in areas such as scholar, academic journal, specialized database, professional website, academic library collection, research services, librarian, and creative competitions evaluations (Zhao et al., 2020), with positive feedback in practice.
(1) Book Evaluation
In 2013, Professor Ye’s team applied SAER to evaluate humanities and social science academic books, establishing an All-round evaluation model for Chinese academic books (Wang & Ye, 2014). Through bibliometric methods and expert reviews, they identified high-quality books as representative works and used them to develop the Chinese Book Citation Index (CBkCI) demonstration database (Ye, 2014). This database improved library collection quality, promoted standardized book publishing, and helped researchers quickly identify quality works, laying a foundation for fair and transparent academic evaluations (Ye, 2013b). Ye later refined this model with a more comprehensive framework for evaluating academic book quality (Ye, 2020a).
(2) Journal Evaluation
Before the formal introduction of SAER, Professor Ye’s team have used its principles to identify 2,770 academic social science journals from 4,500 Chinese journals (Ye et al., 2008). They later evaluated the performance of the Ministry of Education’s “Outstanding Academic Journals Program” and laid the groundwork for funding 200 academic journals representing national-level standards (Ye & Zhu, 2024). In 2012, at the request of the Library Society of China, Ye applied SAER as an independent third-party evaluator for selecting outstanding journals in library science (Ye, 2013a). The journals selected received widespread recognition in both academic and industry sectors. SAER also contributed to the establishment of a quality evaluation system for Chinese academic journals (Ye & Yuan, 2021), and it has been used to evaluate non-core Chinese philosophy and social science journals (Zang & Ye, 2018) and the publishing quality of humanities and social science journals in universities (M. Han, 2021).
(3) Library Evaluation
SAER has been widely applied in library evaluations. Most notably, Professor Ye’s team has created a logical and adaptable quality evaluation framework for academic libraries in China, referencing ISO standards and global evaluation systems for academic libraries (Ye et al., 2021). The team will implement a comprehensive quality assessment of Chinese academic libraries this year. Additionally, based on SAER, Zhejiang University Library, assessed their collection of foreign-language humanities and social science books (Z. Han et al., 2021), and Tianjin Foreign Studies University Library evaluated its subject librarians (Fan, 2012). Besides, Nanjing Engineering University Library developed a framework for evaluating university library print collections (Li, 2020), and the National Library of China created an evaluation system for library exhibition services (Ma, 2020). Nanjing Library also used this for library research and learning activities (Qin, 2024).
(4) Website Evaluation
Professor Ye’s team used SAER to evaluate Chinese humanities and social science websites, helping researchers access high-quality academic information. Additionally, Lu et al. (2018) evaluated Chinese library websites, and Zhang et al. (2013) assessed websites dedicated to Chinese calligraphy and painting, all contributing to the improvement of website quality and the provision of better information services.
The representative cases demonstrate that the SAER exhibits robust applicability across diverse domains and has garnered substantial practical recognition. Guided by SAER, it becomes feasible to construct research on multiple evaluation objects that demonstrates strong logical coherence, explanatory power, and predictive capability.
5. Conclusion
SAER is a systematic framework developed through a comprehensive examination of global evaluation practices in recent years, and has been recognized as one of China’s top ten original theories in philosophy and social sciences(Information Center for Social Science Renmin University of China, 2025). It is characterized by the following distinctive features: (a) Drawing on research findings from natural science evaluation while emphasizing the unique attributes of humanities and social sciences and their evaluation principles (X. Yang, 2023); (b) balancing the universality of academic evaluation with the diversity and specificity across disciplines, fields, and research outputs (S. Yang & Jiang, 2023), reconciling the characteristics of different evaluation domains; (c) providing a relatively stable analytical framework while allowing for dynamic development (Jiang, 2023); and (d) offering a structured analysis of the historical trajectory of academic evaluation, a clear explanation of its current status and challenges, and a general forecast of its future directions (Geng & Zhao, 2023), bridging the gap between history and the future.
Over the past fifteen years, the SAER has pioneered a comprehensive path in addressing the global challenge of bridging the gap between evaluation theory and practice. In fact, when the 2012 DORA and the Leiden Manifesto published after STI 2014 called for “indicators must not substitute for informed judgement”, SAER had already proactively established a systematic evaluation theory that interlinks aspects such as evaluation purpose. This system, built on six elements and three dimensions, resonates theoretically with principles like the Leiden Manifesto’s “Measure performance against the research missions…” (Hicks et al., 2015). It also showcases systematic and dialectical Chinese experience through theoretical inferences like the Peer Expert Dominant and Meta-Evaluation Theory and the Evaluation Mechanism Guarantee Theory.
Looking to the future, just as scientific research strives to uncover the truth, the construction and refinement of academic evaluation must evolve with the times. Key challenges and opportunities will emerge in addressing the relationship between Chinese characteristics and internationalization, and how to cope with new technologies like generative AI. We wish that SAER will continuously improve through its widespread application in the international academic community, driving the progress of global academic evaluation systems, promoting the high-quality output of scientific research, and ultimately contributing more to academic prosperity, knowledge accumulation, and social progress.
Open science practices
In this study, the case data primarily originate from publicly accessible academic databases and published scholarly works, including CNKI (China National Knowledge Infrastructure), CSSCI (Chinese Social Sciences Citation Index), other institutional repositories like Renmin University, and academic monographs authored by Professor Ye (see references for citations). These datasets are published and can be accessed upon purchase. Regarding software, the free version of Xmind was utilized to visually construct the theoretical framework. The use of this software falls within its commercial licensing terms. Additionally, no formal preregistration of the research plan was conducted; however, the theoretical framework and methodology were iteratively designed and thoroughly documented throughout the research process.
Acknowledgments
The authors would like to express their gratitude to Dr. Robin Haunschild for his interest in this subject matter and for his gracious invitation, which has significantly contributed to this research.
Author contributions
Yu Zhu: Conceptualization, Investigation, Writing – original draft, Writing – review & editing Jiyuan Ye: Supervision, Funding acquisition, Validation, Writing – review & editing
Competing interests
The authors certify that there’s no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.
Funding information
This work was funded by the major project of National Social Science Foundation of China (Grant No. 24&ZD323).
References
Abramo, G., & D’Angelo, C. A. (2011). Evaluating research: From informed peer review to bibliometrics. Scientometrics, 87(3), 499–514. https://doi.org/10.1007/s11192-011-0352-7
Ali, P. A., & Watson, R. (2016). Peer review and the publication process. Nursing Open, 3(4), 193–202. https://doi.org/10.1002/nop2.51
Association of Universities in the Netherlands. (2020). Strategy Evaluation Protocol 2021–2027. https://storage.knaw.nl/2022-06/SEP_2021-2027.pdf
Bertalanffy, L. V. (1969). General system theory: Foundations, development, applications (Revised ed.). George Braziller.
Brooks, T. A. (1986). Evidence of complex citer motivations. Journal of the American Society for Information Science, 37(1), 34–36. https://doi.org/10.1002/(SICI)1097-4571(198601)37:1<34::AID-ASI5>3.0.CO;2-0
Derrick, G. (2018). Take peer pressure out of peer review. Nature, 554(7), 7. https://doi.org/10.1038/d41586-018-01381-y
DORA. (2012, December 16). About DORA. Retrieved from https://sfdora.org/about-dora/
Editorial in Nature. (2020). China’s research-evaluation revamp should not mean fewer international collaborations. Nature, 579(7797), 8. https://doi.org/10.1038/d41586-020-00625-0
Fan, J. (2012). Research on full evaluation theory in the evaluation of the subject librarians: A case study of subject librarians evaluation of Tianjin Foreign Studies University Library. Library Journal, 31(9), 75–82. https://doi.org/10.13663/j.cnki.lj.2012.09.017
Franceschet, M., & Costantini, A. (2011). The first Italian research assessment exercise: A bibliometric perspective. Journal of Informetrics, 5(2), 275–291. https://doi.org/10.1016/j.joi.2010.12.002
Garfield, E. (1955). Citation indexes for science: A new dimension in documentation through association of ideas. Science, 122(3159), 108–111. https://doi.org/10.1126/science.122.3159.108
Geng, Z., & Zhao, X. (2023). An important theoretical innovation of multi-dimensional academic evaluation: A review of on the system of all-round evaluation of research. Library Journal, 42(9), 51–54, 76. https://doi.org/10.13663/j.cnki.lj.2023.09.007
Han, M. (2021). Research on the Evaluation Index System for the Publication Quality of Journals of Humanities and Social Sciences in Universities (Doctoral dissertation). Nanjing University.
Han, Z., Wu, J., & Kong, L. (2021). Practice and thoughts on the collection evaluation of printed foreign books for humanities and social sciences in academic libraries—Taking the philosophy discipline in Zhejiang University as an example. Journal of Academic Libraries, 39(4), 28–35. https://doi.org/10.16603/j.issn1002-1027.2021.04.004
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). The leiden manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a
Higher Education Funding Council for England. (n.d.). Panel criteria and working methods (2019/02). Retrieved January 13, 2025, from https://2021.ref.ac.uk/publications-and-reports/panel-criteria-and-working-methods-201902/index.html
Information Center for Social Science, Renmin University of China. (2025). Orientation • Standards • Examples: An Analytical Report on Original Academic Theories in Chinese Philosophy and Social Sciences. Retrieved from https://zszwx.cn/originalList
Jiang, C. (2023). Put forward a new academic evaluation theory and expand the field of academic evaluation practice—Book review of On the system of all-round evaluation of research written by Professor Ye Jiyuan. Information Science, 41 (11), 185–187. https://doi.org/10.13833/j.issn.1007-7634.2023.11.022
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17. https://doi.org/10.1002/asi.22784
Li, Y. (2020). The evaluation of paper collection in university library based on all-around evaluation system. Journal of Academic Library and Information Science, 38(2), 44–48.
Lu, W., & Ye, J. (2018). Evaluation of library websites under the analytical framework of “full evaluation.” Library Journal, 37(10), 31–42, 60. https://doi.org/10.13663/j.cnki.lj.2018.10.004
Ma, X. (2020). Research on the evaluation of library exhibition service based on the theory of full evaluation. Library, 10, 104–110.
MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: A critical review. Journal of the American Society for Information Science, 40(5), 342–349. https://doi.org/10.1002/(SICI)1097-4571(198909)40:5<342::AID-ASI7>3.0.CO;2-U
Mallapaty, S. (2020). China bans cash rewards for publishing papers. Nature, 579(7797), 18. https://doi.org/10.1038/d41586-020-00574-8
Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008). Improving the peer-review process for grant applications: Reliability, validity, bias, and generalizability. American Psychologist, 63(3), 160–168. https://doi.org/10.1037/0003-066X.63.3.160
Marx, K., & Engels, F. (1975). Marx & Engels collected works vol 01: Marx:1835-1843. Lawrence & Wishart. Retrieved from https://muse.jhu.edu/pub/248/edited_volume/book/31171
Qin, Z. (2024). Research on the quality evaluation index of research activities in libraries: Based on the theory of “full evaluation.” Journal of Academic Library and Information Science, 42(4), 113–117.
School of Information Resource Management RUC. (2021, October 19). Review of the “Development of academic evaluation in the data-intelligence era” forum conference. Retrieved from https://mp.weixin.qq.com/s/-fybfxuD118XbEbrEeaS0A
Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
Technopolis Group. (n.d.). REF accountability review: Costs, benefits and burden. Retrieved January 11, 2025, from https://www.technopolis-group.com/report/ref-accountability-review-costs-benefits-and-burden/
Tollefson, J. (2018). China declared world’s largest producer of scientific articles. Nature, 553(7689), 390. https://doi.org/10.1038/d41586-018-00927-4
Wang, L., & Ye, J. (2014). Bottleneck factors and countermeasures of the evaluation of chinese academic book in humanities and social sciences. Library & Information, 6, 82–87.
Wiener, N. (1965). Cybernetics: Or the control and communication in the animal and the machine (2nd ed.). Mit Press.
Yang, S., & Jiang, M. (2023). Planning for the whole, innovating for the new chapter: Another masterpiece in the field of academic evaluation—Reflections on reading On the system of all-round evaluation of research. Knowledge Management Forum, 8(2), 168–170. https://doi.org/10.13266/j.issn.2095-5472.2023.014
Yang, X. (2023, June 9). Academic evaluation: Rising demand and methodological innovation. Chinese Social Sciences Today.
Ye, J. (2005). Citation method is the quantitative is the qualitative evaluation method again. Library, 1, 43–45.
Ye, J. (2010a). Analyzing the essence of citations and their functions in academic assessment. Journal of Library Science in China, 36(1), 35–39. https://doi.org/10.13530/j.cnki.jlis.2010.01.018
Ye, J. (2010b). Approaching evaluation system in humanities and social sciences. Journal of Nanjing University (Philosophy, Humanities and Social Sciences), 47(01), 97-110.
Ye, J. (2013a). Research and revelation of the “all-round evaluation” of the quality of journals in library sciences. Journal of Library Science in China, 39(4), 83–92. https://doi.org/10.13530/j.cnki.jlis.2013.04.012
Ye, J. (2013b). The innovation, significance and enlightenment of the database of Chinese Book Citation Index (CBkCI) research and development. Journal of Academic Libraries, 31(1), 48–53.
Ye, J. (2014). Development, significance and background information about the “Chinese Book Citation Index” (CBkCI) demonstration database. Scientometrics, 98(1), 557–564. https://doi.org/10.1007/s11192-013-1092-7
Ye, J. (2020a). Re-exploration of the quality evaluation system for Chinese humanities and social sciences academic books. Modern Publishing, 5, 35–43.
Ye, J. (2020b). The keys, roots, and solutions to “SCI Supremacy.” Journal of the China Society for Scientific and Technical Information, 39(8), 787–795.
Ye, J. (2021). On the system of all-round evaluation of research. Social Sciences Academic Press.
Ye, J., Guo, W., Zheng, D., Yuan, X., Ou, S., & Cheng, Y. (2021). Discussion on the quality evaluation index system framework of academic libraries. Journal of Library Science in China, 47(2), 53–66. https://doi.org/10.13530/j.cnki.jlis.2021012
Ye, J., Yuan, P., & Wu, T. (2008). Research on distribution of academic publications of philosophy and social sciences in China. Social Sciences Academic Press.
Ye, J., & Yuan, X. (2021). Discussion on the framework of academic journal quality evaluation index system based on journal core functions. Journal of Jiangnan University (Humanities & Social Sciences), 20(4), 5–17.
Ye, J., & Zhu, Y. (2024). A comparative study of the influence of comprehensive and specialized journal papers in the same discipline :Taking CSSCI-indexed economics papers as an example. Journal of Jiangnan University (Humanities & Social Sciences), 23(6), 5–18.
Zang, L., & Ye, J. (2018). Research on non-core journal evaluation based on the theory of full evaluation. Publishing Research, 6, 46–50. https://doi.org/10.19393/j.cnki.cn11-1537/g2.2018.06.012
Zhang, L., Chen, H., & Fan, J. (2013). Research on professional website evaluation system construction based on all-around evaluation system—Taking painting and calligraphy art website as an example. Library Work and Study, 10, 40–44. https://doi.org/10.16384/j.cnki.lwas.2013.10.004
Zhang, L., & Sivertsen, G. (2020). Combination of scientometrics and peer-review in research evaluation: International experiences and inspirations. Journal of the China Society for Scientific and Technical Information, 39(8), 806–816.
Zhao, Y., Liu, Z., Liu, W., Gao, S., & Zhang, L. (2020). Building the evaluation index for creative open data competitions: A case study on the project of digital humanities. Journal of Library Science in China, 46(2), 75–95. https://doi.org/10.13530/j.cnki.jlis.2020016
Zhong, Y. (1991). Complete information: definition and measures. Journal of Beijing University of Posts and Telecommunications, 3, 1–14.
Editors
Ludo Waltman
Ludo Waltman
Editorial assessment
by Ludo Waltman
In this article the authors present the System of All-round Evaluation of Research (SAER), a theoretical framework for research evaluation developed in China. The article has been reviewed by two reviewers. Both reviewers have reservations about the article. The reviewers raise questions about the purpose of the article and the intended audience. They also express concerns about issues related to theory development, and in particular about the role of Marxist value theory in the SAER framework. In addition, reviewer 1 questions the uniqueness of the SAER framework. Drawing on a comparison between research assessment in China and in the Netherlands, reviewer 2 raises some issues concerning the use of peer review at the scale needed in the Chinese research system.
Peer review 1
The paper addresses the topic of holistic research assessment systems and provides the case of System of All-round Evaluation of Research (SAER) in China. I believe it is an important collective exercise to build up a library of different reform frameworks from around the world and China’s growing importance within global research make it a particularly important setting.
Unfortunately I have significant reservations about the paper’s contribution. My comments below are intended to be constructive, and I hope they are useful regardless of how the authors decide to proceed from here. The good thing about MetaROR at least is the authors are in charge of what to do next, and MetaROR readers can make up their own minds about whether my comments are fair or not.
The presentation of an ‘evaluation theory’
The term ‘evaluation theory’ in respect to SAER (e.g. on page 9) in the paper is, as it stands, unclear as to what kind of theory it refers. Does it refer to a specific middle-range framework (i.e. SAER), to the broader scholarly canon of evaluation and programme theory, or to a more general abstract theoretical tradition (e.g. Marxist value theory, cybernetics etc)? The ambiguity matters because ‘evaluation theory’ carries distinct connotations in each of these contexts, and readers from different communities may interpret it quite differently. Even in the abstract section, it is unclear whether the paper is about macro-level theories underpinning the design of SAER or the SAER as a practical, ‘theory-in-use’ evaluative framework. Without signposting or explanation, it made it difficult for me to understand what I was reading about.
The Marxist theory of value
I found the engagement with Marxist value theory difficult to follow. In its current form, it was hard to distinguish the contribution of this framing from a more general account of international trends in research impact and utilisation. Maybe I’m embarrassing myself here because I know very little about Marxism, but I could not tell the difference. If the genealogical linking of utility to Marxism is intentional, and if this genealogy is important to how this framework has emerged and evolved in China, the paper needs to make that intellectual scaffolding explicit and demonstrate clearly what the Marxist value theory lens adds that other framings would not. So I would suggest either developing this thread substantially or removing it.
Empirical grounding and uniqueness of the framework
The paper introduces a framework (SAER) and implies it can address a range of dysfunctions in research assessment, but I was not convinced on two counts. First, accounts of how the framework has been tested in practice and what the empirically observed results have been are unclear, so its proposed advantages seem largely promissory. So my understanding is there is not yet adequate proof-of-concept. Second, and more fundamentally, I was not persuaded of SAER’s distinctiveness. There are other frameworks that move beyond binary evaluation logics through multi-dimensional and holistic principles (or claim to). What the SAER framework offers that is unique is not stated, so it is difficult to credit the big claims being made about SAER and its contributions to the advancement of the global assessment reform agenda.
Audience and purpose
This is perhaps the core issue. I could not get a stable sense of the target audience: it could be for academic readers, but possibly also policymakers, instrument designers, and practitioners simultaneously. I did not feel the paper really commits to any of them though, and these are different audiences requiring different kinds of contribution. As it was sent to an academic conference, my best guess is it is aimed at an academic audience, in which case the relevant scholarly literature and the shared problematic the paper is positioning itself within are not signalled. I would strongly encourage the authors, should they develop this further, to decide on a primary audience and think hard about the purpose of the contribution, and to restructure accordingly. In its current form the paper moves in too many directions, and the lack of coherent signposting and focused purpose makes it difficult to navigate for this reader and I suspect for other readers.
Peer review 2
Reviewing this manuscript has been burdensome to me, as I did not know, and still do not know, how to see this manuscript. Is it a full academic manuscript, presenting a piece of research, or is it a political pamphlet? Of course, the way one perceives a manuscript for review determines how one assesses the manuscript. This paper has been sent to a conference, so I assume it is meant to reach an international academic audience, I still feel that the main message of the manuscript is mainly a political one, for which legitimation by an international academic audience was sought.
I see three major issues with the manuscript, which I discuss below.
Lack of theorization in global research assessment practices
The authors present their work as a contribution to an international debate about research assessment, and as such as the finally expected theoretical framework for the evaluation of research, thereby claiming that this has been a missing element in the global landscape of research assessment. Here, I care to disagree with the authors, as many if not all countries that periodically assess their national research system, accompany and support such exercises with guiding principles and instructions, in which research and research quality is being conceptualized, values expressed, and choices made.
As a Dutch researcher in the field of science studies, and in support of research assessment exercises from the 1990s onwards, I am familiar with already the fifth national evaluation protocol from 2003 (and their predecessors in the 1990s), which have over time shown and presented the developments in the national science system, and the thinking on how to assess research quality best, adjusted to national and international developments, initiatives and interventions on the process of quality control of scholarly work, here research.
The protocol is a helpful tool, of which some elements are mandatory, that is, some elements need to be dealt with, but still allowing many degrees of freedom on how to design and organize the assessment procedure. Over time we have seen the advent of societal relevance next to academic quality, the increasing space for qualitative methods to support research assessment, reducing the emphasis on bibliometrics, although bibliometrics as such are not excluded from the procedures. It is this high degree of freedom that makes the Dutch Strategy Evaluation Protocol such a helpful tool across the whole scholarly landscape, from physics and astronomy through biomedicine to the social science, humanities and law domains.
Functioning of peer review in research assessment procedures according to the SAER model
As a Dutch national, working in academia for almost 40 years, and having a vast experience in research evaluation, I realize myself that both geographical scale and political-governmental issues by which we organize quality control on research in the country, but actually any country, are quite important aspects to take into consideration. In the Netherlands we are dealing with in total 14 universities in the country, of which seven classify as classical universities, covering nearly the whole range of scholarly domains, four universities of technology or agriculture, two universities without a faculty of science, but with a large academic medical center, and finally the Open University. The Dutch evaluation system can be qualified as “informed peer review”, in which the assessment and judgment of peers is dominant, which can be supported by for example making use of research metrics. This limited geographical scale of the Dutch research system means that national assessments for one particular scholarly discipline are relatively easy to organize, involving one peer review committee for the whole country and the universities involved, as geographical distances for in-person site visits can be covered within an acceptable travel time range, and as such taking care of a more or less consistent and uniform assessment by the assigned peer review committee.
Comparing the Dutch landscape with the situation in and for which SAER was designed, namely China, the situation is of course completely different: a multitude of universities, and hence staff, spread over a much larger geographical space. This makes organizing research assessment in which peer review is the leading principle much harder, in particular if one cares about the consistency of the assessment, by not having too much variation in the composition of the peer review committees at work. Basically, I think this is unavoidable and hence the consistency of the peer review process is easily lost. If one cares to ignore that inconsistency, and accepts that as a consequence of the large geographical differences compared to the Netherlands, one also introduces a certain degree of unfairness, as peer review committees do not always think similarly, put other emphases, etc. The empirical material in the form of the cases included in the paper further confirmed that the model might work on a smaller scale, but not for the whole of China at the same time.
Central role of Marxist value theory in the design of the SAER model
The most recent version of the Dutch Strategy Evaluation Protocol, launched in March 2026, covering the time period 2027-2033, carefully describes in detail the various aspects of the procedure, the various roles, tasks and responsibilities of the various actors and stakeholders involved, without becoming too prescriptive, and emphasizing the formative character of the procedure over the summative, whereby it is important to stress that accountability is still there but is considered less important than the learning aspect of the exercise. I must also stress that in the Dutch context, there is no direct link between the outcomes of assessment and research funding, unlike the UK, and apparently, unlike China. The dependence on Marxist value theory in SAER makes one wonder what function this part of Marxist theory has in the SAER model, as Marxist theory emphasizes the economic realm as the dominant realm in society, compared to for example the cultural, political and other realms of society, which makes me think that economic part of an evaluation model might be oriented towards the accountability part of the model, namely about efficiency on costs in the governance of research. One fundamental criticism on Marxism is its economic determinism, whereby everything is related to and explained from economic reasoning. This brings me back to the second issue I raised earlier, namely the political context in which an evaluation model is being implemented. China, in its centrally-led society and, consequently, top-down management of research is probably helped by a system in which the basic societal philosophy returns in its assessment framework, whereas the situation in the Netherlands, which can be qualified as an open democratic society, differs from China, which is reflected in the Dutch approach to research assessment, in which the academic community, both local policy makers and scholars, have an influence on the way the assessment protocol is designed and adjusted over time. In that sense, community participation and academic freedom plays an important role in the organization of research assessment procedures in the Netherlands.
Concluding remark
Apart from the three major issues discussed above, the way the SAER model is described in much detail, together with a high degree of directiveness, does create a feeling of top-down managerial intervention of which I think academia can do without. The SAER model in that sense does seem to be conflicting with the basic principles of science. But that might be due to me being biased, being Dutch and working in the Dutch science system.
Author response
We thank Ludo Waltman for organizing this review, and we thank Alex Rushforth and Thed van Leeuwen for the time and care they gave to our manuscript. Their comments approach the paper from different angles and both have been valuable to us. Several of the problems they identify are ones we could not easily see from inside the Chinese literature on research evaluation, and it took the eyes of international colleagues to bring them into view. We have decided to revise the article substantially and would welcome a second round of review at MetaROR. We set out below our response to the main concerns.
1. Purpose and audience
The paper was written to introduce SAER to an international readership, not to construct or defend the theory here; the full demonstration is carried by a Chinese monograph (see: Ye, J. (2021). On the System of All-Round Evaluation of Research. Social Sciences Academic Press. [in Chinese]). A six-page conference paper cannot do both, and by attempting an introduction without telling readers where the argument lies we left the purpose and the intended audience unclear. That failure is ours. Our primary readers are scholars of research evaluation; any policy relevance we hope for is indirect. The revision will state this at the outset.
2. Where SAER comes from, and what it is
SAER was not deduced first and applied afterwards. Its practical basis goes back to the first author’s work on academic journals from the 1980s onwards, and the framework itself was abstracted from the wider practice of research evaluation — more specifically, from the characteristics and problems presented by the many evaluation practices in the Chinese humanities and social sciences. Once it had taken shape, its earliest objects of testing and application were academic books, journals, library collections and subject websites, all of them problems internal to library and information science. Since it was formally proposed in 2010 it has been tested against historical and practical material.
This bears on the difference in level between SAER and a national evaluation protocol. A protocol such as the Dutch SEP governs how an assessment exercise is organized and carried out. SAER is a theory abstracted from practice which in turn guides practice; it is neither an evaluation manual nor national policy. The article was funded by a major project of the National Social Science Foundation of China, which means the subject is considered worth serious study, not that the theory has been adopted as state-sanctioned evaluation policy. Whether it can eventually inform evaluation at the national level remains a question for further research. By placing SAER alongside international declarations without explaining this difference in level, our text invited misreading.
3. On the term “theory”
The reviewers are right to ask what “evaluation theory” refers to in our text. When SAER was first proposed in 2010 we did describe it as an analytical framework. Over the following decade and more, through continued application, testing and refinement, it developed into a systematic theory, and that work continues. The Chinese term “理论/lilun/theory” is also broader in usage than the English “theory”, a difference our text did not address. The revision will be more careful in its English wording and will make this history of development explicit. We also accept the related point that the paper should engage more fully with existing scholarship on evaluation; that engagement was insufficient.
4. The function of Marxist value theory
Our treatment of this element was too compressed for its function to be legible. That function is axiological rather than economic. Marxist value philosophy understands value as a socio-historical relation between subject and object grounded in human needs, from which it follows that evaluation must rest on judgements of fact yet cannot stop there. This is the source of the objectivity standard in SAER and of our claim that factual judgement alone is insufficient for value judgement. It belongs to a different register from the base-superstructure thesis of historical materialism, so concerns about economic determinism do not bear on it. As for what it adds, it supplies the argumentative ground for holding that value judgement is possible and that it cannot be reduced to indicators. We would add that the utility dimension of SAER emphasises long horizons, testing through practice, and events and cases as its main evidence, which places it closer to societal relevance and formative assessment than to accountability in the sense of cost efficiency.
5. Consistency in large-scale peer review
We take the analysis of geographical scale and the consistency of peer review seriously. One clarification: SAER does not call for a single uniform national assessment of Chinese research. Its principle that each evaluation should serve one clearly defined purpose requires the opposite, and its classification of evaluation objects requires designs fitted to the characteristics of each. As for drift between review committees, meta-evaluation and appeal mechanisms are the institutional answer the theory provides, and the paper did not explain this adequately. We also accept that consistency is an implementation problem facing any large-scale peer review system, and that the remedy lies in refining institutional design rather than abandoning the leading role of peer experts.
6. Relation to international reform initiatives
On substance we agree with the Leiden Manifesto: indicators must not substitute for informed judgement, and performance should be measured against research missions. What SAER represents is a theory formed independently, in a different scholarly tradition and out of different practical material. We make no claim about precedence; our wording on this point was careless and will be revised. That frameworks grown in different contexts can be read against one another seems to us one of the ways evaluation reform advances.
We thank the reviewers and the editor again. We will revise the article along the lines set out above and look forward to a second round of review.
