Introduction

In recent years, discussions and campaigns in favor of a transformation in research evaluation have multiplied and deepened (Peruginelli and Pölönen 2023; Rushforth and Hammarfelt 2023; Morgan-Thomas et al. 2024; Rushforth 2025a). In this context, concrete reform proposals have been developed, calling on various stakeholders (universities, researchers, funding agencies, policymakers, journal publishers, bibliometric data providers, among others) to revise their evaluation mechanisms and criteria. Interventions such as the San Francisco Declaration on Research Assessment (DORA, 2013), the Leiden Manifesto (Hicks et al. 2015), the Metric Tide report (Wilsdon 2016), and the Hong Kong Principles (Moher et al. 2020) are considered landmark declarations that have sought to raise awareness and, in a way, “nudge” (Rushforth 2025a) research actors to reconsider their positions regarding the use of quantitative indicators, open science, and research integrity.

This reform movement, known as Responsible Research Assessment, emphasizes the need to recognize and value a broad spectrum of knowledge, languages, outputs, career trajectories, and institutional profiles in the scientific realm (UNESCO, 2021; CLACSO, 2022). The basis for this claim is to promote greater inclusion, equity, and fairness in research evaluation processes (Rovelli, 2024), but also to signal the types of scientific outcomes and career paths that should be encouraged to achieve the production of socially impactful knowledge (Donovan, 2007). Evaluation is viewed not merely as a technical domain—it involves political and strategic considerations about the kind of research and knowledge that should be promoted. Therefore, it is essential to consider different geographic and institutional contexts and to question the idea of a global consensus surrounding the reform agenda (Vasen, 2025).

Recognizing the diversity of scientific outputs and career paths implies valuing achievements that go beyond scientific publishing and narrow notions of productivity (Pontika et al. 2022). In particular, TAPs form part of the incentive and reward systems for applied research (Mollas Gallart et al. 2002). In our work, we understand TAPs as encompassing both the activities of technological and applied work, and the outputs that result from them. These may include intellectual property rights, technological services, advisory work, and similar undertakings. Such activities are typically carried out in collaboration with, or in response to the needs of, non-academic entities. Unlike the notion of social impact, which emphasizes the effects of research on external stakeholders or broader society, TAPs are examined primarily from the perspective of the researcher and the R&D-performing institution, focusing on their role, contributions, and interactions with nonacademic actors. Whether in the context of hiring, tenure, or promotion in academic careers (Genshaft et al., 2016; Alperin et al 2019; Rice et al., 2020; Bouwma-Gearhart et al. 2021; Melkers, Woolley & Kreth, 2023; Pekkola & Siekkinen, 2024; Cañibano et al., 2024) or in funding calls for innovation and R&D projects (Ramos-Vielba, Thomas and Aagaard, 2022; Norn et al., 2024), TAPs are increasingly part of evaluation processes.

Although the analysis of social impact may offer relevant perspectives, the evaluation of TAPs involves a different set of challenges that must be addressed on their own terms. A number of recent studies have focused on the evaluation of social impact, both at the level of universities and in the context of academic careers and R&D project funding (Ma et al. 2020; Ma and Agnew 2022; Rice et al. 2020; Benneworth and Olmos Peñuela, 2019; Urbanc and Jong, 2025). A variety of methodologies for measuring ex post social impact have been developed, which are qualitatively different from those used in traditional research evaluation, as they usually involve the development of case studies and interviews or workshops with external stakeholders and aim to assess impact within a broader time frame (Smit and Hessels 2001). By contrast, TAPs may form part of more routine ex post evaluation processes to which academics are subjected. Just as published papers are assessed over a given period, so too are applied and technological activities. However, since the traditional evaluation methodologies used in those contexts, such as peer review, were not designed for this kind of output, they may show significant limitations when applied to TAPs.

This article explores current practices for evaluating TAPs in researchers’ applications for promotion within the Argentine research system. Based on an analysis of external reviewers’ reports on promotion applications, we seek to gather information on how TAPrelated content in this contexts is evaluated and weighted. What do evaluators highlight about TAPs? What difficulties do they perceive? What perspectives do they hold on the relationship between TAPs and scholarly publishing? What criteria do they use to judge the relevance of TAPs? Does the type of counterpart, the number of activities, the amount of funding received, the applicant’s role in the team, or the level of formalization of the activity affect the evaluation?

We believe this study can help anticipate the challenges that may arise if the principles of “responsible research assessment” are followed, especially in terms of recognizing a wider diversity of contributions and career profiles. In the next section, we discuss broader debates around the assessment of social impact and then focus on how it has been evaluated specifically within academic careers. We then outline Argentina’s scientific career system—the context for our study—along with the methodology and data sources employed. Following this, two sections present our findings from the quantitative analysis and the qualitative text analysis, respectively. Finally, we highlight the key challenges in evaluating TAPs and explain why this topic warrants separate consideration from social impact assessment, all within the larger goal of fostering a more inclusive understanding of academic contributions.

The Evaluation of Social Impact as a Research Problem

Concern for the social impact of knowledge has been present since the early development of public policy in science and technology (Langfeldt et al., 2020). What is new in recent years is its incorporation as a formal criterion in evaluation processes (Luo, Ma, and Shankar, 2022). The challenges of assessing social impact begin with its very definition. Depending on the purpose of the evaluation, the institutional setting, and the broader sociopolitical context, the concept of impact can be interpreted in multiple ways, giving rise to significant difficulties (Ma et al., 2020). How impact is defined directly influences how it is measured; both are political choices, as they determine which types of outcomes are considered valuable and potentially rewarded, and which are rendered invisible. Whereas early attempts to measure research impact were grounded in the notion that science should enhance a country’s international competitiveness and generate wealth—leading to improved metrics based on economic performance and STI indicators—more recent perspectives understand social impact in broader terms (Donovan, 2007). It is now acknowledged that research should respond to urgent societal challenges, including not only economic and technological issues but also social, cultural, and environmental concerns (Donovan, 2011).

Addressing problems and biases in peer review within evaluation processes is receiving increasing attention in the literature, driven by the high volume of applications and the growing diversity of research outputs and evaluation contexts (Kolarz, 2023). In this context, studies have examined the potential benefits of two-stage evaluation processes (Seeber, Svege, and Hesselberg, 2024) and the use of randomization (Roumbanis, 2019; Schweiger, 2024; Davies and Ingram, 2025). When it comes to social impact, evaluation outcomes tend to exhibit particularly high levels of variability (Derrick and Samuel, 2016; Derrick and Samuel, 2017; Oxley and Gulbrandsen, 2025). Building on previous studies, Bornmann (2013) identifies four key challenges in the ex post evaluation of social impact: causality, attribution, the international nature of R&D, and time scale. Other issues go beyond the intrinsic characteristics of impact and relate instead to the evaluation process itself— affecting both ex ante and ex post assessments. For example, Luo, Ma, and Shankar (2022) found that modifying panel composition to include non-academic stakeholders leads to more appropriate assessments of social impact. Similarly, Abma-Schouten et al. (2023) analyze the vocabulary used by members of ten evaluation panels and observe that, during meetings, evaluators’ arguments regarding social impact vary depending on the panel’s diversity. In their case study, panels composed of researchers, patients, and health professionals placed greater emphasis on arguments related to social relevance.

Oxley and Gulbrandsen (2025) investigate the sources of variation in social impact evaluations by examining how review panels assess impact, focusing on the interplay between individual-level and environment-level factors, and the moderating role of panel interaction processes. At the environmental level, they show that clearly defined social impact criteria—distinct from scientific impact—are essential for producing more robust evaluations. Equally important is the use of collectively agreed-upon scoring systems, rather than relying on individual discretion, as well as calculating overall scores by averaging specific criteria instead of allowing evaluators to weight them arbitrarily. At the individual level, reviewers’ dispositions also contribute to variability. While most researchers acknowledge that social value is an important dimension of high-quality research, there are mixed views about reviewers’ ability and willingness to assess social impact criteria. The authors distinguish between two forms of variation in assessment: productive and unproductive. Productive variation stems from the diverse professional trajectories and perspectives of reviewers. Unproductive variation, by contrast, arises from flaws in the design of the evaluation process. These sources of unproductive variability can be influenced—and potentially reduced—by the organizing authority responsible for overseeing the evaluation process.

Other mismatches may arise in terms of values -between what reviewers perceive as important and what the evaluating institution defines as important-. Ross-Hellauer et al. (2023) found a disconnect between the factors scientists view as important for career success (e.g., conference presentations, service, research impact, and engagement with local communities) and what they perceive as valued for academic tenure and promotion within their institutions (e.g., number of publications, journal quality, acquisition of external funding). In other words, there is a misalignment between what researchers personally consider relevant and what they believe their institutions reward. However, these institutional policies for career evaluation are partly defined by the same academics who must later apply them through their participation in evaluation committees. This overlap of roles blurs the distinction between institution and individual, creating a kind of value dissonance. Bruhn (2008) defines this as a distressing mental state in which people find themselves doing things they do not particularly value or holding opinions that clash with institutional norms or the views of those who supervise and enforce them.

Scientific Careers and Social Impact

In a context shaped by the rise of the impact agenda, a variety of institutionalized evaluation approaches have emerged that aim to highlight how research is disseminated into society and produces effects at multiple levels. Impact has become yet another “box to be ticked” on the list of academic achievements—another component of so-called “academic excellence” (Wróblewska 2021). As a result, social impact has increasingly been incorporated as a dimension to be assessed within many national evaluation systems. More recently, this has been accompanied by a new specialized vocabulary, the formalization of procedures and requirements, the creation of dedicated offices or departments, and other related developments—all contributing to what Power (2015) describes as an emerging “impact infrastructure.”

The impact case study methodology implemented in Australia and the United Kingdom has been one of the most widely discussed strategies and has influenced similar approaches in countries such as Poland and Norway (Williams and Grant, 2018; Wróblewska, 2025). Within these frameworks, institutions submit case studies based on selected research topics that have demonstrated a positive societal impact, as part of their overall institutional evaluation. University rankings may also play a complementary role, insofar as they assess the level of innovation produced by universities (Vasen, 2025).

In contrast to these institutional evaluations—which generally fall under the category of Performance-Based Research Funding Systems (Hicks, 2012)—this article focuses on how TAPs are assessed in the individual evaluation of researchers’ trajectories. Within the context of academic careers, social impact is often reflected in a range of activities, including engagement with the industrial sector through knowledge transfer or the provision of services (Friesike et al., 2022; Fecher and Hebing, 2021; Reale et al., 2018; Vasen and Sierra 2022). Knowledge transfer may take many forms, depending on the discipline and on the institutional missions. At the institutional level, these activities are typically grouped under the university’s so-called “third mission” (Molas-Gallart et al., 2002).

In research evaluation processes, these industry-related activities were among the first to be measured using quantitative indicators—such as the number of commercial agreements, spin-offs created, partnerships with industry (including funding received, coauthorships, and joint patents), new technologies, new products, or research leading to commercial outcomes (Donovan, 2007). However, Derrick and Samuel (2016) point out a growing difficulty in evaluating such activities, as the criteria remain non-standardized and are often ambiguous for both applicants and reviewers, leading to various types of disparities.

Within these evaluation settings, sections dedicated to products and activities developed with or for non-academic actors have gained increasing prominence and significance. This has added a new layer of responsibility for researchers: the need to clearly and concisely demonstrate how their work contributes to collective well-being. What has become increasingly evident is the contradiction between the growing expectations for researchers to respond to societal needs and the prevailing reward and incentive structures in academia, which continue to favor scientific performance metrics—such as journal impact factors—over broader contributions (Cañibano et al., 2024; Vasen, 2018).

In an effort to recognize a broader diversity of outputs and activities within academic careers, several reform initiatives have emerged—both globally, such as the Coalition for Advancing Research Assessment (CoARA), and nationally, including the Recognition & Rewards program in the Netherlands and the sexenios de transferencia in Spain (Repiso and Montero, 2024; Rushforth, 2025b).

In this article, we examine how technological activities and products are evaluated within the context of academic careers in Argentina. Although discourse around the social impact of knowledge and university–industry engagement has existed for decades in the country, it has not been matched by significant initiatives or reforms in research evaluation practices. Our analysis focuses on disciplines such as veterinary sciences, computer science, and engineering, which—due to their applied nature—are expected to exhibit frequent interaction with non-academic environments. The aim is to identify the main challenges reviewers face when issuing ex post assessments of candidates’ technological activities.

Background: Evaluation in the Research Career in Argentina

Since 1961, the National Scientific and Technical Research Council (CONICET) has administered a research career track for scientists in Argentina. Members hold permanent research positions as civil servants, working in CONICET’s own institutes, in universities, or in joint institutes between the two (Niembro and Svampa, 2024). The general framework shares some similarities with those of the CNRS in France or the FNRS in French-speaking Belgium. The career is structured in five successive stages: Assistant, Adjunct, Independent, Principal, and Superior Researcher. Entry at the first stage typically requires a doctoral degree and some postdoctoral research experience.

Scientific research in Argentine universities is highly dependent on CONICET, as universities employ relatively few full-time faculty members. Researchers in the CONICET career track usually combine their research activity with teaching in universities, thereby integrating teaching and research. Currently, the system includes 12,176 members, distributed across five broad areas of knowledge: Agricultural, Engineering and Materials Sciences (25%), Social Sciences and Humanities (22%), Biological and Health Sciences (27%), Exact and Natural Sciences (22%), and Technology (3%) (CONICET en Cifras, 2024).

Entry into the career track typically occurs at the first stage—Assistant Researcher— although in exceptional cases, candidates with a strong track record may be admitted at a more advanced stage. Once admitted, and after spending a minimum period in one category, researchers become eligible to apply for promotion to the next. For this process, they must submit a detailed report of all outputs produced since attaining their current rank, along with a statement justifying the request. Applications are evaluated by disciplinary panels, which assign external reviewers^[1]While we refer to these reviewers as “external,” it is important to clarify that some may themselves be members of CONICET’s research career, often based at other institutions or in different cities from the applicant. Panel members are careful to avoid selecting reviewers with close personal or professional ties to the applicant. Given that a large proportion of full-time researchers in Argentina are affiliated with CONICET, it would be nearly impossible to rely exclusively on reviewers from outside the organization. The term “external” is thus used to indicate that these individuals are not part of the evaluating panel and are only involved in the production of an expert report—they do not participate in the deliberative stages of the evaluation process. according to the applicant’s field of research. The reviewers’ reports inform the panel’s analysis but are not binding. Based on this assessment, the panel issues a recommendation (promotion or no promotion), which is then reviewed by a multidisciplinary board and, ultimately, by the agency’s executive council.

For this study, we obtained access—under a confidentiality agreement—to a database of external reviewer reports corresponding to promotion applications submitted in 2017 and 2018. The data cover three disciplines selected for their strong applied and technological orientation: veterinary sciences, civil engineering, and computer science. In total, we analyzed 234 promotion applications: 112 in veterinary sciences, 79 in civil engineering, and 43 in computer science. Since each application is typically reviewed by more than one external expert, the dataset includes a total of 421 reviewer reports: 207 in veterinary sciences, 158 in civil engineering, and 56 in computer science. These external reports, rather than the assessments made by the disciplinary panels, the board, or the executive council, constitute the core empirical material for our analysis.

External reviewers complete their reports using an online platform provided by CONICET, known as SIGEVA. Through this system, reviewers can access standardized curricular information submitted by applicants, as well as supporting documents such as full-text publications and other materials relevant to the application. They also have access to a freely formatted PDF version of the applicant’s CV. The evaluation form itself is divided into four distinct sections: (1) Publications, (2) Technological Activities, (3) Career trajectory, current status, and future prospects, and (4) Additional comments. Reviewers may leave any section blank if they choose.

The analysis of this material followed two different methodological approaches. First, we conducted a textual quantitative analysis using the Alceste method (Analyse des Lexèmes Cooccurrents dans les Énoncés Simples d’un Texte), implemented through the Iramuteq software. This semi-automated method is designed to analyze large corpora by identifying patterns of repetition in simple statements (Reinert 1986, 1990, Romano et al., 2020). It detects dominant “lexical worlds,” defined as groups of words and expressions associated with specific topics or fields, and examines their proximity or distance based on word positioning and repetition patterns. We applied this tool twice to the evaluation reports to assess the extent to which the evaluation of technological activities and products features prominently in the discourse of external reviewers. Second, we carried out a qualitative content analysis of section 2 of the expert’s evaluation form (“Technological Activities”) from all 421 reports, identifying the key issues reviewers raised in their assessment of these activities.

The Lexical Worlds of Evaluation

The Alceste method was applied in two separate runs. In the first, we used the full text of all 421 reports as the corpus. This analysis revealed five distinct lexical clusters, which showed only weak connections with one another (see Figures 1 and 2). By examining the most representative words in each cluster, we can infer the main focus of the reviewers’ comments within each group. Class 2 (grey) centers on applicants’ publication records, with an emphasis on journals and their indexing, reflecting the importance reviewers place on traditional scholarly output. This cluster appears largely isolated from the others. Class 5 (lavender) is associated with the evaluation of applicants’ involvement in training and mentoring activities. Cluster 1 (red) captures reviewers’ assessments of the significance of the applicant’s research results and their future potential as a researcher. Finally, Class 3 and 4 (green and blue) are shaped by discipline-specific language: the green cluster is linked to engineering and computer science, while the blue cluster reflects biological terminology, particularly in the context of veterinary science.

Figure 1. Factor analysis based on the full text of expert review reports

 Figure 2. Dendrogram of lexical clusters derived from the full text of expert review reports

Interestingly, no lexical world emerged related to technological production or knowledge transfer, despite the evaluation form including a dedicated section for this purpose. This absence contrasts with the presence of a distinct lexical world related to training and mentoring graduate students (Class 5, lavender), which reviewers often elaborate on even though it is not explicitly prompted.

Given the lack of specific insights into the evaluation of technological activities and products in the analysis of the full reports, we conducted a second Alceste analysis restricted to Section 2—the section of the form explicitly addressing applied and technological activities (see Figures 3 and 4). Notably, only 274 of the 421 reports (65%) contained processable content in this section; the remaining reports were either left blank or included minimal responses such as “none” or “not applicable.”

Figure 3. Factor analysis based on the section on TAPs from experts reports

Figure 4. Dendrogram of lexical clusters derived from TAPs section of review reports.

This time, the method identified four distinct lexical worlds. Class 1 (red) stands apart from the others and centers on domain-specific references to the biological and veterinary sciences. Class 4 (violet) appears to focus on the perceived social impact of technological developments. Meanwhile, classes 2 (light blue) and 3 (green) are closely interconnected and pertain to the formal aspects of evaluating and certifying technological products and activities.

As mentioned, only 274 of the 421 reports contained evaluable content in this section. Here, we did observe disciplinary differences: in engineering, 77.78% of the reports contained usable information, compared to 58.0% in veterinary sciences and 55.4% in computer science.

Text length of reviewer comments ranged from 11 to 916 words, with an average of 113, a median of 74, indicating that most texts are relatively brief, with only a few outliers. Only 37 reports (13%) exceeded 200 words. Disciplinary differences in this regard were not particularly marked, although the longest texts were found in the engineering field (see Figure 5).

Figure 5. Boxplot showing the distribution of length of reviewer comments on Technological Activities and Products, by discipline.

The results of this initial exploratory analysis of the text corpus offer several preliminary insights. First, when considering the full reviewer reports, technological aspects do not appear as a central focus. Instead, reviewers tend to emphasize other dimensions—such as publications, training and mentoring activies, or the development of the trajectory of the applicant’s research agenda. However, when attention is restricted to the specific section dedicated to technological production, certain key terms do emerge. These are primarily related to the formalization of such activities and include references to “STAN,”^[2]The acronym STAN stands for “High-Level Technological Service” (Servicio Tecnológico de Alto Nivel), an administrative category through which CONICET centers can provide R&D services to third parties. “consulting,” “agreement,” and “patent,” as well as mentions of “SIGEVA,” “documentation,” and “evidence.” To better understand how these terms are used in context and what they reveal about the evaluation process, we conducted a qualitative content analysis of this section. Notably, this section of the expert report seems to be more actively engaged with and valued in engineering, as reflected by a higher proportion of reports containing usable information and a longer average text length compared to the other two disciplines.

Mistrust, Dissonance, and Noise: The Peers’ Perspective

In the second stage of our study, we conducted a content analysis of the texts written by external reviewers in the section of the evaluation reports dedicated to technological activities. Two issues consistently stood out as particularly challenging for reviewers: insufficient documentation and contradictions between the various sources of information provided by applicants. In addition to these, we identified three other areas marked by significant variation in reviewers’ criteria: the originality of the work, its social relevance, and the relationship between scientific and technological output. Together, these five issues highlight the key challenges reviewers face when assessing technological activities in the context of promotion applications. Below, we describe each of these in more detail.

Insufficient Documentation

A recurring theme in the TAPs section of external reviewers’ reports is the difficulty of assessing activities due to a lack of adequate information. Reviewers frequently remark that the data entered into the SIGEVA platform often lacks the detail needed to support a robust assessment. Although SIGEVA allows for the uploading of supplementary documentation and includes sufficient fields to record relevant information, the problem—unlike what was found in our previous study on Uruguay (Vasen and Sierra 2022)—does not stem from the platform’s design. Rather, it relates to the quantity and quality of information applicants choose to submit about their TAPs.

Importantly, there is variation not only in the completeness of the information provided by applicants but also in reviewers’ expectations regarding what constitutes adequate documentation. The following excerpts illustrate these discrepancies: in one case, a signed agreement is considered insufficient; in another, reviewers explicitly request either such an agreement, a certificate issued by the university, or a detailed report. Elsewhere, a case is criticized for including a report that is considered too brief.

In the cases I reviewed, only a one-page technical report was attached, making it difficult to assess the complexity of the work carried out. I must also point out that the documentation is inadequate; there should be a certificate issued by the university confirming the work performed and the amount received. (123)

Only a copy of the signed agreement was attached, with no document or report describing the work undertaken. Under these conditions, it is impossible to evaluate the impact, quality, or relevance of the transfer, or to determine the applicant’s contribution. (14)

It is unclear whether this refers to an informal agreement or a paid technology transfer service. Neither the agreement/contract nor a technical report that would enable a better evaluation has been included. (389)

In any case, the core issue is not whether the supporting material takes the form of an agreement, a report, or a certificate. Rather, the problem lies in the fact that the materials provided are often insufficient for carrying out the evaluation appropriately.

In addition to seeking information about the quality and relevance of the activities described, reviewers also try to assess the applicant’s specific role within the team. Was the applicant part of a large team, or was the activity carried out by a small group in which the applicant played a decisive role? In most cases, external experts note that the information available does not allow them to answer such questions with confidence.

Dissonance Between Sources

Since the platform used for submitting background information does not present structural limitations for recording the necessary details, reviewers tend to direct their criticisms at the candidates rather than at the system itself. In addition to concerns about insufficient documentation, we identified a recurring issue that we refer to as “dissonance between

sources,” which complicates the evaluation process. This occurs when relevant information about TAPs is inconsistently presented across different parts of the application—for example, activities may be described in the uploaded CV but not properly registered in the dedicated sections of the SIGEVA platform. Such inconsistencies hinder the reviewers’ ability to form a clear and comprehensive understanding of the applicant’s technological contributions.

Although the applicant has not uploaded data in SIGEVA, her CV reveals significant involvement in transfer services in her laboratory and in the validation of antibodies for the species on which most of her research focuses. (43)

Although the candidate mentions “the completion of numerous technical consultancies” in the “Justification of the Application” section, SIGEVA shows no evidence that he participated in the development of production systems, processes, products, or technological developments. Nor is this evident in his CV. (7)

No elements that allow for the evaluation of this activity have been uploaded to SIGEVA. However, from the narrative in the applicant’s presentation and some of the grants obtained, it can be inferred that there was technological activity. (206)

The issue arises when applicants fail to report their TAPs in the designated section of the SIGEVA platform, choosing instead to mention them in their CV—uploaded as a separate file—or in the open-text field intended for the justification of the promotion request. This fragmented presentation complicates the task of external evaluators, who must go beyond the structured sections and piece together information from different parts of the application. As a result, reviewers are often forced to ‘dive’ into disparate documents to reconstruct a coherent picture of the applicant’s technological activities.

There is no single explanation for why applicants often leave out TAP information from the section specifically designed for it in SIGEVA. One possible reason is that they may not see these activities as important or worthy of inclusion, because they doubt their value in the evaluation and feel the effort of uploading the details isn’t justified. It is also possible that applicants lack the required documentation to support these activities. While uploading TAPs in SIGEVA requires attaching documents, these same activities can be briefly mentioned in the CV without any attachments. A third explanation may be related to commercial or institutional concerns. SIGEVA asks for detailed information, such as the amount of money involved and the names of all parties, whereas the CV allows for more discretion. In some cases, the activity may not have been formally approved by the institution, and the candidate prefers not to draw too much attention to it.

In any case, it is worth noting that reviewers often notice these dissonances. Their responses, however, are not uniform. Some express frustration with the inconsistencies and lack of coherence in the application, while others take a more constructive approach, using the information found in these alternative sources to better understand—and at times even emphasize—the applicant’s profile.

Regarding “technological, organizational, and socio-community developments” and “services,” the applicant does not provide information. However, in the justification section, they state that they have developed a service (sports pharmacology), that this service keeps their laboratory running, that they do outreach, and have participated in organizing scientific events. I highlight that the sports pharmacology service is of high impact for the production system. (395)

Questionable Originality

A key distinction evaluators seek to make regarding TAPs is the level of originality involved. Their main concern is differentiating activities that entail novel developments grounded in original knowledge from those that are more routine tasks or services. The former are viewed positively, especially when researchers and universities are actively involved. In contrast, the latter are seen as activities that could be performed by professionals or independent firms, raising concerns that the university might be ‘competing with its own graduates.’ Moreover, in these cases, there is often suspicion that the researchers’ motivation is driven more by immediate personal or institutional financial gain than by intellectual challenge or the creation of truly innovative products or services. The following quotes illustrate these perspectives:

It appears that only the STAN related to the company XXXX involves creative work; the rest seem to be routine tasks of sizing, advising, and testing. (156)

It seems to be an instrument repair service that could be performed by a company in the socioproductive sector or an engineer hired by the SME. The impact does not appear to be high, but there is no data on the SME’s billing, who the clients are, or the impact the faulty device has on the company’s productivity. (201)

The work consisted of a visual inspection, accompanied by sampling and subsequent physicochemical analysis of the structure. This technical report evaluates the structural safety of a church, however it does not introduce any technique that involves innovation or an increase of knowledge in the area. (413)

Assistant Researcher XXX reports two advisory tasks on molecular biology techniques to a company, but the certificates presented do not clarify whether the work was advisory or service provision, as implied by the attached certificates. (414)

Another feature that evaluators negatively observe relates to TAPs that are not clearly connected to the applicant’s “official” line of work, as shown in the following quotes:

The researcher participated as main author for only one technical report, which aimed to evaluate weight loss in dogs and cats following a dietary manipulation with a commercial food. The topic is not directly related to her work plan. The researcher has no other technological contribution or input. (327)

The researcher participated in nine audit and technical service projects. Of these, two are partially related to his research topic. It is considered that his training and specialization allowed him to contribute to these services. However, I do not consider that he has transferred results from his research through them. (384)

In these cases, it is implied that the researchers’ motivation for diverging from their main area of work stems more from a commercial interest in supplementing their income through additional activities. Although no explicitly negative judgment is stated, evaluators seem to suggest that such activities should not be given significant weight when assessing the applicant’s research trajectory. In this context, simply accumulating low-complexity TAPs may be viewed more negatively than having none at all, as it blurs the applicant’s profile as an innovator and researcher, leaning instead toward a purely professional role. However, the exact criteria for what constitutes creative work or innovation remain unclear (Derrick and Samuel 2016), especially since many of these activities can also be positively valued for their relevance to the productive sector, as we will discuss below.

Social and Productive Relevance

When reviewing TAPs, evaluators most frequently highlight their potential relevance to both productive and social spheres. These activities are valued as a channel through which research can exert a more direct impact on the economy. However, the ways in which TAPs connect to impact are highly varied, and evaluators lack consensus on how to assess this dimension beyond broad statements regarding the social or productive significance of the topic.

One initial point of analysis concerns the temporal dimension of impact. Career evaluations are, by nature, ex post processes, in which researchers present detailed accounts of their past activities, and decisions are made regarding their eligibility for promotion within the career system. However, evaluators often comment on the applicant’s future potential as well. Consequently, two temporal dimensions coexist in the evaluation: one focused on assessing past accomplishments, and another centered on estimating potential future impact. This duality is illustrated in the following three quotes: the first refers to completed work, while the latter two highlight potential future impact:

The researcher presents a satisfactory record of technology transfer activities, with noteworthy efforts to work jointly with the productive sector toward high-impact applications at a global level, such as smart electronics applied to energy distribution networks, power systems in electric vehicles, and highly efficient energy storage and management systems in autonomous systems. (99)

Given that this is an area with a direct link to concrete interventions that improve livestock health— and thus human disease prevention—and that the researcher shows a deep knowledge of the subject, I would encourage activities related to technology or knowledge transfer that help accelerate the development of such interventions and/or diagnostic methods. (270)

The academic and technological impact that will stem from the research lines developed by the applicant will be highly significant both for the health of our dairy herds and for the socio-economic sectors linked to this activity. (74)

As illustrated in the last two quotes, evaluators sometimes adopt a formative tone, suggesting that given the potential significance of the research, it would be desirable for the researcher to become more directly involved in translating their findings into products or activities relevant to socioeconomic sectors. Potential or actual counterparts are also frequently mentioned to emphasize the importance of technological activities for the country’s economy. In this context, private companies and agricultural producers are most often highlighted, with public companies receiving somewhat less attention.

It is generally clear that the research line aims to address concrete problems with local impact. There is clear linkage with the company INVAP, and with other institutions such as CONAE, YPF, UNLP, Instituto Balseiro, etc. (20)

His experience as a researcher within a company is also important, as it undoubtedly brought him closer to the realities, needs, and particularities of the productive sector, which is highly valuable for promoting interaction with firms and for channeling research findings into the productive environment. (138)

In some cases, critical comments are also made about the counterparts, suggesting they are not ready or willing to engage with researchers: “These new tools are still not easy for companies to adopt, so transfer remains limited due to this factor.” (359)

Some evaluators go beyond noting economic relevance to emphasize the alignment of TAPs with national science and technology policy objectives, particularly those focused on developing strategic sectors and boosting high value-added exports:

Regarding the applicability and potential for transfer of the results, the development of systems to monitor radiation levels is of vital importance for Argentina. Specifically, within the framework of the “Argentina Innovadora 2020” Strategic Plan, this proposal directly contributes to the Strategic Area of Electronic Components, Medical Applications, Bioengineering, and Environment. (177)

It is very important to once again highlight the direct transfer of these inventions to society and the productive sector, as well as the possibility of generating genuine foreign currency inflows for the country through the sale of equipment and services of extremely high added value. (226)

Linkage with Publications

Finally, a crucial aspect frequently raised in evaluations is the relationship between scientific productivity and involvement in technological activities. This topic features prominently in ongoing debates about academics’ engagement in technology-related work. The following quotes illustrate the range of perspectives on this issue:

I believe that the work done in this area and what has been accomplished so far is very significant and constructive for their career. It may have had a negative impact on scientific publications, but it has had a positive impact on the environment where the research is conducted, in a field that undoubtedly has absolute regional importance. (78)

This aspect is very important because, beyond scientific publications, this young assistant researcher demonstrates technical capacity to engage in solving problems that arise from the demands of the productive sector. (241)

I am surprised that Dr. XXX states in his report that no publications are expected from the project, as it seems to be a complex task that could require novel methods. (388)

In the first two quotes, evaluators support the applicants by justifying a potentially lower number of publications due to their engagement in activities closely connected to societal needs. In contrast, the third quote criticizes an applicant who preemptively claims that a technological project will not result in any publications.

Within this debate, some argue that involvement in innovation-related activities naturally leads to fewer publications, as these activities follow different quality criteria and often require confidentiality. Others contend that there is always some portion of the work that can be published and suspect that if no publications arise, it may be due to a lack of sufficient originality.

Discussion

The analysis of the materials allows us to extract several useful insights regarding how technological and applied products are evaluated, as well as the difficulties and challenges that arise in this process. First, we find that most of the literature on the assessment of applied research focuses on the evaluation of social impact, often overlooking TAPs. While these two domains may appear similar at first glance, in practice they demand distinct evaluation approaches. Unlike ex ante impact assessments commonly used in grant peer review, ex post assessments of social impact typically require the involvement of external stakeholders and consideration of medium- to long-term outcomes. This type of multi-actor evaluation is resource-intensive and remains uncommon in individual career assessments, especially in developing countries, where limited budgets and institutional capacity make such approaches particularly difficult to implement. In contrast, the evaluation of TAPs, also conducted ex post, tends to be framed as part of routine performance reviews (Genshaft et al. 2016). This was true in our case study, where external reviewers were expected to assess TAPs much like other standardized outputs, such as publications, often without adequate guidance or consensus on how to do so. Similar challenges were documented by Derrick and Samuel (2016) in their study of the REF, where reviewers expressed discomfort and uncertainty in evaluating societal impact due to the absence of prior experience or methods of benchmarking this measure.

Second, our analysis revealed several procedural issues that interfere with the evaluation of TAPs, particularly related to the quality and consistency of the documentation submitted by applicants. A recurring concern among reviewers was the lack of detailed, verifiable information necessary to support a sound assessment. Although SIGEVA allows for the inclusion of supporting documents—such as agreements, certificates, or technical reports—these were often absent or insufficiently detailed.

This issue becomes especially clear when compared to the evaluation of scientific publications. In that case, providing the full text of the article and a DOI is usually sufficient: evaluators can independently verify the work and judge its originality and significance. In contrast, many TAPs involve commercial partners or confidential data, and may not be publicly accessible, making verification difficult. As a result, evaluators are left to base their assessments solely on what the applicant submits, which often lacks the rigor or completeness needed.

Moreover, we observed frequent dissonance between sources of information. Technological activities may be mentioned in the CV or the narrative statement but omitted from the formal SIGEVA database where evaluators expect to find them. This inconsistency forces reviewers to “dig” through different parts of the file, trying to reconstruct the applicant’s trajectory—a process that many find frustrating. A possible explanation for this dissonance is that many technological activities involve consultancy fees, or royalties that may not have been officially reported to the institution. As a result, applicants might feel uncertain about whether to include these activities in formal sections of the platform, leading to their appearance only in more informal parts of the application.

These discrepancies introduce “noise” into the evaluation process: instead of a clear and coherent representation of the applicant’s work, reviewers encounter a fragmented and sometimes contradictory account. While these issues may initially appear to be technical flaws in the organization of the evaluation process (“unproductive variability” in terms of Oxley and Gulbrandsen 2025), they may also point to deeper systemic problems. For instance, dissonance between sources can reflect researchers’ need to generate additional income due to inadequate salaries, combined with bureaucratic or cumbersome procedures for formally reporting such activities.

Third, reviewers also express uncertainty and mistrust regarding the originality and relevance of TAPs. In several cases, they question whether the activities described truly represent innovation or simply routine professional services. Without clear criteria for what constitutes an original or impactful TAP, assessments become highly subjective. Some reviewers imply that these activities are motivated more by personal financial interests than by scientific goals, particularly when the TAPs fall outside the applicant’s main research line. Others, however, recognize their potential importance for the productive sector and socioeconomic development. They also express frustration in cases where applicants do not appear to “play fair” and present low-complexity activities as part of their R&D-related tasks, maybe just to tick another box (Wróblewska 2021).

Fourth, as with the issue of originality, reviewers also differ significantly in how they approach other fundamental dimensions of evaluation—particularly the temporal horizon of assessment and the relationship between technological outputs and scientific publications. Regarding the temporal dimension, some evaluators adopt a strict ex post perspective, focusing only on completed TAPs, while others are more flexible and consider the potential future contributions of ongoing research lines, even when these have not yet resulted in tangible outputs. In such cases, reviewers may “rescue” a candidate who lacks declared TAPs by valuing the expected downstream impact of their work. Similar divergence appears in the way reviewers connect TAPs to academic publications. While some acknowledge that deep involvement in applied or technological activities may come at the cost of publishing less, others view this as a weak justification and argue that at least part of the work should always be publishable. These discrepancies suggest an implicit hierarchy in how most reviewers view outputs: publications are seen as the core currency of scientific credibility in scientific career (Cañibano et al., 2024), while TAPs are considered secondary—valuable, perhaps, but only once the standard benchmarks of academic productivity have been met.

Fifth, the variability we observed in the evaluation of TAPs can be related to the distinction proposed by Oxley and Gulbrandsen (2025) between productive and unproductive variability in peer review. According to their framework, productive variability arises when differences in evaluation stem from the reviewers’ disciplinary expertise or divergent, yet legitimate, professional perspectives. In contrast, unproductive variability is linked to flaws in the design or implementation of the evaluation process—such as unclear guidelines or inadequate documentation—that hinder fair and consistent assessment.

Our findings suggest the presence of both types. On the one hand, cases of insufficient documentation exemplify unproductive variability. These issues interfere with the reviewers’ ability to form a sound judgment and could be mitigated through improved institutional practices, such as clearer instructions about what evidence to include and how to ensure consistency across sections of the application. On the other hand, disagreements about what constitutes originality, how to weigh future versus current outputs, or whether publications should take precedence over technological work reflect productive variability. These are not administrative failures but rather points of interpretative flexibility rooted in different understandings of what academic merit entails. While Oxley and Gulbrandsen emphasize disciplinary expertise as the main source of productive variability, our study suggests that in the Argentine context, reviewers inevitably also draw on personal normative views of what should count in an academic career.

Finally, despite the increasing attention in research assessment reform to broaden what is valued in academic careers, TAPs still appear to carry limited weight in actual evaluation practice. Our lexicometric analysis of reviewer reports showed that, even in applied fields, TAPs do not form a prominent “lexical world” within the broader discourse of evaluation. Other dimensions—particularly publications, training and mentoring activies, and the projected development of the applicant’s research—receive significantly more attention.

Conclusions

As movements to reform research assessment increasingly promote the recognition of diverse contributions, our study shows that evaluating technological and applied products remains a particularly challenging area. TAPs differ significantly from traditional outputs not only in form, but also in how they can—or cannot—be evaluated within standard peer review frameworks.

Our findings suggest that institutional ambiguity and variability in reviewer criteria complicate the fair evaluation of TAPs. Unlike publications, which can be more readily validated through established norms, TAPs often demand more intensive engagement from reviewers, requiring context-specific interpretation and a deeper understanding of application, intent, and outcome. This reinforces the idea that a “one-size-fits-all” evaluation model risks overlooking the value of applied research.

We argue that TAPs should be analyzed and assessed distinctly from broader concepts like social impact. While related, they refer to different stages in the knowledge appropriation process and thus require tailored evaluation approaches. While some of the issues identified here might be addressed through methodologies developed for assessing social impact—such as narrative case studies and multi-actor reviews—these approaches are resource-intensive and complex to implement. In developing countries like Argentina, where evaluation systems operate with limited budgets and institutional capacity, importing such models is often unfeasible.

In this context, micro-reforms within existing frameworks offer a more pragmatic route. These could include clearer definitions of what constitutes a TAP, more explicit expectations around documentation, and guidance for evaluators on how to assess factors such as originality, relevance, or researcher involvement. Yet while such procedural improvements may reduce some of the technical obstacles, they will not resolve the deeper tensions —such as conflicting views on merit or the marginal status of applied research. As with any change to research assessment practices, reforms must be discussed, validated, and supported by disciplinary communities, whose interpretive authority and expertise remain central to the process.

References

Abma-Schouten, Rebecca, Joey Gijbels, Wendy Reijmerink, and Ingeborg Meijer. 2023. Evaluation of research proposals by peer review panels: broader panels for broader assessments? Science and Public Policy, 50(4): 619-632. https://doi.org/10.1093/scipol/scad009

Alperin, Juan P., Carol, Muñoz Nieves, Lesley A. Schimanski, Gustavo E. Fischman, Meredith T. Niles And Erin C. Mckiernan. 2019. How significant are the public dimensions of faculty work in review, promotion and tenure documents? ELife, 8, e42254.Balazs, Aczel, AnnSophie Barwich, Amanda B. Diekman, Ayelet Fishbach, Robert L. Goldstone, Pablo Gomez, Odd Erik Gundersen, Paul T. von Hippel, Alex O. Holcombe, Stephan Lewandowsky, Nazbanou Nozari, Franco Pestilli, and John P. A. Ioannidis. 2025. The present and future of peer review: Ideas, interventions, and evidence. Proceedings of the National Academy of Sciences, 122(5), e2401232121. https://doi.org/10.1073/pnas.2401232121

Benneworth, Paul, Julia Peñuela. 2019. Ex ante evaluation of societal impact in research: Towards a conceptually-based reflection. Working paper. Online: https://hvlopen.brage.unit.no/hvlopen-xmlui/handle/11250/2631353.

Bornmann, Lutz. 2013. What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for information science and technology, vol. 64, no 2: 217-233. https://doi.org/10.1002/asi.22803 Bouwma-Gearhart, Jana, Cindy Lenhart, Rich Carter, Karl Mundorff, Holly Cho and Jessica Knoch. 2021. Inclusively recognizing faculty innovation and entrepreneurship impact within promotion and tenure considerations. Journal of Open Innovation: Technology, Market, and Complexity, 7(3), 182.Bruhn, John G. 2008. ‘Value Dissonance and Ethics Failure in Academia: A Causal Connection?’, Journal of Academic Ethics, 6: 17–32.

Bouwma-Gearhart, Jana, Cindy Lenhart, Rich Carter, Karl Mundorff, Holly Cho, and Jessica

Knoch. 2021. Open Innov. Technol. Mark. Complex, 7(3), 182; https://doi.org/10.3390/joitmc7030182

Cañibano, Carolina, Richard Woolley, Eric J. Iversen, and Carmen Corona-Sobrino. 2024. The state-of-the-art of research on science research careers. In Handbook of MetaResearch, ed. Oancea Alis, Gemma E. Derrick, Nuzha Nuseibeh and Xin Xu, 293-308.

Edward Elgar Publishing. https://doi.org/10.4337/9781839105722.00032

CLACSO – Consejo Latinoamericano de Ciencias Sociales. 2022. Declaración Una nueva evaluación académica y científica para una ciencia con relevancia social en América Latina y el Caribe. Foro Latinoamericano de Evaluación Científica. https://bibliotecarepositorioclacso.edu.ar/bitstream/CLACSO/169563/1/ Declaracion-CLACSO-FOLECversion-extendida.pdf. Accessed 15 September 2024.

CONICET – Consejo Nacional de Investigaciones Científicas y Técnicas. 2023. Sitio web oficial CONICET en Cifras. https://cifras.conicet.gov.ar/publica/grupografico/showpublico/36

Davies, Catherine and Holly Ingram. 2025. Sceptics and champions: participant insights on the use of partial randomization to allocate research culture funding. Research Evaluation, 34, rvaf006. https://doi.org/10.1093/reseval/rvaf006

Derrick, Gemma E. and Gabrielle Samuel. 2016. The evaluation scale: Exploring decisions about societal impact in peer review panels. Minerva, 54, 75-97. https://doi.org/10.1007/s11024-016-9290-0

Derrick, Gemma E. and Gabrielle Samuel. 2017. The Future of Societal Impact Assessment Using Peer Review: Pre-evaluation Training, Consensus Building and Inter-reviewer Reliability. Palgrave Communications, 3: 17040. https://doi.org/10.1057/palcomms.2017.40

Donovan, Claire. 2007. The qualitative future of research evaluation, Science and Public Policy, Volume 34, Issue 8, Pages 585–597, https://doi.org/10.3152/030234207X256538

Donovan, Claire. 2011. State of the art in assessing research impact: introduction to a special issue, Research Evaluation, Volume 20, Issue 3: 175–179, https://doi.org/10.3152/095820211X13118583635918

DORA, San Francisco Declaration on Research Assessment. 2013. Declaración. https://sfdora.org/read/

Fecher, Benedikt and Marcel Hebing. 2021. How do researchers approach societal impact?. PLoS One, 16(7), e0254006. https://doi.org/10.1371/journal.pone.0254006

Friesike Sascha, Leonhard Dobusch, and Maximilian Heimstädt. 2022. Striving for societal impact as an early-career researcher: Reflections on five common concerns. In Organizing for societal grand challenges, edited by Gümüsay Ali Aslan, Marti Emilio, Trittin-Ulbrich Hannah and Wickert Christopher: 239-255, Bingley, Emerald Publishing Limited.

Genshaft, Judy, Jonathan Wickert, Bernadette Gray-Little, Karen Hanson, Richard Marchase, Peter E. Schiffer, and Michael Tanner. 2016. Consideration of technology transfer in tenure and promotion. Technology & Innovation, 17(4): 197-204.

Hicks, Diana. 2012. Performance-based university research funding systems. Research policy, 2012, vol. 41, no 2: 251-261. https://doi.org/10.1016/j.respol.2011.09.007 Kolarz Peter, Anete Vingre, Aaron Vinnik, Antonio Neto, Carlos Vergara, Claudia Obando Rodriguez, Kalle Nielsen, and Laura Sutinen. 2023. Review of Peer Review. Final Report. https://www.bmj.com/content/349/bmj.g4145.full. Accessed 20 de April 2025.

Langfeldt Liv, Maria Nedeva, Sverker Sörlin, and Duncan A. Thomas. 2020. Co-existing notions of research quality: A framework to study context-specific understandings of good research. Minerva, 58(1): 115-137. https://doi.org/10.1007/s11024-019-09385-2

Luo Junwen, Lai Ma, and Kalpana Shankar. 2021. Does the inclusion of non-academic reviewers make any difference for grant impact panels?. Science and Public Policy, 2021, vol. 48, no 6: 763-775. https://doi.org/10.1093/scipol/scab046

Ma Lai and Rachel Agnew. 2022. Deconstructing impact: A framework for impact evaluation in grant applications. Science and Public Policy, 49(2), 289-301. https://doi.org/10.1093/scipol/scab080

Ma, Lai, Junwen Luo, Thomas Feliciani, and Kalpana Shankar. 2020. How to evaluate ex ante impact of funding proposals? An analysis of reviewers’ comments on impact statements. Research Evaluation, 29(4), 431-440. https://doi.org/10.1093/reseval/rvaa022

Melkers, Julia, Woolley Richard, and Kreth, Quintin. 2023. Research Funding and Scientific Careers. En Handbook of Public Funding of Research, edited by Lepori Benedetto, Jongbloed Ben, Hicks Diana. Edward Elgar Publishing, 301-321.

Moher, David, Lex Bouter, Sabine Kleinert, Paul Glasziou Virginia Barbour, Anne-Marie Coriat, Mai Har Sham, Nicole Foeger, Ulrich DirnaglI. 2020) The Hong Kong Principles for assessing researchers: Fostering research integrity. PLoS Biol 18(7): e3000737. https://doi.org/10.1371/journal.pbio.3000737

Molas-Gallart Jordi, Ammon Salter, Pari Patel, Alister Scott, and Xavier Duran. 2002. Measuring Third Stream Activities. Final Report, Russell Group of Universities SPRU, University of Sussex, Vol. 81. Brighton, UK.

Morgan-Thomas, Anna, Serafeim Tsoukas, Adina Dudau, and Paweł Gąska. 2024. Beyond declarations: Metrics, rankings and responsible assessment. Research Policy, 53(10), 105093. https://doi.org/10.1016/j.respol.2024.105093

Niembro, Andrés and Fernando Svampa 2024. Territorial Inequalities and (de) Concentration of Public Investment in Science: A Study on CONICET (Argentina) and the Tensions Between Academic Excellence and Equity. Minerva, p. 1-35. https://doi.org/10.1007/s11024-024-09550-2

Norn, Maria Theresa, Kaare Aagaard, Josefine Bjørnholm, and Andreas Kjær Stage. 2024. Funder strategies for promoting research addressing societal challenges: thematic, impact, and collaboration targeting. Science and Public Policy, 51(5): 910-922. https://doi.org/10.1093/scipol/scae033

Oxley, Kristin and Magnus Gulbransen. 2025. Variability and negligence: grant peer review panels evaluating impact ex ante. Science and Public Policy,, vol. 52, no 2, p. 254-268. https://doi.org/10.1093/scipol/scae081

Pekkola, Elias and Tarus Siekkinen (Eds.). 2024. Tenure Tracks in European Universities: Managing Careers in Academia. Cheltenham: Edward Elgar Publishing Peruginelli.

Peruginelli, Ginevra and Janne Polonen. 2023. The legal foundation of responsible research assessment: An overview on European Union and Italy. Research Evaluation vol. 32, no 4: 670-682. https://doi.org/10.1093/reseval/rvad035

Pontika, Nancy, Thomas Klebel, Antonia Correia, Hannah Metzler, Petr Knoth and Tony Ross-Hellauer. 2022. Indicators of research quality, quantity, openness, and responsibility in institutional review, promotion, and tenure policies across seven countries. Quantitative Science Studies; 3 (4): 888–911. https://doi.org/10.1162/qss_a_00224

Power, Michael. 2015. How accounting begins: Object formation and the accretion of infrastructure. Accounting, organizations and society, vol. 47: 43-55. https://doi.org/10.1016/j.aos.2015.10.005

Ramos Vielba, Irene, Duncan Thomas, and Kaare Aagaard. 2022. Societal targeting in researcher funding: An exploratory approach. Research Evaluation, vol. 31, no 2: 202-213. https://doi.org/10.1093/reseval/rvab044

Reale, Emanuela, Dragana Avramov, Kubra Canhial, Claire Donovan, Ramon Flecha, Poul Holm, Charles Larkin, Benedetto Lepori, Judith Mosoni-Fried, Esther Oliver, Emilia

Primeri, Lidia Puigvert, Andrea Scharnhorst, Andràs Schubert, Marta Soler, Sàndor Soòs, Teresa Sordé, Charles Travis, René Van Horik

Reinert Max. 1986. Un logiciel d’analyse lexicale: ALCESTE. Les cahiers de l’Analyse des Données.11(4) : 471-84. https://www.numdam.org/item/CAD_1986__11_4_471_0.pdf

Reinert, Max. 1990. Alceste une méthodologie d’analyse des données textuelles et une application: Aurélia De Gerard De Nerval. Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique, 26(1), 24-54. https://doi.org/10.1177/075910639002600103

Repiso and Montero, 2024. A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation, Volume 27, Issue 4: 298–308, https://doi.org/10.1093/reseval/rvx025.

Rice, Danielle, Hana Raffoul, Joan Ioannidis, and David Moher. 2020. Academic criteria for promotion and tenure in biomedical sciences faculties: cross sectional analysis of international sample of universities, BMJ. https://doi.org/10.1136/bmj.m2081

Romano, Silvina, Valeria Car, and Valentina Locher. 2020. Reflexiones metodológicas acerca del uso de estadísticas de datos textuales para el análisis de los discursos. RevCom. http://sedici.unlp.edu.ar/handle/10915/107660

Ross-Hellauer Tony, Thomas Klebel, Petr Knoth, and Nancy Pontika. 2023. Value dissonance in research(er) assessment: individual and perceived institutional priorities in review, promotion, and tenure, Science and Public Policy, Volume 51, Issue 3, 337–351, https://doi.org/10.1093/scipol/scad073

Roumbanis, Lambros. 2019. Peer review or lottery? A critical analysis of two different forms of decision-making mechanisms for allocation of research grants. Science, Technology, & Human Values, 44(6), 994-1019. https://doi.org/10.1177/0162243918822744

Rovelli, Laura, Pablo Vommaro. 2024. La evaluación académica y de la investigación en la transición hacia la ciencia abierta en el ecosistema científico-universitario del Perú. https://doi.org/10.15381/dds.n13.28655

Rovelli, Laura. 2024. Debates actuales y reformas en curso en la evaluación responsable de la investigación en América Latina y el Caribe. Oficina Regional de UNESCO en Montevideo y CLACSO: Montevideo. https://repositorio.ciedupanama.org/handle/123456789/399

Rushforth, Alexander and Björn, Hammarfelt. 2023. The rise of responsible metrics as a professional reform movement: A collective action frames account. Quantitative Science Studies, vol. 4, no 4, p. 879-897. https://doi.org/10.1162/qss_a_00280

Rushforth, Alexander. 2025a. Research assessment reform as collective action problem: Contested framings of research system transformation. Minerva, 1-21. https://link.springer.com/article/10.1007/s11024-025-09573-3

Rushforth, Alexander. 2025b. Beyond Impact Factors? Lessons from the Dutch Attempt to Transform Academic Research Assessment. Preprint SocArXiv, https://doi.org/10.31235/osf.io/hza3c_v3

Schweiger, Gerald, Adrian Barnett, Peter van den Besselaar, and Stijn Conix. 2024. The costs of competition in distributing scarce research funds. Proceedings of the National Academy of Sciences, 121(50), e2407644121. https://doi.org/10.1073/pnas.240764412

Seeber, Marco, Svege, Ida, and Hesselberg, Jan Ole. 2024. Improving the efficiency of research proposals evaluation: A two-stage procedure. Research Evaluation, rvae020. https://doi.org/10.1093/reseval/rvae020

Smit, Jorrit, and Laurens Hessels. 2021. The production of scientific and societal value in research evaluation: a review of societal impact assessment methods. Research Evaluation, vol. 30, no 3: 323-335. https://doi.org/10.1093/reseval/rvab002

UNESCO 2021. Recomendación sobre la Ciencia Abierta. https://unesdoc.unesco.org/ark:/48223/pf0000378841_spa

Urbanc, Mimi, Stefan and De Jong. 2025. Navigating Societal Impact: Strategic Management in Horizon 2020 SSH Projects. Minerva, p. 1-24. https://link.springer.com/article/10.1007/s11024-024-09566-8

Vasen, Federico. 2018. The ‘ivory tower’ as safe bet: Science policies and research evaluation in Mexico. Education Policy Analysis Archives, 26, 95. https://doi.org/10.14507/epaa.26.3594

Vasen, Federico. 2025. A delicate balance: responsible evaluation, tenure review, and the pursuit of institutional innovation, Science and Public Policy, 2025, scaf003, https://doi.org/10.1093/scipol/scaf003

Vasen, Federico and Sierra Pereiro, Miguel. 2022. “The Hardest Task”—Peer Review and the Evaluation of Technological Activities. Minerva 60, 375–395. https://doi.org/10.1007/s11024-022-09461-0

Williams, Kate, and Grant, Jonathan. 2018. A comparative review of how the policy and procedures to assess research impact evolved in Australia and the UK. Research Evaluation, 27(2), 93-105. https://doi.org/10.1093/reseval/rvx042

Wilsdon, James. 2016. The Metric Tide: Independent review of the role of metrics in research assessment and management. Research England. https://doi.org/10.4135/9781473978782

Wróblewska Marta. 2021. Research impact evaluation and academic discourse. Humanit Soc Sci Commun 8, 58. https://doi.org/10.1057/s41599-021-00727-8

Wróblewska, Marta. 2025. One size fits all? A comparative review of policy-making in the area of research impact evaluation in the UK, Poland and Norway, Research Evaluation, 34, rvaf010, https://doi.org/10.1093/reseval/rvaf010