Published at MetaROR

November 16, 2026

Table of contents

Cite this article as:

Clark, R., Duncan, L., Hsing, P.-Y., Jacobs, N., & Strang, R. (2026). Mechanisms for Research Culture Change within Higher Education Institutions: METEOR study report. Zenodo. https://doi.org/10.5281/zenodo.20488064

Mechanisms for research culture change within higher education institutions: METEOR study report

Robbie Clark1ORCID, Lorna Duncan1ORCID, Pen-Yuan Hsing1ORCID, Neil Jacobs1,3ORCID, Rosalind Strang1ORCID

1 University of Bristol, School of Psychological Science
3 UK Reproducibility Network

Originally published on June 1, 2026 at: 

Abstract

Meta-research has investigated various concerns around ‘research culture’ in Higher Education Institutions (HEIs). Despite sector-wide efforts to improve institutional cultures, evidence suggests that such initiatives are seldom evaluated. This study explored how meta-research informs decisions affecting research culture within English HEIs, along with the mechanisms which enable or hinder meta-research from impacting decisions. Interviews were conducted with 14 institution-level ‘decision-makers’ from 11 diverse English HEIs, plus 7 meta-researchers in the UK. Data were analysed using deductive content analysis.

Participants discussed how research has greatly informed HEI strategic directions to improve research culture. This primarily happened indirectly, brought into decisions by advocates and extrinsic forces. Some institutions have also conducted local research into their own cultures. Nevertheless, innovations to improve research culture are less evidence-based and infrequently evaluated. Participants identified several mechanisms by which meta-research could impact on such decisions. Engagement, consultation, and co-creation could make meta-research more impactful, whereas career stage, prestige, and personal connections could either enable or hinder researchers from achieving impact. Generating sufficiently persuasive evidence to inform impactful decisions was an additional barrier to meta-research impact. Lastly, findings which reflected poorly on institutions or individuals could reduce uptake or sharing of results.

This research highlights the need to more rigorously evaluate the impact of research culture reforms. Given the scale of institutional changes, the research encourages sector-wide collaboration and sharing of findings, suggesting ways in which researchers and institutions can seek to maximise the impact of meta-research on research culture.

Introduction

“Research Culture encompasses the behaviours, values, expectations, attitudes and norms of our research communities.” – Royal Society

The 21st century has seen increased attention towards many problematised aspects of research culture (RC), within both United Kingdom (UK) Higher Education and internationally. This attention has been catalysed and compounded by a host of interconnected issues. For example, concerns around the replicability and trustworthiness of published scientific literature have provoked questions about research integrity, prompting a movement towards greater openness and transparency (Aguinis et al., 2017; Allen & Mehler, 2019; Baker, 2016; Baxter & Burwell, 2017; Byington & Felps, 2017; Ciubotariu & Bosch, 2022; Cook et al., 2018; Open Science Collaboration, 2015). This has led many to examine the systemic, cultural forces which have allowed or encouraged untrustworthy and opaque research.

It has been suggested that centralised governance, managerialism, and a singular focus on pursuing metrics have significantly contributed to issues of research integrity (Dienes, 2023; Erickson et al., 2021; Hicks et al., 2015; UKRI, 2015; Wellcome, 2020). Inter-competition and the over-measurement of researchers perversely incentivise problematic behaviours, such as publishing quantity over quality, hiding negative results, and other ‘questionable’ research practices (Bruton et al., 2020; Fanelli, 2010, 2012; John et al., 2012; Nature, 2010; Rosenthal, 1979; Ware & Munafo, 2015; Xie et al., 2021). Such issues are truly systemic, as it is not only individuals but also institutions for whom competition is necessary to both survive and secure funding (Musselin, 2018).

Many other cultural problems have been spotlighted and included under the broad umbrella of necessary cultural reforms. Significant among these are concerns around equity, diversity, and inclusion (EDI) within academia, and the need to improve access to and participation in research. (Dewidar et al., 2022; Hattery et al., 2022). UK Research and Innovation (UKRI) – the primary public research funding body in the UK – has also suggested potential investments to improve: career trajectories for non-academic pathways; bullying and harassment; leadership skills; cross-sector collaboration; public dialogue, and community-led research (UKRI, 2022).

Within this context, many UK Higher Education institutions (HEIs) have been expending considerable efforts to improve RC, supported in England by Research England funding (UK Reproducibility Network, 2023; UKRI, 2022). Funders have also been investing heavily in RC improvements (Lewis-Wilson et al., 2023; RoRI, 2024; Wellcome, 2020). This work is important for several reasons. First, it should enable researchers and allied staff to excel in their work and to be content in their workplace. This should foster collaboration and healthy competition, and promote the inclusion of diverse perspectives into research on an equitable basis. Ultimately, this should yield higher quality and more impactful research. The value placed on improving institutional cultures has recently been indicated in the increased attention being paid to ‘people, culture and environment’ in the forthcoming Research Excellence Framework 2029 (REF29) (REF, 2024), a major sector-wide research assessment of HEIs in the UK.

Much of the research that has helped to identify and investigate problems with RC could be classified as meta-research (i.e., research-on–research, whose topics include the methods and practices of research and the environments within which it occurs). One might therefore expect meta-research to be employed in the development and evaluation of solutions to the problems it has identified, creating a complete translational framework (Hardwicke et al., 2019). Nevertheless, a recent study indicated that the majority (~71%) RC initiatives developed within the UK have not been evaluated (UKRI, 2024b).

Given the diversity of UK HEIs, strengths and concerns around institutional cultures are varied. Indeed, whether research culture is something to be improved upon if contingent on the amount of research activity within a given institution. Consequently, the relevance of and barriers to the use of research in supporting decision making are likely to be multi-faceted. Factors such as size, research-intensity, organisational structure, and study disciplines could all affect whether and how research could inform decisions about RC. Similarly, barriers may be related to institutions’ abilities to identify and utilise relevant research, as well as arising from aspects of the research itself (e.g., contextual applicability, quality, etc).

Consequently, this study set out to explore how research informs HEI decisions around RC, with the ultimate aim of identifying barriers and enablers to the utilisation of research. Throughout, the term ‘research’ is often used interchangeably with ‘meta-research’. It is assumed that any research on RC – by virtue of studying the cultures and systems within which research occurs – is implicitly a kind of meta-research.

Methods

Methodological overview

This qualitative study used directed content analysis (Hsieh & Shannon, 2005) on data from interviews and a focus group with two demographics within UK higher education institutions (HEIs). The first demographic were institution-level decision makers (DMs) with a remit to influence RC within their institution. The second group were meta-researchers (MRs) who had conducted research relating to RC. Initial conclusions were subsequently validated and refined through a participant check workshop with (n = 7) of the participants.

Research aims

This exploratory research sought to understand:

  1. How meta-research has informed HEI decisions affecting RC in English
  2. The key mechanisms which enable and inhibit meta-research from impacting on institutional decisions affecting RC.

Participants

The researchers conducted 18 interviews and 1 focus group with 14 decision-makers and 7 meta-researchers in the UK (see Tables 1 and 2 for demographic information). Participants were distributed across 14 HEIs of varying sizes and research intensity. The decision-makers were located across 11 of those HEIs, all in England.

Participant roles have been generalised to ensure anonymity whilst retaining contextual information. For example, roles such as Dean or Vice Chancellor are referred to as senior leadership roles. Meta-researcher institutional affiliations are not stated to reduce the chance of identification. All were based within 5 of the 24 research-intensive Russell Group universities. However, two MRs are identified as being affiliated with the Research-on-Research Institute (RoRI). This is due to the unique nature of RoRI, and the contextual importance of this affiliation for analytic purposes.

Table 1. Demographic information for the decision-makers. Participant roles indicate their position held within the institution. ERC Funding 23/24 indicates the amount of Research England Enhancing Research Culture funding awarded to the institution for the academic year 23/24, split into quartiles (1=lowest, 4=highest). UoAs REF21 is an indicator of the number of ‘Units of Assessment’ submitted to the RF exercise in 2021, split into quartiles (1=lowest, 4=highest). UKRN Link states institutions’ links with the URKN, see https://www.ukrn.org/community/ for details; DM2a-c were a focus group, and DM6a and DM6b were in an interview with two participants.
ID Participant role ERC Funding 23/24 UoAs REF21 UKRN Link
DM1 Senior leadership role for Research Culture 4 4 Institutional Member
DM2a Senior leadership role for Research 3 3 Local Network
DM2b Professional services role for Research 3 3 Local Network
DM2c Professional services role for Research 3 3 Local Network
DM3 Senior leadership role for Research 3 3 Local Network
DM4 Senior leadership role for Research 3 3 Local Network
DM5 Senior leadership role for Research Culture 4 4 Institutional Member
DM6a Senior leadership role for Research 4 4 Local Network
DM6b Professional services role for Research Culture 4 4 Local Network
DM7 Senior leadership role for Research 4 4 Institutional Member
DM8 Senior leadership role for Research Culture 3 4 None
DM9 Senior leadership role for Research 1 2 None
DM10 Senior leadership role for Research 1 1 None
DM11 Senior leadership role for Research 2 3 None

 

 Table 2. Meta-researchers and their academic roles. Roles are stated very generically to reduce the risk of indirect identification.
Participant ID Role
MR1 Researcher
MR2 Researcher
MR3 Professor
MR4 Researcher
MR5 Senior researcher
MR6 Professor
MR7 Senior researcher

Sampling and recruitment

Decision-makers

Decision-makers were recruited from English HEIs with the use of a sampling matrix to ensure representation across four strata: Research Intensity; Research Focus; region within England; and links to the UK Reproducibility Network (UKRN). Research intensity was determined by the amount of Research England (a subdivision of UKRI) ‘Enhancing Research Culture’ funding received in the academic year 2023/24, as this funding was allocated in proportion to the quality-related research (QR) block funding they received, which is allocated based on REF results. Research Focus was assessed by the Units of Assessment submitted to REF2021. The UKRN is a consortium representing UK HEIs, studying and promoting best research practices. UKRN links were therefore included, as they might indicate institutional commitment to RC change. The final sample included DMs representing all regions of England, those from each quartile of ERC funding and UoAs received, and a range of UKRN affiliations.

Participants were identified from the sampled institutions through personal contacts and web searches. Starting with initial contacts, snowball sampling was used to find the most suitable individuals. Staff were considered decision-makers if they held senior leadership roles and/or were involved in decision making affecting RC. Initially, ‘influencers’ and ‘change agents’ (staff capable of influencing decisions but not directly involved in decision making) were approached for interview. However, this was amended after one interview to focus on senior leaders, due to their capacity to provide insights into institution-level decisions.

Potential participants were contacted via email, provided with information about the project, and invited to participate. Those who expressed interest were provided with additional information about the study. If no staff member from a sampled institution expressed interest, the research team selected another institution with similar characteristics from the sampling matrix. Participation was voluntary, and no incentive or reimbursement was offered.

Meta-researchers

Meta-researchers were sampled through a mixture of theoretical and snowball sampling. Sampling began with a web-search of individuals who identified as meta-researchers. These individuals’ research profiles were screened for projects relating to aspects of RC. Potential participants were added to a recruitment matrix until 50 had been gathered. Participants were then recruited through the same process as decision-makers, with the aim of ensuring that there was a diverse mix of career stages and aspects of research culture represented.

Data generation

Semi-structured interviews and a focus group were undertaken by 3 members of the research team (LD, PYH, RS) between February and May 2024. Interviews lasted 73 minutes on average, ranging from 50 to 107 minutes. All interviews except one were online, for feasibility and convenience. One interview involved two decision-makers simultaneously, at the participants’ suggestion that they could provide different perspectives. The focus group was conducted by LD and PYH with three decision-makers as a site visit. Again, the aim was to elicit more nuanced and diverse data.

Semi-structured interviews were selected to ensure that key research questions were answered, whilst allowing room to explore novel ideas. The interviews followed topic guides tailored for the two demographics (see online Supplementary Materials). Participants were sent simplified topic guides prior to their interviews, plus a table of the nine areas of RC identified by Research England as potential areas for investment (24). The aim was for participants to reflect on the breadth of RC and possible examples from their contexts.

Interviews began with an introduction to the project aims and interview goals. Working definitions of RC and meta-research were given to participants. It was made clear that different individuals and institutions would have different perspectives, and that the goal of the interview was to obtain participants’ experiences and reflections. After obtaining consent, the interviews were audio-recorded. Participants were asked to describe their current role(s) and background. The topic guide was then used flexibly to ensure key topics were addressed. All interviews were transcribed verbatim. Participant and institution names were pseudonymised during transcription. Contextually important, identifiable information (such as job titles) was not redacted.

Analysis

The content analysis was directed towards answering the two central research aims; focusing almost exclusively on the ways in which research informs HEI decisions affecting research culture, and the barriers and enablers to improving this relationship. Data were analysed by RWAC and RS using directed content analysis (Hsieh & Shannon, 2005), guided by deductive coding frameworks (see online Supplementary Materials). These were created around the questions in the topic guide, and iterated across multiple meetings. RWAC coded the data for both decision-makers and meta-researchers. RS reviewed the coding, checking for the appropriateness of the identified codes, and comprehensiveness in capturing all of the data. Initial code categories were then discussed with NJ and GD, identifying emerging themes and avenues for exploration. RWAC then developed and revised sub-themes, which were grouped around the research aims they related to. The writing process involved ongoing discussion of emerging themes with project members, as well as multiple revisions and reformulations of thematic structuring.

Results

This section is organised in relation to the Research Topic being addressed (see Table 3).

Table 3. Research Topics and Themes

Research Topic Themes
1. How evidence informs HEI decisions affecting research culture 1.1. Research into cultural ‘issues’ has indirectly shaped institutional agendas
1.2. Research about solutions to research culture issues is lacking
2. Enablers and barriers to meta-research impact on HEI decisions affecting research culture 2.1. Engagement, consultation, and co-creation can facilitate meta-research impact
2.2. Career stage, prestige, and connections can both help and hinder impact
2.3. Many challenges exist in acquiring and acting upon research evidence

(1).  How evidence informs HEI decisions affecting research culture

The following two themes explore the extent to which research and other types of evidence have informed HEIs decisions affecting RC. The first theme explores how research about cultural issues has both directly and indirectly informed institutional priorities and decisions. Theme 2 expands on how comparatively little research has informed solutions to RC – i.e., policies and initiatives implemented to improve institutional cultures.

Theme 1.1: Research into cultural ‘issues’ has indirectly shaped institutional agendas

This theme presents the factors underpinning HEI priority setting around RC, paying particular attention to the ways in which research has directly and indirectly been utilised by decision-makers. It begins by looking within institutions, focusing primarily on the roles of individuals and groups in bringing research into decisions about RC. Then, it looks across institutions at sector-wide developments in RC: how these have influenced HEI decisions, and how much of this has involved research and other types of evidence.

Interviewees reported that RC developments had often relied on motivated individuals advocating for change within receptive institutions. Pockets of activity into specific aspects of RC (e.g., EDI) were ongoing before being subsumed under “the umbrella of research culture” (DM6b). As RC became an institutional priority, many HEIs appointed senior individuals with a strategic remit to address RC. This often manifested in formalised RC groups, committees, and roles such as Dean for Research Culture.

Centralising efforts in this way helped to bring together individuals with the expertise and remit to positively influence RC. Participants felt that individuals within these groups might informally utilise knowledge of relevant research in their decision making, by virtue of their longstanding interests. In this way, meta-research evidence could indirectly “percolate in through people’s expertise… rather than explicitly” (MR7). DM2c expressed the concern that this could lead to influential senior leaders having a disproportionate influence on decisions, dictated more by their personal priorities than “evidence-based argument”.

RC groups also provided a direct means through which research could inform decisions, having become a focal point for institutional efforts to understand and improve aspects of their own culture. Interviewees reported a variety of data-gathering exercises, which could be considered ‘institutional meta-research’ into RC. Their primary goal was to understand current cultures and thereby inform future initiatives. In at least three of the sampled institutions, some evaluations had been outsourced.

“What we’re actually doing is gathering that kind of broad-based evidence to kind of understand what [our institution’s] culture looks like from that kind of bird’s eye perspective, really… that will help to inform kind of the initiatives on the ground.” – DM2c

Engagement and co-production were common features of this work. Using surveys and workshops; listening “to researchers, in terms of what the issues they feel are important and what can support them” (DM4). The other major type of institutional meta-research involved quantitative observations on internal data. For example, a research-intensive university “did some research on the diversity of grants, both applications and grant awards” (DM2a), helping to inform which aspects of RC to prioritise, as well as the creation of new policies and initiatives.

Institutional efforts to understand and improve their cultures are microcosmic of the sector-wide attentional shift towards RC, which has been heavily shaped and informed by research. Several meta-researchers referenced the significant impact of cornerstone meta-research projects, such as the “Centre for Open Science… Reproducibility Projects” (MR4) and the “Metric Tide Report” (MR6). A few decision-makers also mentioned specific, impactful projects. For example, DM6a mentioned the “TALENT Commission”, and how that had influenced their policies about research technicians. Similarly, DM10 referred to a book on EDI, grounded in academic research, which had informed their institutional policies. Some decision-makers also referred to reports and sector-wide developments on a more general level – “the reports and evidence that had come out through the big research culture kind of splurge” (DM6a).

Despite numerous references to such evidence, decision-makers nevertheless concurred that RC decisions had not been informed by research as much as they ought to be. A possible explanation for this is that it was not necessary for them to directly consult source literature, due to the “huge profile that research culture was being given through Wellcome, through the Royal Society, through the government” (DM6a).

“When funders start wanting to see stronger EDI policies and EDI more strongly within research proposals. When they want to see a greater emphasis on research culture in applications to have PGR doctoral training paths or whatever, that clearly drives university behaviour.” – DM7

Arguably, the evidential grounding for cultural changes was manifest in the priorities of influential sector bodies, such as government and research funders. Participants viewed these bodies as enormous forces for change within the sector. Consequently, it may be that research ‘percolated into’ decisions through the sector-wide dialogue, digested in a wide range of formats besides source literature. Moreover, it is possible that our interview questions did not sufficiently provoke decision-makers to interrogate the origins of their own convictions and expertise. Many decision-makers had been promoted to such senior roles because of their expertise in these areas. Therefore, the interviews may not have prompted sufficient introspection into how much research had percolated into their own arguments and decisions.

Another factor in decision making was what other institutions were doing. Others’ policies, strategies, and initiatives could serve as both positive and negative exemplars. Importantly, these provided valuable insights for historically teaching-intensive institutions who were only recently establishing a culture of research. Because their ‘research culture’ was so new and still developing, they did not report having the same problematic cultures of research-intensives. Consequently, they drew from the lessons being discussed across the sector in order “not to recreate that problematic precarious culture that a lot of other research institutes basically are based on” (DM11).

To summarise, most participants felt that most HEI decisions affecting RC did not seem influenced by specific research related to those decisions. The exception to this was ‘institutional meta-research’, investigating local cultures to inform RC priorities and initiatives. Nevertheless, various types of evidence about cultural issues have unquestionably shaped the movement across the sector, thereby indirectly informing HEI decisions to make improvements. Novel committees and specific roles to improve institutional cultures comprise individuals whose interest and advocacy is unquestionably informed by the zeitgeist. Consequently, research likely ‘percolates into’ HEI decisions through these individuals, as well as more broadly at a sector level, through funder requirements, concordats, and inter-institutional competition.

Theme 1.2: Research about ‘solutions’ to research culture issues is lacking

The previous theme concluded that strategic priorities to improve RC have often been driven, if indirectly, by research. By contrast, this theme describes how comparatively few solutions (i.e., policies and initiatives) to address cultural issues have been informed by research. The overarching conclusion is that RC initiatives typically have not been informed by research, largely because these developments are so new that “research hasn’t been done – or there’s relatively little of it” (DM1).

“I wouldn’t say published research is a very common source of insight for us just because a lot of what we’re doing feels quite new. I mean, [partially randomised allocation] has been the exception to that. There’s a nice collection of models and, some papers on that. Even though it’s not had very good take up, it has been quite long established.” – DM5

Where RC initiatives were informed by research, they either drew on established literatures, or institutional information-gathering. Research policy, science metrics, EDI, and partially randomised allocation (PRA, where funding is randomly awarded among all grant applications meeting a baseline, peer-reviewed quality threshold) were all instances where bodies of literature could be drawn on. However, such cases were the exception rather than the rule, due to the wide remit of RC, and relative novelty of many discussed improvements. Where there was a paucity of existing evidence, institutions gathered their own information. At times this could be considered ‘institutional meta-research’, whereas at others it was closer to consultation and co-production “to make sure that the relevant voices and functions are represented” (DM1).

Given the novelty and lack of evidence underpinning many RC reforms, one might expect that their effects would have been evaluated. Indeed, various meta-researchers were working on precisely that, and a few decision-makers provided examples where initiatives had been evaluated. However, some felt that previous evaluations had been of limited evidential value, primarily restricted to feedback surveys, without comparative baselines or causal methods that could assess change across time. Despite a few examples of evaluation, participants reported that it had been much more common to develop and implement innovations, rather than evaluate them.

“The evidence base isn’t as robust as it needs to be, because we’ve all focused on doing what we think are really good projects, and we’re focused much less on putting a proper, rigorous evaluation framework around about.” – DM7

This is something which decision-makers were acutely aware of. Many emphasised the importance of embedding evaluation into future initiatives, and the need to share lessons with others “so that you don’t have every institution doing the same meta-research” (DM1).

Interviewees provided few direct answers as to why evaluation had been so uncommon. During our validation workshop, two participants from very different-sized institutions mentioned how the fast turnaround to spend Research England’s ERC funding prevented some HEIs “from taking a longer-term strategic approach” (DM2c). In a small institution with sporadic funding for RC, DM10 reported that only one year to spend money prevented adequate evaluation. Considering the time it takes to recruit, one year is inadequate to get a “deep institutional understanding”. Consequently, “evaluation ends up being very rushed… and quite superficial” (DM10). DM1 provided a different experience from a larger institution, stating that some HEIs could underwrite positions for more than a year, if they expected the funding. Nevertheless, they acknowledged that the short timeframe of the funding allocations resulted in more short-term evaluation, rather than looking at more longitudinal measures.

Another cause behind the lack of evaluation might be that RC is hard to operationalise and measure. This was discussed by numerous interviewees in different ways, typically in relation to REF29. An issue with operationalisation is that lots of aspects of RC “are very intangible and tacit in nature” (DM4), which may partially explain why instances of RC evaluation have relied so much on qualitative feedback. Moreover, given how problematic metrics have contributed to the RC movement, there have been some concerns around turning RC into another metric, “another game that we have to play and tick the box for when we get to REF” (DM11). Furthermore, many of the intended cultural changes are recognised to be “much longer-term” (DM8) than can be captured within a few years.

Regardless of why evaluation has been uncommon, decision-makers perceived this as a barrier which needed to be overcome in the future. This is something that they were actively seeking to change and improve. Indeed, interviewees described numerous instances of ongoing and planned evaluation, with the goal of creating a feedback loop to inform future decisions.

Robust evaluation was also needed to generate support for RC agendas. Decision-makers perceived the importance of justifying changes with strong evidence, particularly to those who might be sceptical. One meta-researcher believed that “hard data” was particularly important when dealing with scientists, because “you can’t argue with the data” (MR3). Several participants echoed this perceived preference for quantitative data, reflecting on experiences of colleagues. A decision-maker from an arts background saw this prejudice as problematic, highlighting that data serves an instrumental purpose and can come in many forms. Fundamentally what matters is “trustworthiness and validity” (DM10). Ultimately, providing strong evidence for changes would reduce perception that reformers were merely “tinkering with the system” (MR1).

In conclusion, RC changes were often implemented without research to justify them. That is not to say that changes were unfounded, just untested. Decision-makers highlighted the importance of more robust evaluation and sharing of findings, whilst recognising that it will “take a couple of years to really feed through” (DM5). Almost every decision-maker worried about the longevity of this work in the current financial climate within HEIs. Although there is a current wave of enthusiasm, largely enabled within English HEIs by Research England funding, they questioned how they would “manage to embed these issues within the culture of the university without that funding” (DM6a).

(2).  Enablers and barriers to meta-research impact on HEI decisions affecting research culture

This section explores factors which both enabled and inhibited meta-research from impacting institutional decision making. could both enable and inhibit meta-research from impacting institutional decisions about RC. Theme 1 describes how engaging relevant communities in the research design was a significant mechanism for research impact. Theme 2 explores how career stage, prestige, and connections could function as both enablers and barriers to impact, highlighting systemic problems. Theme 3 presents several barriers to meta-research impact, all relating to qualities of the evidence.

Theme 2.1: Engagement, consultation, and co-creation can facilitate meta-research impact

“If you’re going to take research culture seriously, one of the fundamental design principles of any organisational approach to these issues has to be both listening to, involving and, you know, giving a meaningful voice to perspectives from across the institution.” – MR6

This theme describes the ways in which engagement, consultation, and co-creation enabled meta-research to be more impactful. At their core, these practices entail involving relevant communities in various parts of the research process. By their nature, RC issues are situated within systems involving a range of stakeholders who can both impact and be impacted by RC changes. Consequently, these individuals are well-placed to advise researchers on such topics as meaningful research questions, outcomes, feasibility, and pathways to impact (Domecq et al., 2014).

We interviewed two researchers from the independent Research on Research Institute (RoRI), who described their unique model of having research funders and policy makers directly involved in developing research questions and intended outputs. “[RoRI] asked the funders, they asked the policy makers: what would be useful? And it was them driving it” (MR4). In this way, they ensured that their research outputs were useful to those for whom they were intended. Moreover, this ensured predetermined pathways to impact, because policymakers already knew about the project and had considered questions around feasibility from the outset.

RoRI’s somewhat unique model has commonalities with the ‘institutional meta-research’ HEIs were conducting about RC. Lots of the institutional meta-research involved engagement; “including [relevant stakeholder] perspectives… that codesign model” (DM2b), and how changing policies might influence that. As with RoRI, pathways to impact were pre-determined, with key decision-makers being linked into the research from early stages. Several meta-researchers echoed how this type of high-profile, large-scale meta-research was more likely to have a practical impact. This is partly because it generates institutional commitment to act upon the research results from the outset.

Theme 2.2: Career stage, prestige, and connections can both help and hinder impact

Career stage, prestige, and connections of both meta-researchers and decision-makers could all influence whether meta-research impacted on RC decisions. Whilst identifying useful mechanisms to impact, this theme highlights concerns around how these same channels might amplify power imbalances between early-career and senior researchers.

Discussions with various interviewees converged on the conclusion that a major route to impact was having senior meta-researchers involved in institutional-level discussions. For example, a Professorial meta-researcher was a programme director within their institution, enabling them to implement and evaluate changes to an aspect of RC within the programme. Their positive results of their evaluation were on such a large scale that the research achieved national reach.

Several meta-researchers provided examples in which the principal investigators on their projects were very senior within their institutions. By virtue of their positions, these individuals were involved in high-level committees and decisions, as well as having connections through which to informally share findings with senior decision-makers. Through these channels, they were able to inform institutional decision making with meta-research. MR1 described their project’s principal investigator as “in a great position to get things done”, as the institutional lead on research culture. By contrast, two early career MRs felt that publishing a paper within their contractual timeframe was their primary goal, due to the pressures of short-term contracts. They chiefly relied on senior members to take research findings forward as the only members of the “project that have permanent posts” (MR1).

Additionally, a third early career meta-researcher experienced difficulties influencing institutional decisions around RC, despite actively offering their expertise. They described “unimaginable levels of prejudice” because of their career stage and corresponding lack of traditional signals of expertise, such as publication and citation counts (MR4).

“When I speak to other academics it’s like, well who are you? You’re not a professor. You’re only an early careers scholar. Where’s your ERC grant? Where’s your research team? Where’s all of this? Where’s your credentials, where are your publications and stuff.” – MR4

Combined, these experiences suggest that systemic issues of career precarity, coupled with problematic notions of prestige and expertise, can all inhibit early career researchers (ECRs) from pursuing impact with their research. These observations correspond with previous research on how research voices without traditional notions of academic capital are unable to change their own cultural conditions and expectations (Wróblewska et al., 2023). Moreover, such experiences reflect Sümer and colleagues’ (2020) typology of academic citizenship, which explores how employment status and gender effect one’s ‘citizenship’. Within their typology, non-citizenship is commonly experienced by those on short term contracts. Non-citizens are seen as disposable by institutions, and are unable to participate in decision-making. Conversely, senior individuals could be said to have ‘full academic citizenship’ which confers recognition, belonging, and capabilities of fully participating in decision making. Consequently, institutions must create opportunities which allow ECRs to influence organisation-level decisions affecting RC (Bankston et al., 2020; Kent et al., 2022). For this to succeed, ECRs must be given time and recognition for these efforts.

Cross-institutional networks and connections were discussed as another route to impact, particularly in the sphere of research culture. Interviewees discussed the benefits and collaborative efforts of cross-institutional networks such as the N8, GW4, GuildHE, and ARMA (Association of Research Managers and Administrators). One meta-researcher described how their regional network shared “experiences of initiatives”, offering advice about the pros and cons (MR5). Furthermore, they mentioned ongoing discussions about joint initiatives in the future within this network.

Although these networks appear to enable better informed decision-making, there is a risk that their exclusivity could limit their potential impact. If discussion and sharing only occurs within these networks, rather than extending across them, this could constrain the reach of valuable lessons that could benefit the sector. To combat this concern, DM1 proposed a public, centralised repository of initiatives and their evaluation. Such a centralised database would help to institutionalise links between RC evidence and policy, laying some structural groundwork to support the normalisation of practices and expectations about the use of evidence in decision making (Parkhurst, 2017, p.31).

Theme 2.3: Many Challenges exist in acquiring and acting upon research evidence

This final theme presents three different barriers to meta-research impact, all relating to difficulties either acquiring or acting upon evidence. The first barrier involves generating the standards of evidence required to inform significant changes in wide-reaching policies. This stemmed from various factors, including the relative novelty of meta-research on RC, the recency of funding into these questions, and the magnitude of evidence required to reliably inform impactful policy changes.

“[The Economic Research Council] could fund a million-quid economic research project, but then if you turn around and say okay, now change your macroeconomic policy based upon it. Well hang on, that’s not a million quid anymore, now we’re talking about billions, you know, we’re changing our foreign trade policy based upon that.” – MR4

As the above quote illustrates, the (potentially negative) cost associated with the consequence was a central factor in deciding whether evidence is sufficient to act upon. This was voiced by various interviewees, who described negotiations with university solicitors and the need for “a bulletproof case” (DM5). For an EDI-related initiative in one university, they compiled data going back years “to prove [that there was a] drop-off in marginalised profiles” as you moved across career stages (DM5). This enabled them to provide such strong evidence of a problem that they were able to ringfence funds.

The second evidential barrier, discussed primarily by decision-makers, was that evaluations of RC initiatives have often lacked informational value. Evaluation has typically taken the form of feedback surveys, “[without] a baseline comparator – or a control group – or anything like that” (DM1). These factors can limit what could be learned from research, and therefore the impact it can have. Although qualitative feedback is ideally suited for certain research questions (such as intervention acceptability), it is less informative for other important questions, such as long-term causal outcomes (Glasziou et al., 2004; Petticrew & Roberts, 2003).

The reason that historic evaluation has been qualitative was partly attributed to the fact that culture is intrinsically hard to define and capture, let alone operationally define and therefore test in such a way. Moreover, notable cultural changes are likely to play out over the course of years, and are therefore hard to measure at this stage in institutions’ journeys to improving RC.

The final identified barrier was that locally produced research could be poorly received when it reflected negatively on individuals and institutions. This had the potential to limit both uptake and sharing of research findings, thereby reducing impact. At an individual level, findings could be taken personally, hindering acceptance. One meta-researcher discussed how local research into EDI had suggested unconscious biases within their institution. Disseminating and leveraging that research into change had been challenging.

“It’s natural defensiveness when you say to people you’ve probably got these biases and then people perceive it as a personal attack, which it’s not but it’s very hard to get around that.” – MR3

A decision-maker at a different institution discussed an analogous experience, in which a colleague had researched differential career trajectory impacts of the COVID-19 pandemic. They sensed that, “bizarrely, it is easier when the research happens somewhere else” (DM11). While having “no evidence to support that”, they speculated that “other institutions looking at data collected from our staff weren’t offended by that, whereas some people here were.”

At an institutional level, it is important to consider that meta-research into RC is often being done within institutions on themselves. When findings reflect poorly on that institution, interviewees perceived a hesitance to share findings that might damage the institution’s reputation. Given the sector-wide importance of sharing lessons on RC, one interviewee highlighted the sector’s need to learn how we can share such findings, “without getting kicked about it” (MR7).

In conclusion, a major challenge was generating research of sufficient quality and quantity that it could reliably be used to effect changes. This barrier was linked to resource constraints, as well as difficulties in operationalising and measuring aspects of RC. Moreover, when qualities of the research generated could cast people and institutions in a poor light, this impacted on their acceptability – at least within the local context within which the research was conducted.

Discussion

This study explored how meta-research has impacted English HEI decisions affecting research culture, alongside the mechanisms by which research can achieve greater or lesser impact. While much of the impetus to improve institutional cultures has been driven by widely-publicised and discussed research (e.g., Nuffield Council on Bioethics, 2014; UKRI, 2015), initiatives to solve these issues are much more recent, and therefore lacking historic evaluation. Indeed, a recent UKRI report found that the majority of research culture initiatives in the UK have not been formally evaluated (UKRI, 2024).

The impact of RC initiatives needs to be more frequently and robustly evaluated (Nair & Kita, 2024; UKRI, 2024). Evidence is needed to justify and refine practice, as it is possible that well-intentioned changes could have negative or unimagined consequences. Positive evidence would support instrumental arguments about the benefits of RC reforms, which participants believed would generate support for changes, particularly from reluctant communities. This point is particularly salient in the context of recent academic dissatisfaction (Erickson et al., 2021). Erickson and colleagues (2021) surveyed 5,888 academic staff across UK HE about their attitudes towards management and university governance, observing a mean satisfaction score of 10.54%. The researchers identified seven themes within academics’ responses. Particularly relevant are the themes on the dominance and brutality of metrics, excessive workload, and perpetual change.

Institutional reforms will fail if they are both cumbersome and perceived as another performative metric, rather than positively instrumental. It is important to consider whether initiative evaluation increases the cost of participating in RC initiatives (DM5). Ideally, if evaluation can be integrated into the running of initiatives, this could reduce or remove the burden on participants. It is also important to recognise existing practice-focused research within professional services. Institutions should look to develop individuals in this space, and foster collaborations with academic researchers, whose combined expertise could provide more informative evaluation of RC initiatives.

The importance of collaboration

“The remit of research culture is ridiculously massive, and we have to collaborate as a sector if we’re going to move the dial on this.” – DM5

Several interviewees mentioned the importance of sector-wide collaboration and sharing – both in terms of RC initiatives and their evaluation. The scope of RC is far too large for each HEI to develop and rigorously evaluate interventions that address every aspect of RC. Institutions need to take the lead on certain aspects of RC, evaluating and sharing the initiatives and findings. This would free up resources for other institutions to do likewise, as well as enabling HEIs without sufficient resource to utilise resources developed elsewhere. Furthermore, cross-institutional collaborations could be used to pool resources towards larger-scale evaluation of initiatives.

A potential obstacle to cross-institutional collaboration is that senior management expect improvements across multiple aspects of RC. Teams might find it challenging to justify temporary neglect of issues because another institution is leading on it. There is therefore a collective action problem to be solved, regarding how to coordinate efforts as a sector to maximise resources.

These points are particularly salient with the forthcoming REF29 exercise, in which ‘People, Culture, and Environment’ will be significant criteria on which institutions are judged. One decision-maker noted how the REF is a “competition with a cash prize” (DM5) and how inter-institution competition could lead to HEIs becoming guarded rather than collaborative on RC. A recurring question was “to what extent will Research England in the next REF recognise and reward collaboration?” (DM7). It is therefore vital that collaboration and openness across the sector are incentivised. Moreover, funding to develop RC has been unevenly distributed across UK universities, impacting where and how initiatives have proceeded. Consequently, there is a moral imperative for those who have received funding to share initiatives and lessons with others.

The need for continued funding

A recurring theme among decision-makers involved concerns about what would happen if RC funding ran out. Across many of the institutions we interviewed, the current scale of RC efforts was “entirely dependent on [external] funding” (DM7). Participants noted the alarming financial situation within the UK HE sector (Bull, 2024) and beliefs that RC initiatives would be among the first things to be cut back. Ultimately, many decision-makers felt that they could not rely on internal funding to continue RC efforts, seeing this as a major concern.

Consequently, there is a clear need “to embed these issues within the culture of the university without that funding” (DM6a). One way to achieve this is through research which evidences the benefits of RC reforms. If research can support the notion that “a really strong research culture is a component of research excellence as it’s traditionally understood” (DM7), then senior management might recognise that investing money in RC initiatives is financially beneficial. It is therefore vital for continued funding to address the current evidence gap on RC reforms. Moreover, stable, longer-term funding would enable longitudinal analysis of RC change, rather than limiting projects to short-term measurements. Funding bodies could use their influence to encourage or require robust evaluation and sharing of findings. However, any evaluation requirements must factor institutional differences in terms of both needs and resource. Evaluation requires additional capacity which needs time and funding.

Strengths and Limitations

This research achieved the desired breadth of perspectives from interviewing several decision-makers from each of our pre-specified strata. However, our sample of meta-researchers was limited by feasibility. There are unquestionably more barriers and enablers to impact to be identified, which future research should explore. Understanding the mechanisms by which early career scholars can achieve greater impact would be particularly useful, given their integral role in the research ecosystem.

Another consideration is that this research singularly focused on addressing the central research questions. Inevitably, an inductive analysis would have yielded insights on related questions that were foregone in this approach. Furthermore, ideas from smaller institutions were occasionally underrepresented in the analysis. This was primarily because they were only just developing cultures of research and were therefore not in the process of reforming established research cultures. Whilst they provided a very useful contrast, with some highly pertinent information, much of the discussion with decision-makers from smaller institutions was not relevant to our central questions. This mismatch of priorities was reflected in how many of the interview questions were not relevant to them. Due to these fundamental differences, smaller institutions are positioned to provide unique innovation and research in this area, which could prove a fruitful area for future investigation be poorly received when it reflected negatively on individuals or the institution. This could limit both uptake and sharing of research findings, thereby reducing impact. Given how much meta-research into RC is being done within institutions on themselves, it is crucial to address how findings which reflect negatively can be shared “without getting kicked about it”.

Conclusion

The effects of initiatives which have been designed to improve institutional research cultures need more rigorous evaluation. Given the broad scope of RC, it is vital that the HE sector develops a cross-institutional culture of collaboration and openness in both the development and evaluation of initiatives to improve RC. The continued financial support and incentivisation of both government and research funders will be crucial in sustaining long-term institutional change and associated evaluation of efforts. Meta-researchers are well positioned to conduct robust evaluation of RC reforms, particularly if they can achieve institutional buy-in from project conception. Embedding evaluation into wide-reaching institutional policy changes is likely to be effective in generating evidence of sufficient quality to inform future decisions.

References

Aguinis, H., Cascio, W. F., & Ramani, R. S. (2017). Science’s reproducibility and replicability crisis: International business is not immune. Journal of International Business Studies, 48(6), 653–663. https://doi.org/10.1057/s41267-017-0081-0

Al-Amoudi, I., & Willmott, H. (2011). Where Constructionism and Critical Realism Converge: Interrogating the Domain of Epistemological Relativism. Organization Studies, 32(1), 27–46. https://doi.org/10.1177/0170840610394293

Allen, C., & Mehler, D. M. A. (2019). Open science challenges, benefits and tips in early career and beyond. PLOS Biology, 17(5), e3000246. https://doi.org/10.1371/journal.pbio.3000246

Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), Article 7604. https://doi.org/10.1038/533452a

Bankston, A., Davis, S. M., Moore, E., Niziolek, C. A., & Boudreau, V. (2020). Why scientific societies should involve more early-career researchers. eLife, 9, e60829. https://doi.org/10.7554/eLife.60829

Baxter, M. G., & Burwell, R. D. (2017). Promoting Transparency and Reproducibility in Behavioral Neuroscience: Publishing Replications, Registered Reports, and Null Results. Behavioral Neuroscience, 131(4), 275–276. https://doi.org/10.1037/bne0000207

Bordignon, F., Chaignon, L., & Egret, D. (2023). Promoting narrative CVs to improve research evaluation? A review of opinion pieces and experiments. Research Evaluation, 32(2), 313–320. https://doi.org/10.1093/reseval/rvad013

Bruton, S. V., Medlin, M. M., Brown, M., Mitch Brown, Sacco, D. F., & Sacco, D. F. (2020). Personal Motivations and Systemic Incentives: Scientists on Questionable Research Practices. Science and Engineering Ethics, 26(3), 1531–1547. https://doi.org/10.1007/s11948-020-00182-9

Bull, L. (2024, August 8). Without further funding the government needs to be clearer about what it expects from universities. Wonkhe. https://wonkhe.com/blogs/without-further-funding-the-government-needs-to-be-clearer-about-what-it-expects-from-universities/

Byington, E. K., & Felps, W. (2017). Solutions to the Credibility Crisis in Management Science. Academy of Management Learning & Education, 16(1), 142–162. https://doi.org/10.5465/amle.2015.0035

Ciubotariu, I. I., & Bosch, G. (2022). Improving research integrity: A framework for responsible science communication. BMC Research Notes, 15, 177. https://doi.org/10.1186/s13104-022-06065-5

Cook, B. G., Lloyd, J. W., Mellor, D., Nosek, B. A., & Therrien, W. J. (2018). Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education. Exceptional Children, 85(1), 104–118. https://doi.org/10.1177/0014402918793138

Dewidar, O., Elmestekawy, N., & Welch, V. (2022). Improving equity, diversity, and inclusion in academia. Research Integrity and Peer Review, 7(1), 4. https://doi.org/10.1186/s41073-022-00123-z

Dienes, Z. (2023). The credibility crisis and democratic governance: How to reform university governance to be compatible with the nature of science. Royal Society Open Science, 10(1), 220808. https://doi.org/10.1098/rsos.220808

Domecq, J. P., Prutsky, G., Elraiyah, T., Wang, Z., Nabhan, M., Shippee, N., Brito, J. P.,

Boehmer, K., Hasan, R., Firwana, B., Erwin, P., Eton, D., Sloan, J., Montori, V., Asi, N., Abu Dabrh, A. M., & Murad, M. H. (2014). Patient engagement in research: A systematic review. BMC Health Services Research, 14(1), 89. https://doi.org/10.1186/1472-6963-14-89

Erickson, M., Hanna, P., & Walker, C. (2021). The UK higher education senior management survey: A statactivist response to managerialist governance. Studies in Higher Education, 46(11), 2134–2151. https://doi.org/10.1080/03075079.2020.1712693

Fanelli, D. (2010). Do Pressures to Publish Increase Scientists’ Bias? An Empirical Support from US States Data. PLOS ONE, 5(4), e10271. https://doi.org/10.1371/journal.pone.0010271

Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904. https://doi.org/10.1007/s11192-011-0494-7

Fletcher, A. J. (2017). Applying critical realism in qualitative research: Methodology meets method. International Journal of Social Research Methodology, 20(2), 181–194. https://doi.org/10.1080/13645579.2016.1144401

Glasziou, P., Vandenbroucke, J. P., & Chalmers, I. (2004). Assessing the quality of research. BMJ (Clinical Research Ed.), 328(7430), 39–41. https://doi.org/10.1136/bmj.328.7430.39

Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2019). Calibrating the Scientific Ecosystem Through Meta-Research. 27.

Hattery, A. J., Smith, E., Magnuson, S., Monterrosa, A., Kafonek, K., Shaw, C., Mhonde, R. D., & Kanewske, L. C. (2022). Diversity, Equity, and Inclusion in Research Teams: The Good, The Bad, and The Ugly. Race and Justice, 12(3), 505–530. https://doi.org/10.1177/21533687221087373

Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), Article 7548. https://doi.org/10.1038/520429a

Hsieh, H.-F., & Shannon, S. E. (2005). Three Approaches to Qualitative Content Analysis.

Qualitative Health Research, 15(9), 1277–1288. https://doi.org/10.1177/1049732305276687

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

Kent, B. A., Holman, C., Amoako, E., Antonietti, A., Azam, J. M., Ballhausen, H., Bediako, Y., Belasen, A. M., Carneiro, C. F. D., Chen, Y.-C., Compeer, E. B., Connor, C. A. C., Crüwell, S., Debat, H., Dorris, E., Ebrahimi, H., Erlich, J. C., Fernández-Chiappe, F., Fischer, F., … Weissgerber, T. L. (2022). Recommendations for empowering early career researchers to improve research culture and practice. PLOS Biology, 20(7), e3001680. https://doi.org/10.1371/journal.pbio.3001680

Lewis-Wilson, S., Towers, S., & Wykeham, H. (2023). The Luck of the Draw: Wellcome’s Institutional Fund for Research Culture [version 2; peer review: 1 approved, 2 approved with reservations]. Wellcome Open Research, 8(525). https://doi.org/10.12688/wellcomeopenres.20057.2

Musselin, C. (2018). New forms of competition in higher education1. Socio-Economic Review, 16(3), 657–683. https://doi.org/10.1093/ser/mwy033

Nair, C., & Kita, S. (2024). Why Do We Need an International Research Culture Conference? : Lessons from IRCC23 and future directions. Exchanges: The Interdisciplinary Research Journal, 11(3), Article 3. https://doi.org/10.31273/eirj.v11i3.1540

Nature. (2010). Publish or perish. Nature, 467(7313), 252–252. https://doi.org/10.1038/467252a

Nuffield Council on Bioethics. (2014). The culture of scientific research: The findings of a series of engagement activities exploring the culture of scientific research in the UK. https://www.nuffieldbioethics.org/publications/the-culture-of-scientific-research

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). https://doi.org/10.1126/science.aac4716

Parkhurst, J. (2017). The Politics of Evidence: From evidence -based policy to the good governance of evidence. Taylor & Francis. https://library.oapen.org/handle/20.500.12657/31002

Petticrew, M., & Roberts, H. (2003). Evidence, hierarchies, and typologies: Horses for courses. Journal of Epidemiology & Community Health, 57(7), 527–529. https://doi.org/10.1136/jech.57.7.527

REF. (2024, March). People, Culture and Environment Update March 2024 – REF 2029. https://www.ref.ac.uk/news/people-culture-and-environment-update-march-2024/

RoRI. (2024). Projects. Research on Research. https://researchonresearch.org/projects/

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638

Royal Society. (n.d.). Research Culture | Royal Society. Retrieved 30 July 2024, from https://web.archive.org/web/20240730085305/https://royalsociety.org/news-resources/projects/research-culture/

Sümer, S., O’Connor, P., & Le Feuvre, N. (2020). The Contours of Gendered Academic Citizenship. In S. Sümer (Ed.), Gendered Academic Citizenship: Issues and Experiences (pp. 1–36). Springer International Publishing. https://doi.org/10.1007/978-3-030-52600-9_1

UK Reproducibility Network. (2023). Research Culture Catalogue. https://osf.io/vbg35/

UKRI. (2015). The metric tide: Review of metrics in research assessment. https://www.ukri.org/publications/review-of-metrics-in-research-assessment-and-management/

UKRI. (2022, September 20). Enhancing research culture funding allocations 2022 to 2023. https://www.ukri.org/publications/enhancing-research-culture-funding-allocations-2022-to-2023/

UKRI. (2024a). Research and innovation culture. https://www.ukri.org/what-we-do/supporting-healthy-research-and-innovation-culture/research-and-innovation-culture/

UKRI. (2024b). Research culture initiatives in the UK. https://www.ukri.org/publications/research-initiatives-in-the-uk-report/

Ware, J. J., & Munafo, M. R. (2015). Significance Chasing in Research Practice: Causes, Consequences and Possible Solutions. Addiction, 110(1), 4–8. https://doi.org/10.1111/add.12673

Wellcome. (2020). What researchers think about the culture they work in. https://wellcome.org/reports/what-researchers-think-about-research-culture

Wróblewska, M. N., Balaban, C., Derrick, G., & Benneworth, P. (2023). The conflict of impact for early career researchers planning for a future in the academy. Research Evaluation, rvad024. https://doi.org/10.1093/reseval/rvad024

Xie, Y., Wang, K., & Kong, Y. (2021). Prevalence of Research Misconduct and Questionable Research Practices: A Systematic Review and Meta-Analysis. Science and Engineering Ethics, 27(4), 41. https://doi.org/10.1007/s11948-021-00314-9

Supplementary information

Abbreviations

DM – Decision-maker

ECR – Early Career Researcher HEI – Higher Education Institution MR – Meta-researcher

RC – Research culture

REF – Research Excellence Framework UKRN – UK Reproducibility Network Author info / CRediT

Contributor roles are stated in the table below. For items where contributors had similar contributions names are listed in reverse alphabetical order (Z-A).

Roles Contributors
Conceptualisation: Neil Jacobs, Lorna Duncan
Data curation: Robbie Clark, Ros Strang, Pen-Yuan Hsing
Formal analysis: Robbie Clark, Ros Strang
Funding acquisition: Neil Jacobs
Investigation: Ros Strang, Pen-Yuan Hsing, Lorna Duncan, Robbie Clark
Methodology: Lorna Duncan, Ros Strang, Pen-Yuan Hsing, Robbie Clark
Project administration: Ros Strang
Resources: Ros Strang, Pen-Yuan Hsing, Lorna Duncan, Neil Jacobs, Robbie Clark
Software: N/A
Supervision: Lorna Duncan, Neil Jacobs
Validation: Lorna Duncan, Robbie Clark, Ros Strang
Visualisation: N/A
Writing – original draft Robbie Clark, Ros Strang
Writing – review and editing Robbie Clark, Ros Strang, Pen-Yuan Hsing, Lorna Duncan, Neil Jacobs

Conflicting interests / Funding

This is a project under the aegis of the UK Reproducibility Network (UKRN: www.ukrn.org), which is a UK-wide academic collaboration hosted in the University of Bristol’s School of Psychological Science, whose aim is to promote and enable rigorous and transparent research.

METEOR has been funded using pooled resources from thirteen UK HEIs committed to this work (University of Bristol; King’s College London; Keele University; University of Leeds; University of Leicester; University of Manchester; Oxford Brookes University; University of Portsmouth; University of Reading; University of Sheffield; University of Southampton; University of Sussex; University College London).

Several months before being recruited to help with the project’s analysis and writing, RC was interviewed as a meta-researcher for the project. This potential conflict and how it was resolved are discussed in the Reflexivity statement.

The authors have declared they have no competing interests.

Methodological rigour

Validation

Internal validation of all study processes was undertaken by LD, an experienced mixed methods researcher. Additionally, all (n = 21) participants were invited to submit anonymous online feedback and join a workshop to discuss, critique, and refine the first complete draft of this manuscript and the executive summary. Seven participants attended the workshop, which is reported in full here: https://osf.io/c6j8n. For concision, this publication contains only the final conclusions, having taken feedback during the workshop into account.

Philosophical framework

This research was underpinned by a critical realist philosophy, due to its coherence with the project aims and analysts’ theoretical convictions (Al-Amoudi & Willmott, 2011; Fletcher, 2017). Critical realism stipulates that what can be known (epistemology) can only reflect a part of the deeper underlying reality (ontology), which is taken to exist independently of observers. Hence, it rejects the view that reality is reducible to the constructions of individuals, whilst simultaneously acknowledging that knowledge production is inevitably mediated by the context of the knowledge-producer.

In this project, taking a critical realist stance entailed striving to represent participants’ experiences and perspectives, whilst recognising that data collection and analysis are inevitably theory-laden, and therefore subject to bias and uncertainty. Methodologically, critical realism involved seeking causal explanations (abduction) of the data, rather than rich description alone. Through such abductive inferences, this study can be situated within the wider literature as contributing empirically-testable hypotheses for future research. Moreover, by triangulating explanations with multiple modalities of evidence, this study offers empirical and theoretical insights to inform current practice in lieu of more certain evidence.

Reflexivity

Much of the project was conducted by early career meta-researchers, whose primary goal is improving RC within HEIs. This offered a situated and embodied perspective which could be considered both asset and hindrance. A potential concern is that the researchers might unconsciously look for things which correspond with their own experiences and opinions, to the exclusion of contrary evidence. This is particularly salient, given that the primary analyst had been interviewed for the project some months earlier. Several strategies were employed to mitigate issues which might arise from this. First, RS independently checked and validated the coding. Second, we ran a participant check workshop to validate interpretations with participants. Finally, the researchers sought to continually question and falsify emerging explanations by revisiting the data and searching wider literature. A potential benefit of this situated understanding was that the researchers might draw more likely explanations, precisely because of their theoretical and personal knowledge of the topics.

This research was unquestionably influenced by a bias towards the pursuit of utility, above other interests. This is reflected in the strict pursuit of the central research aims, which were intended to maximally inform key stakeholders (such as decision-makers and meta-researchers). The data could undoubtedly answer additional questions which were neglected in this pursuit.

Ethical considerations

Ethics approval was obtained from the School of Psychological Science Research Ethics Committee at the University of Bristol (approval code: 17685). All participants provided written consent before participation in the study, along with retrospective consent for their data to be shared with restricted access.

Data availability

Due to the sensitivity of the data involved, these data are published as a restricted dataset at the    University    of    Bristol    Research    Data    Repository data.bris, at https://doi.org/10.5523/bris.1atv1d062i7c52mtfiz6xc44l1. The metadata record published openly by the repository at this location clearly states how data can be accessed by bona fide researchers. Requests for access will be considered by the University of Bristol Research Data Service, who will assess the motives of potential data re-users before deciding to grant access to the data. No authentic request for access will be refused and re-users will not be charged for any part of this process.

Editors

Kathryn Zeiler
Editor-in-Chief

Alex Holcombe
Handling Editor

Editorial assessment

by Alex Holcombe

DOI: 10.70744/MetaROR.521.1.ea

The reviewers consider this work a highly valuable contribution to the understanding of actions taken to improve research culture in higher education institutions in England, specifically when and how meta-research has informed such actions. The reviewers praised the study’s methodology and findings. They had some suggestions for improving the clarity of various aspects, and some potentially-worthwhile suggestions for adding to the article, for example to enrich the analysis and to make the findings more digestible.

Recommendations for enhanced transparency

  • Add an email address for the corresponding author.

For more information on transparency recommendations, please refer to our author guidelines.

Competing interests: None.

Peer review 1

Yu Zhu

DOI: 10.70744/MetaROR.521.1.rv1

I read this manuscript as a researcher working on research evaluation, and my comments are offered mainly from that perspective. I enjoyed the paper a great deal. It turns attention to institutional decision-makers, a group rarely studied directly, and the notion of “institutional meta-research” strikes me as a genuinely useful contribution. The participant validation workshop and the candid reflexivity statement are also exemplary. The suggestions below are meant only to help an already valuable study make its contribution even clearer.

Main comments

  1. Purposes of evaluation. The paper’s central recommendation is that research culture initiatives be evaluated more rigorously, but the purposes of such evaluation are not distinguished. Learning and improvement, building support for reform, accountability to funders and the REF, and justifying investment all appear in the text, and each implies different standards of evidence. Distinguishing these purposes would give “rigorous” a clearer reference point. The sampling strata might be a natural resource here, since institutions with different levels of funding and research intensity seem to face rather different evaluation needs. A related observation from my own context: an internal research-integrity survey I am familiar with risked being shelved because its results might affect reputation and resource allocation. This suggests that the reluctance to share unfavorable findings may be structural as well as psychological, arising when learning-oriented evidence is drawn into accountability.
  2. Meta-evaluation. The judgment that existing evaluations have “limited evidential value” is itself a meta-evaluative judgment, made largely against the criterion of causal rigor. Making the criteria explicit and relating them to purpose could strengthen this part of the argument. Bornmann (2026) likewise counts the evaluation of assessment procedures among the principles of responsible assessment.
  3. The concept of research culture. The Royal Society definition is descriptive, whereas the analysis in effect operationalizes research culture through Research England’s nine funding areas, which also serve as coding categories. How participants themselves understand research culture remains unexplored, and the implicit normative baseline, which is close to Mertonian norms (ideals), might be stated explicitly. The authors’ own remark that an inductive analysis would have yielded further insights suggests a natural place to address this.
  4. The evaluability dilemma. To me, one of the most interesting findings is that research culture is hard to operationalize, yet operationalizing it risks turning it into another metric game. This could perhaps be foregrounded as a contribution in its own right; how acute the dilemma is likely depends on the purpose of evaluation.
  5. Substantive versus symbolic change. The present material may not allow substantive change to be distinguished from symbolic adoption. The channels described in Theme 1.1 (funder requirements, emulation of other institutions, advocates and networks) closely resemble coercive, mimetic, and normative isomorphism (DiMaggio & Powell, 1983), which can spread formal structures without changing practice (Meyer & Rowan, 1977). This lens, together with the literature on research utilization (e.g., Weiss, 1979), might help theorize “percolation” and inform evaluations able to detect null or symbolic effects (cf. Bornmann, 2026).
  6. Organization of Theme 2. Since the second research aim asks what enables and inhibits impact, the authors might consider organizing the mechanisms by stage (for example, evidence production, channels, and reception), with enabling and inhibiting aspects discussed together, rather than by valence. Even a brief integrative synthesis in the Discussion could help, and might also reduce the overlap between Themes 1.2 and 2.3.
  7. Institutional conditions and transferability. Several mechanisms seem to rest on features of the English system: institutional autonomy, competitive funding, the REF, short-term ERC funding, and an independent meta-research community. Noting these conditions, and perhaps qualifying the generality of the title and conclusions, would help readers elsewhere. Seen from China, where research culture is shaped largely through national policy and evaluation reform, the study is a reminder of Merton’s older question of how the ethos of science fits different institutional orders.

Minor points

  • The critical realist and abductive stance described in the Supplementary Information sits somewhat apart from the largely descriptive directed content analysis. A brief comment on how the inclusive coding rules (“clutch at straws”) leave the conclusions open to disconfirmation would be helpful.
  • A fuller positionality statement may be warranted, covering both the primary analyst’s earlier role as an interviewee and the relationship between the funding consortium and the recommendations for continued funding.
  • Some strong wording (e.g., “unquestionably”) could be softened to match the qualitative evidence.
  • The Methods section reports 18 interviews, while Tables 1 and 2 seem to imply 17. Given the small meta-researcher sample, the identifiability of MR4 may also deserve a second look.

Overall

This is a thoughtful and timely study. In my view, clarifying the purposes of evaluation and engaging with how participants themselves understand research culture would do most to enhance its contribution. I thank the authors for a stimulating read.

References

Bornmann, L. (2026). To sign or not to sign initiatives to reform research evaluation: A discussion of the pros and cons. Journal of Informetrics. https://doi.org/10.1016/j.joi.2026.101872

DiMaggio, P. J., & Powell, W. W. (1983). The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields. American Sociological Review, 48(2), 147–160.

Meyer, J. W., & Rowan, B. (1977). Institutionalized organizations: Formal structure as myth and ceremony. American Journal of Sociology, 83(2), 340–363.

Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 39(5), 426–431.

Competing interests: None.

Peer review 2

Annamaria Carusi

DOI: 10.70744/MetaROR.521.1.rv2

This paper reports on a study of the extent to which evidence about research culture informs decision making at UK HE institutions.  Since this entire policy area in HE is little understood, it makes a welcome and informative contribution,.

The issue that the research addresses is clearly outlined, the methodology is well described and is appropriate, and the results are systematically and clearly presented.

The paper does a very good job of showing the discrepancies regarding the lack of evidence for decisions regarding research culture change in institutions, even when there is evidence available, and in principle referred to; it also points out the dominance of funding in driving research culture change. In discussion the second theme, the researchers point to co-creation as one of the most effective ways of closing the gap between research and policy, as well as the many tensions experienced by researchers in different career stages. In discussing the evidential barriers, the paper shows the many difficulties and challenges regarding evidential standards, which is at the crux of the problems faced by research culture as a policy area.

Overall, I think that this is solid research that sheds light on the many tensions and challenges in the research culture domain. It was probably not within the remit of this paper, but it would be good to see more analysis of why these tensions exist; specifically when it comes to evidence, how power plays out in the demand for evidence – in research institutions which are not, after all, without academic expertise on how to research culture, or organisations.

Competing interests: None.

Peer review 3

Stephen Curry

DOI: 10.70744/MetaROR.521.1.rv3

This manuscript is the report of the METEOR study run by the UK Reproducibility Network (UKRN) which examined how extensively meta-research has been used to inform and measures taken to improve research culture in higher education institutions (HEIs) in England and how often or effectively these intervientions have been evaluated. The study involved interviews with 14 senior institutional decision-makers from 11 different institutions (spanning a range of sizes and degrees of research intensity) and 7 meta-researchers. A consistent picture emerged suggesting that insights about research culture from meta-research were absorbed ‘indirectly’ (percolating via individual expertise rather than more systematically and that measures to ameliorate research culture were rarely, if ever, evaluated for their impact.

Although the study is of a relatively small sample of UK HEIs, there is good representation across the sector and the lessons learned – namely that research culture reforms should be evidence-based and evaluated systematically – is in tune with wider calls for more metascientific approach to all aspects of organisational, procedural and cultural change within research performing and research funding organisations. A particularly valuable aspect of the study is that interviews were conducted not only with meta-researchers, but also senior leaders within HEI institutions who will be familiar with the challenges posed by implementing change within real-world organisational and financial constraints. Overall, the manuscript is candid, clearly written and accessible to a wide readership. As such, it is a valuable addition to a growing literature.

There are a number of issues that I would like to see addressed in a revised manuscript.

First and most importantly, the body of the manuscript does not contain a sufficiently clear definition of what the authors mean by ‘research culture’. It is defined largely negatively in the introduction in relation to concerns about replicability, research integrity and ‘many other cultural problems’. The manuscript does state that a definition was supplied to the research participants, which is included in the supplementary information. However, the concept of research culture is so central to the report that I think this definition (i.e. the Royal Society quote and a nine-point table) should be included in the main text of the manuscript (along with the definition of meta-research that was provided to interviewees).

Page 12: The first sentence in the section on “Enablers and barriers…” contains internal repetition which looks like an unfinished edit.

Page 12, last para. I am one of those people who does not think there are degrees of uniqueness so would like the authors do dispense with the construction “somewhat unique”.

Page 15: The authors rightly surmise that “The reason that historic evaluation has been qualitative was partly attributed to the fact that culture is intrinsically hard to define and capture, let alone operationally define and therefore test in such a way.” This is a well-known issue ,but one that risks propelling people into further ‘admiration of the problem’, rather than taking concrete steps to solve it. I think there could be value in discussing some of the lessons learned from the implementation of equality charter marks, such as the Athena Swan or Race Equality Charter, which also address aspects of ‘research culture’. With a strong focus on gathering and analysing data to identify priority issues to be addressed and a requirement to formulate an action plan that must be reported on after 5 years, these can be useful tools for making reforms and evaluating their impact. Although these schemes come with an administrative overhead and although there are ongoing concerns about rates of progress, when implemented well (with top-level support), they can provide a mechanism for breaking down complex cultural issues into actionable steps.

Page 16, 2nd para: Quoting a mean satisfaction to 4 significant figures (10.54%) is unnecessarily precise. A quoted figure of 11% or 10.5% is easier to parse and still gives the reader the information they need.

Page 17-18: There appears to be text missing. The end of the last sentence on p17 does not join up with the beginning of the first sentence on p18.

Page 18: The Conclusions rightly note that research funders will have a crucial role in sustaining ling-term institutional change and evaluation efforts. This was a point highlighted in a 2020 paper by myself and others (see https://rori.figshare.com/articles/report/The_changing_role_of_funders_in_responsible_research_assessment_progress_obstacles_and_the_way_ahead/13227914?file=25518674). Clearly there is a COI for me here, so I leave it entirely to the authors to decide whether or not to cite this article.

Page 18: I think it would give the findings and conclusions of the paper more punch if the recommendations were listed as a set of recommended actions. The study is necessarily limited by the relatively small numbers of interviewees but it is nevertheless informed not only by meta-research experts but by a group of senior institutional decision makers whose perspective are not so frequently discussed in research reform literature but who can provide a degree of organisational realism to ground more feasible initiatives to change culture. I suspect that the authors would like their paper to be brought to the attention of all HEI leaders across the UK and to that end would recommend inclusion of a 1-page ‘executive summary’ of the key findings and recommendations.

Competing interests: From 2022 to 2026 the reviewer was a member of the OR4 Global Advisory Group which is a UKRN project for Open and Responsible Researcher Reward and Recognition. Although meetings of this group included Neil Jacobs, one of the co-authors of this report of the UKRN METEOR project, the OR4 Global Advisory Group was not involved in that project. The reviewer is also one of the co-authors of the Harnessing the Metric Tide report, which is cited in the present manuscript; as such he has been an advocate for measures to improve academic research culture, on of its central themes.

Leave a comment