1  Introduction

Agent based models (ABMs) have been used to help develop a better understanding of research-related issues as diverse as uptake of data-sharing (Klebel, Bianchi, Ross-Hellauer, & Squazzoni, 2025), article review and publication (Bianchi, Grimaldo, Bravo, & Squazzoni, 2018), (Allesian, 2012), (Paolucci & Grimaldo, 2014), (Righi & Takács, 2017) and job insecurity in academia (Silverman, Geard, & Wood, 2016). Processes used to allocate research funding have also proved to be of interest to modellers, prompting the publication of a number of studies (Bollen, Crandall, Junk, Ding, & Borner, 2017), (Geard & Noble), (Harnagel, 2019), (Roebber & Schultz, 2011). A desire or perceived need to make the processes of research funding more efficient is a common theme of these ABM-based studies. That theme has also been addressed in related work which makes use of methods other than ABMs (Gross & Bergstrom, 2019), (Dresler, et al., 2023), (Barnett, Clarke, Vaquette, & Graves, 2017).

Research funding systems are both appealing and daunting as subjects for ABMs. Their appeal comes the obvious relevance of models of that kind to issues which will almost certainly directly affect their creators, and from the diversity of factors, roles and behaviours that could be used to build a relevant ABM. Yes, they represent a challenge, but it is (probably) a tractable one and there is external evidence and experience from related models of peer review that can be used to support model development (Harnagel, 2019). If such a model can be made to work, in the sense of incorporating factors and producing patterns of behaviour which it is generally agreed reflect the real world, there is clearly great potential to use them to guide improvements in funding allocation processes.

They represent daunting subjects for much the same reasons. The processes which govern individuals’ decisions about when, where and how to seek research funding are complex and currently far from being fully mapped, never mind understood. The funding processes themselves are diverse and complicated. They involve individuals and organisations who are driven by different aims and beliefs. The funding system is highly connected and full of feedback loops. All this suggests that it is, for the moment, unreasonable to expect to create anything other than a limited simulation of the entire system within a single model.

Despite inherent and well-known limitations in terms of their ability to replicate precisely real-world social systems (Agar, 2003)1 ABMs provide a unique facility to allow their users to investigate, before they are implemented (if that is the intention), how changes to assumptions might affect the behaviour in and of the systems they simulate. While experimental approaches to designing interventions in research funding systems have become more common recently (DSIT, 2025) implementation of some experiments remains either impossible or implausible. In these cases, ABMs may be the only way in which useful, auditable, evidence in support of policy development could be made available. One such case forms the motivation for this work: determining whether a funding process that uses two-stages (outline proposal followed by a full proposal) requires less effort from the applicants who participate in it than would a process that uses only one, straight- to-full-proposal, stage.

2  Background to the issue addressed by the ABM, and rationale for this approach

Funding processes which involve the preparation and assessment of research proposals may be single stage or multistage (usually two-stage, but there could be any number.) Multistage processes typically, but not universally, will be specified so that the earlier stages require less effort of applicants than do later stages and, by definition, have the feature that the total possible effort required from applicants will not be expended in the first stage. The underlying intention is that applicants who fail at an early stage of the process will have expended less effort than would have been the case had they needed to prepare a full proposal in a single stage process. Only those working on the most promising candidate ideas, as identified in an initial stage, will be required to expend the additional effort of a full proposal. It is assumed that this distribution of workload will result in a net saving of applicant effort.

This assumption has never been tested, and it is hard to conceive of an experiment that could test it without requiring a large and complex intrusion on funding processes. Doubts about acceptability, to funders and to applicants, mean that it seems unlikely that such an intervention would ever be carried out. In place of the information that might arise from this all-but-forbidden experiment, there is a body of evidence that tends to suggest that researchers feel that the level of effort needed to participate in funding processes could usefully be reduced (possibly through the use of two-stage processes (Fackrell, et al., 2024)) and that two-stage processes may ( (Morgan, Yu, Solomon, & Ziebland, 2020), (Seeber, Svege, & Hesselberg, 2024)) or may not (Barnett, Graves, Clarke, & Herbert, 2015) require less effort overall.

Much of the available evidence either is derived from the results of surveys or relates to interventions in which a new process replaced an older one. This does not allow direct, controlled, quantitative comparison of the two process types. Overall, it remains unclear which approach of the two might involve the least effort, whether it is right to focus solely on effort or whether other metrics associated with processes ought also to be considered (Seeber, Svege, & Hesselberg, 2024), and whether change from the dominant paradigm of single-stage processes would be welcomed (Mervis, 2016). Still, the question remains live, and research funders are being challenged to give responses to it (Tickell, 2024).

The main issue preventing definitive resolution is that applicant behaviour will almost certainly change in response to a change in process, meaning that we cannot compare what happens in a one-stage process with what happens in exactly the same process run with two (or more) stages – there can be no such thing as ‘exactly the same process’. Specifically, it is reasonable to assume that when the level of effort required to prepare a proposal in response to a call for proposals decreases, all other things being equal the number of applicants responding to that call will increase. This assumption is supported by available evidence (Seeber, Svege, & Hesselberg, 2024) but hardly seems contentious.

So, an ABM is perhaps the only option available if we wish to compare ‘exactly the same process’ in single- and multistage versions. It is the only way we can hope to maintain equivalence in other respects, without temporal or sequencing variability potentially skewing the results. It is the only option that is likely to be practically feasible. And it is the only approach that can reasonably be expected to provide anything other than descriptive, anecdotal and situation-specific information.

In section 3 the ABM developed in response to the need for evidence relating to this issue is described. The model description follows the ODD (Overview, Design concepts, Details) protocol for describing individual- and agent-based models (Grimm, et al., 2006) as updated by (Grimm, et al., 2020). In section 4 the patterns arising from the model (that is, the characteristics of its outputs that may be used to judge the model’s verisimilitude) are shown, while in section 5 results of the model are reported. Section 6 shows the effects of varying some of the key variables of the model while section 7 contains a summary and conclusions.

3  Model description

A complete, detailed model description, following the ODD (Overview, Design concepts, Details) protocol (Grimm, et al., 2006) is provided as an annex to this work. The code used to run the model is at https://doi.org/10.5281/zenodo.18254022.

The logic underlying the model and determining applicant behaviour within it has three core features:

1. applicants respond to an opportunity for funding, represented by a call, in a way that reflects how much effort they believe it will take for them to prepare a proposal. For simplicity, the amount of funding available varies only stochastically across calls (meaning that that particularly small or large funding pots are unusual and the average level is fixed) so that applicants’ expectations of funding do not affect their Calls are either one-stage (‘Straight-to-full’, from here on STF) or two-stage (‘Outline + full’, from here on OPF.)
2. applicants each have a desire for funding (again, stochastically determined) and this desire is compared to their accrued level of funding as part of determining their propensity to respond to a call update. Higher levels of funding held currently or in the past tend to reduce an applicant’s propensity to apply for funding.
3. over time applicants accrue effort that can be used to prepare proposals, consuming it only when they prepare proposals. The amount of effort they consume depends on whether the proposal being prepared is an outline proposal, a full proposal subsequent to an outline (these two occurring only in OTF processes) or a STF proposal.

The overall purpose of the model is to predict the relative effort required of applicants who prepare proposals in response to OPF and STF funding processes and so to determine which, if either, of the two tends to require the least total effort from applicants. Specifically, the aim is to establish the average relative levels of effort per unit of funding and effort per award made for the two process types. As the level of funding is stochastically determined these two indicators will tend to align with each other, but they represent different interpretations of the result and both are of interest.

To consider the model realistic enough for its purpose, I use patterns in outputs describing the level of funding available in each call update, probability of application across the whole population, distributions of frequency of application and award among applicants, variation in levels of application across call types, suitable responses to the level of discounting of anticipated effort required to prepare a full proposal in an OPF process and control over individual applicants’ success rates. Distributions of success rates of calls are also of interest in a diagnostic sense, as is the diversity of probabilities of application. Neither of these last two factors is considered to be essential when judging the plausibility of the model.

The model includes the following entities: applicants (each with their own characteristics, usually stochastically determined), a call (with characteristics that update each time an updated call instance is offered to applicants) and the environment (which defines various features that shape the behaviours of applicants.) The state variables characterizing these entities are listed in Table 1.

The passage of time is represented in the model solely through updates to the state variables of the call. A call’s processes complete and updates to relevant applicant state variables are made before the call itself updates for its next round. The model is specified so that any number of call updates can be used, but for the results here the number of updates has been set at 240. Only 80% of simulations arising from these updates are used for the analysis, as the first and last 10% of call updates are discarded. For the presentation of the main results, 49 runs of the simulation are used (49 is a square number, making it easy to show each run individually within a faceted grid of results.)

The most important processes of the model, which are repeated every call update, are

• updating of the call state variables (which determine the nature of the call, either OPF or STF randomly determined with equal probability, the level of funding offered and how much effort it will take to prepare a proposal, or proposals if the call is OPF)
• determination by applicants, on the basis of their state variables and those of the call, whether they will prepare a proposal
• preparation and submission of proposals, consuming effort, which are then ranked by their quality and either funded in the case of an outline stage, invited to the next stage) or not depending on the availability of funding in the call
• update of applicant state variables in light of the results of the application and decision process

The most important design concepts of the model are the effort-dependent response of applicants to the calls presented to them, and the accrual and consumption of the effort needed to prepare a proposal. Other, less central, concepts that help the model better represent real world factors that likely shape applicant behaviour are the representation for applicants of a target level of funding, a sense of when current funding might end and a sense of the funding currently held.

Despite these and other refinements, the model cannot represent every aspect of a real process. For this reason call updates are assigned to be either OPF or STF at random throughout the simulation. As comparisons are made between the two call types, the expectation is that the effects of missing factors will cancel out when call types are compared. This expectation will be realised only in so far as the effect of a missing factor manifests itself evenly across call types.

The model is self-contained and requires no input data, but it does make use of a few variables which reflect common practice in, or the features of the funding system of, a large research funder (UKRI.) These are variables that define the inherent quality of applicants and a rule that when full proposals are invited following an outline stage, their number is fixed so that the resulting stage’s success rate will be around 50%. These variables can be changed to reflect differing assumptions.

4  Patterns

Figure 1 shows the level of funding available and the fraction of applicants applying in response to each call update as those updates progress.

Figure 1. Patterns of funding availability (left) and levels of application (right) for 49 runs of the model, each run featuring 240 call updates. Y axis scales are omitted as the numbers are of little interest. Panels in the same position in the left and right images represent the results of the same simulation run. Lines and standard error ranges are the outputs of the default values of ggplot’s geom_smooth() function.

Levels of funding and application are sufficiently stable across the full duration of a simulation run while still having some variation within that run.

Figure 2 shows distributions of counts of instances of application and award for each simulation.

Figure 2. Patterns of funding application (left) and awards made (right) for 49 runs of the model, each run featuring 240 call updates. Y axis scales are omitted as the numbers are of little interest. Panels in the same position in the left and right images represent the results of the same simulation run.

The patterns of application and award that the model produces are along the lines expected: most applicants apply only a few times (for 240 call updates the mean and median number of applications across all 49 replications are 29.8 and 26) but the distribution is skewed by a small number of applicants who apply relatively frequently (while the minimum number of applications is 1, the maximum is 135). A similar pattern is both expected and seen when counting instances of successful application (that is, awards made.) The mean and median number of awards are 2.4 and 2 (maximum number of awards is 9, minimum 0.)

In the counts of applications there is a persistent feature centred at around 80 applications which seems to be an artefact of the model. As the pattern of application is an emergent property of the model it is not possible to identify a specific cause for this feature. The number of applicants affected by this divergence from reality is small and it is not expected to affect the validity of the results.

Figure 3 shows the distribution of counts of applications submitted to calls of the two types.

Figure 3. Distributions of counts of instances of application by call type. Data shows combined results of 49 runs of a simulation with 240 call updates. The applicant population is fixed at 1000, so this chart can easily be read as showing a percentage scale on the x axis.

As anticipated, OPF calls tend to attract a greater number of applicants than do STF calls. However, there is a great deal of overlap in these distributions, with some OTF calls attracting few applicants while some STF calls attract very many. Table 2 gives single-figure summaries of the distributions of counts of applicants responding to each call type. To estimate the potential range of results across many instances of simulation, bootstrap intervals for these values have also been calculated by sampling with replacement from the 49 simulation runs used.

We should expect that the more realistic applicants are about the amount of effort that might be required to prepare a full proposal if they are invited to submit one in a OPF process, the smaller the difference between the attractiveness of the two call types. Figure 4 shows that this is a pattern seen in the results of the model².

Figure 4. Variation in median number of applicants applying to each of the two call types as the environment state variable realism_beta_1 varies. Higher values of realism_beta_1 indicate applicants anticipating having to commit a greater proportion of the true effort required to prepare a full proposal in an OPF process. 500 uniformly drawn values of realism_beta_1 in the range 1 to 100 are used (note that the x axis is on a log scale, and that the sampling is also on a log scale) each value being used to run one simulation. Each point is the median value for calls of each type in each simulation, so the plot shows 1000 points. The value of realism_beta_1 used to produce the main results is shown with a dotted vertical line. Horizontal dashed line is 0. Lines are the outputs of the default values of ggplot’s geom_smooth() function.

The value of realism_beta_1 used (5, with realism_beta_2 being set at 10) is chosen so that on average applicants anticipate consuming only about a third of the true effort required to prepare a full proposal in an OPF process. Applicants’ values are drawn from a distribution defined by realism_beta_1 and realism_beta_2, so that their individual levels of ‘realism’ vary stochastically.

Figure 5 shows the distributions of success rates of calls of the two kinds. The results are further split by stage, but for STF processes these two distributions are of course the same.

Figure 5. Distributions of success rates of calls for OPF (top) and STF (bottom) processes, split by call stage (outline on the left, full proposal on the right.) Data shows combined results of 49 runs of a simulation with 240 call updates.

As might be expected, OTF processes tend to have lower success rates than do STF processes (median success rates across 49 replications of the simulation are 10% and 15% respectively) reflecting the fact that the sum of funding is (stochastically) fixed across the two call types while the number of applicants is allowed to vary.

Both call types show a number of calls having very high (at or near 100%) success rates. This feature is not something built into the model’s structure, but it is a pattern seen in real world funding data³ and so its appearance here is not considered to be a cause for concern.

It should be very unusual, perhaps unheard of, for an applicant to both apply frequently and have a high personal success rate. Figure 6 plots individual applicant success rates against the volume of applications made in their simulation (1,000 applicants are created afresh for each new simulation run.) It is presented in the form of a funnel plot (Spiegelhalter, 2005) (Hulkes, 2025), with the dashed lines showing control limits at 3 standard errors (based on the normal approximation this is around a 99% range.)

Figure 6. Funnel plot of applicant personal success rates as a function of their number of applications made. Central line gives an expected success rate based on a nonlinear regression while dashed lines show control limits set at 3 se. To limit file size only 5% of the 49,000 simulated applicants’ results are shown. A chart showing all the data is not perceptibly different.

There are no instances of applicants having both high application volumes and high personal success rates. Most applicants’ rates sit well within the expected range and so are ‘in control’ (Spiegelhalter, 2005). The success-volume relationship is negative, meaning that in general applicants who apply more frequently have lower success rates. It would likely be possible to re-specify the model to produce the opposite relationship if it was believed that was a better reflection of reality but this is not considered an important factor in model functioning⁴.

5  Results

Figure 7 gives a simple visual summary of the distributions of the results of most interest: the levels of applicant effort required per unit of funding awarded, per applicant and per award made.

Figure 7. Visual comparison of the distributions of values of effort required for three measures of interest across 49 simulations each with 240 call updates. Median values are shown with a back circle. Y axis units are not shown as they have no meaning, but note that the scale is logarithmic. The ‘Violin plot’ uses the default values in ggplot2’s geom_violin function.

It is clear from Figure 7 that the distributions of values of these three measures are very broad in both call types, but also that in general OPF processes tend to require the least effort. The combination of a log scale and use of the median somewhat downplays the actual size of the differences, so a numeric summary is given in Table 3. The ranges given are again derived from sampling with replacement across the 49 simulations.

The values in Table 3 represent the ratio of the measures for each call type, that is:

And

This formulation allows a direct comparison of the efficiencies of the two processes. Based on the figures in Table 3, to achieve the same outputs in terms of awards made or funding allocated STF processes require around 20% greater applicant effort than do OPF processes, having accounted for the varying levels of application that we might expect to see in response to the two processes. The level of uncertainty in this figure which derives from inter-simulation variation is rather low (about ± 0.05 based on 95% bootstrapped intervals) but of course how reliable the figure is in an absolute sense is unknowable.

Figure 8 shows the relationship between the total effort consumed and the percentage of (simulated) applicants which applies. Note that as the value of funding awarded through each call update does not vary systematically, Figure 8 can be read as a plot of the ratios (1) and (2) above.

Figure 8. Applicant effort expended (arbitrary units) by call type and level of application (shown as a percentage of the 1,000 simulated applicants.) Lines are the outputs of the default values of ggplot’s geom_smooth() function.

The model confirms the reasonable assumption that OPF processes tend to be more efficient for the same level of application and show that an OPF process might attract substantially more applications and yet still be as efficient as an STF process. While in general we see greater efficiency for OPF than STF processes, this cannot be assumed to be the case in every instance as there is considerable overlap between the distributions. There is sufficient data in Figure 8 for it to be safe to conclude that the levelling off that occurs once about half the applicant population applies is a real feature of the model, but whether it represents a feature that would be observed in real processes is unknown.

6 Varying key variables

Some of the state variables in the model represent expected applicant behaviour, so it is useful to understand how their values influence the model’s results. No claim of specific correctness can be made for any of the choices needed to create the model which produced the results presented.

Figure 4 has already shown how realism_beta_1 (which controls the level of effort that applicants perceive will be needed to complete an OPF process) affect the level of application. Ranges of realism_beta_1 in Figure 4 are such that the level of discounting of effort varies from 91% (at the left) to 9% (at the right). The value used in the main simulation is 67% (that is, applicants ‘see’ only one third of the full proposal effort they will need to consume if they are invited to prepare a full proposal.) As expected, the relative appeal of the two process types converges as the level of discounting decreases.

Figure 9 shows how the ratio of effort ratios (in this case the effort ratio is the effort required per unit of funding awarded) varies with realism_beta_1.

Figure 9. Variation in effort (per unit value awarded) ratio (STF/OPF processes) with varying realism_beta_1. Data is 500 instances of a single simulation. The central line shows the outputs of the default values of ggplot’s geom_smooth() function. The value of realism_beta_1 used in the main simulation is shown with a dotted vertical line. The horizontal dashed line is a ratio of 1 (in other words, equality.)

The nonlinear response reflects the fact that realism_beta_1 is one of the variables that defines a Beta distribution. It may at first seem odd that the ratio increases as the level of discounting decreases – we might expect that less discounting would lead to greater similarity between calls of the two types, and hence a decrease in the ratio. The explanation is that as discounting decreases, the number of full proposals prepared by applicants in OPF processes decreases, increasing the relative efficiency of the OPF process.

Figure 10 shows how application volumes vary with the value of full_proposal_ratio. full_proposal_ratio is used to determine the effort required to produce a full proposal subsequent to an outline. The value used in the main simulation is 1, indicating that the levels of effort will be approximately equal in the two-stages⁵ although in the real world it should be possible to specify a process so that the fraction of the total effort of preparing a submission that is required for the outline stage may be much lower than this (Luebber, Krach, Paulus, Rademacher, & Rahal, 2025). As full_proposal_ratio increases, full proposals comprise a greater fraction of the total effort required in OPF processes. Due to the way in which effort is defined in the model, this also increases the total effort required and so decreases the level of application in both process types.

Figure 10. Variation in number of applicants, across call types, with changing full_proposal_ratio. Data is 500 instances of one simulation (data points not shown due to unhelpful overlap), the median value being the median value of the number of applicants applying across all call updates of each process type in a simulation. Lines are the outputs of the default values of ggplot’s geom_smooth() function. The value of full_proposal_ratio used in the main simulation is shown with a dotted vertical line. The horizontal dashed line is 0.

OPF processes always on average attract more proposals but the relative attractiveness of OPF and STF processes is somewhat sensitive to the composition of effort of the application process. The difference, as expected, approaches zero when the two processes require the same level of effort, the gap between the two widening until full_proposal_ratio is about 3, at which point it becomes reasonably stable. This suggests that the process-related difference in perceived effort has less effect on applicant behaviour as the absolute level of perceived effort increases (for example, to simulated applicants it might make no real difference whether a full proposal consumes 80% or 90% of the total effort.)

When looking at the way in which full_proposal_ratio influences the ratio of effort per unit value awarded in the two processes (Figure 11) we can see that on average effort is least for OPF processes but that their advantage becomes smaller as full_proposal_ratio decreases. The grey band in Figure 11 shows 95% quantile ranges for the median values of the effort ratio, taken for data ± 0.2 units of full_proposal_ratio either side of the value shown along the x axis. A full_proposal_ratio of 1 gives an effort ratio in the middle of the observed range and models a situation in which the effort ratio is likely never to be less than 1 (that is, the grey band containing 95% of values does not cross y = 1, which is the line of equality.)

Figure 11. Variation in effort ratio (STF/OPF) with changing full_proposal_ratio. Data is 500 instances of one simulation. The red line shows the output of the default values of ggplot’s geom_smooth() function. The value of full_proposal_ratio used in the main simulation is shown with a dotted vertical line. The grey band shows the 95% range (that is the range of values between the quantiles 2.5% and 97.5%) of rolling windows of full_proposal_ratio of width 0.4, between 0.2 and 9.8 and increasing in increments of 0.1. The horizontal dashed line is a ratio of 1 (in other words, equality.)

When full_proposal_ratio is low, indicating little difference between the effort required in each of the two processes, simulations in which, on average, OPF processes require more effort per unit of funding awarded than do STF processes become increasingly common. More formally, the 95% range starts to include 1 for values of full_proposal_ratio of below around 0.5. This is territory in which OPF processes create an increasing risk of inducing so much extra demand that they require more effort overall.

7  Summary and conclusions

Even a relatively simple Agent-Based Model can produce patterns which are strongly reminiscent of those seen in, or expected of, real-world funding systems. These patterns emerge spontaneously from the application of simple rules based on reasonable expectations of agent (applicant) behaviour and do not require specification as part of the model.

The aim of the model described is not to replicate perfectly such a system, but is more limited. It is focused on the question of whether offering potential applicants two-stage, ‘Outline + full’ calls rather than one-stage ‘Straight-to-full’ calls will require less effort overall from those applicants and so make the system more efficient in terms of the effort required to commit a defined sum of money or to issue a defined number of awards. The model described here is able to reproduce a variation in propensity to apply that is required to answer the question, in effect running the same calls but under different conditions.

Although the numbers produced should not be taken too literally, this model suggests that, all other things being equal, one-stage processes might require about 15% to 20% more effort than might two-stage processes to achieve the same outputs, even though two-stage processes may induce about 40% more applicants to submit a proposal. The gap is substantial and relatively insensitive to realistic model specifications.

An alternative, more conservative, phrasing of these findings is that it may be possible to induce considerably more headline demand (if that demand comes in the form of outline proposals) while still making the process overall more efficient from the point of view of applicants and the system as a whole. Demand in ‘Outline + Full’ processes could increase by nearly half relative to ‘Straight-to-full’ processes while still offering a substantial efficiency saving. It seems reasonable to believe that even higher demand might continue to be associated with net efficiency savings, but the way in which the two factors are related is not answered by this specific model. The model is not able to identify the point at which excess demand induced by outline processes leads to decreased efficiency overall, although this may be a useful estimate to have available.

Estimates of the cost of preparing research funding proposals are rare, but if we take the results of (Herbert, Barnett, Clarke, & Graves, 2013) as a guide we might expect that if a funder which previously had used only one-stage processes moved to using only two-stage processes (neither being plausible states) the saving of applicant time might be in the range of 2% to 3% of the budget of that funder (based on a 15% to 20% reduction in a cost that is, per (Herbert, Barnett, Clarke, & Graves, 2013), perhaps 14% of the funder’s budget.) For a large funder, this would be a substantial cost saving, although the cost is of course unlikely to be borne directly by that funder – it is the research system and the people who support it that pay.

The model’s outputs suggest that efficiency gains might be greatest when calls are more popular within their pool of potential applicants. Such high volume, general interest calls typically characterise ‘responsive mode’ or applicant-led (UKRI, 2025) type activities, where there are few restrictions on what applicants can do with the funding they seek. But the evidence derived from the model also suggests that there may be great overlap between efficiencies in individual calls. The result applies to funding processes in the round – attempting to segment them and apply a certain policy to just one part of a research funding system could not be guaranteed to produce a beneficial effect.

The model also suggests that the level of effort per applicant may be lowest in two-stage processes (Figure 7, central panel.) In fact, the difference in this measure is the largest of the three. This is not at all surprising, and in fact is inevitable: for applicants on average, the level of effort required to apply must be lowest in two-stage processes. This highlights that there is a certain element of acceptability to, and consideration of the personal experiences of, applicants that funders might usefully consider when deciding what process to use, this factor further tipping the scale in favour of two-stage processes.

The increased volume of applications has a downside though, even if it does not result in lower overall efficiency. While the per-person effort is lower, it is likely that more extensive use of two-stage processes will result in more unfunded, and hence disgruntled, applicants. Depending on how they are calculated, success rates may also plummet. These outcomes may tend to tip the funder’s scale in the opposite direction. The acceptability of the additional time taken to progress a funding application through a two-stage process also is not part of the model. As is always the case with multifactor decisions like this, identification of the right answer depends on what the decision maker values and to what extent they value it.

Attracting a wider pool of applicants may have a plus side though. It has been claimed that the effort required to prepare a research proposal has value, even if that proposal is not funded (Ayoubi, Pezzoni, & Visentin, 2019). If this is true (and counterarguments have been put forward (Dresler, et al., 2023)) encouraging more applicants may benefit the system overall.

The model does not include the effort that a funder and, likely, members of the applicant community will expend on the assessment process. This is assumed to be small relative to the effort required to prepare proposals, and there is some evidence to support this assumption (Luebber, Krach, Paulus, Rademacher, & Rahal, 2025). It is reasonable to assume that the effort required to review proposals will be in proportion to the effort required to prepare them. If this is the case, the efficiency results presented are unlikely to be affected by the omission from the model of assessment effort.

This work suffers from the fundamental limitation of all ABMs that attempt to investigate complicated systems, especially those which involve human decision making. It is impossible to include all the features that exist in the real world, and it is impossible to turn those features that are included into a precise mathematical or rule-based representation of reality. I hope I have been clear enough about this limitation, and cautious enough in the interpretation of the results, to reflect this unavoidable fact adequately.

Potential features that could influence a model’s results but which I have omitted from this one include quantification of other benefits of receiving an award (as opposed to the resource value that an award provides (Hussinger & Carvalho, 2022)), the effect of the available level of funding on applicant choices, structure within the applicant community (for example, different segments of the population requiring different funding amounts), the effects of rejection on subsequent behaviour (Borgstrom, et al., 2023), re-use of previous ideas in a way that reduces effort to apply or increases the efficacy of the system overall (Myers, 2024). I could go on listing these plausible factors but will not, as the reader doubtless gets the general idea. The model is incomplete and wrong (Box, 1976) but perhaps like some models it is useful.

Finally, the sensitivity analysis in Figure 11 sounds a further note of caution. While the model parameters used all but rule out finding that OPF processes are less efficient overall, if those parameter values are not reflected in the real world this inference would be in doubt. While it is reasonable to assume that in an OPF process about half the effort expended will be expended on the outline stage (Seeber, Svege, & Hesselberg, 2024) this is not certain. If those preparing outline proposals do much of the work of preparing a full proposal upfront, perhaps in order to de-risk their application or to meet organisational requirements, full_proposal_ratio will decrease, perhaps to levels where it is far from certain that more extensive use of outlines would be of benefit anyone.

Notes

¹ One of the key contentions in (Agar, 2003) is that the numbers and functions in an ABM are not in themselves meaningful, that they do not (need to) correspond to a real-world measurement akin to, for example, Planck’s constant, and that what matters is whether the model makes sense: “the right question isn’t, does the number mean anything? The right question is, does the number correspond to a difference that makes a difference in the kind of world being modelled? And if that claim is made, then the next question is, who is the modeller to make that claim? Does he/she know anything about the world being modelled?”.

² I have not attempted to implement any kind of modifier that reflects applicants’ expectations of success. In real life, the level of effort they perceive will probably reflect an expected return, not just the hypothetical level of effort.

³ For example, in real processes it might arise from invitation-only calls or other processes for which the successful applicants have been pre-identified.

⁴ UKRI does not have sufficiently long time series data to determine which way, if any, the success-volume relationship for individuals might play out in the real world over the sort of timescales implied by the model’s specifications.

⁵ full_proposal_ratio is the mean of a truncated normal distribution that is used to determine, for each call update, the actual value of the ratio to use for that call. This means that the ratio varies across calls but broadly speaking is centred on 1.

Acknowledgement

I would like to thank Martin France whose comments on an early draft of this work have helped to improve it substantially.

CRediT author statement: Alex Hulkes: conceptualization, methodology, software, writing – original draft, visualization, validation.

Data availability statement: No external data is needed to reproduce the work here so no data is available. A copy of the R code used is available at https://doi.org/10.5281/zenodo.18254022.

Competing interests statement: the author has no competing interests.

Funding source statement: no named sources of funding were used to support the work.

References

Agar, M. (2003). My Kingdom for a Function: Modeling Misadventures of the Innumerate. Journal of Artificial Societies and Social Simulation, 6(3). Retrieved from https://www.jasss.org/6/3/8.html

Allesian, S. (2012). Modeling peer review: an agent-based approach. Ideas in Ecology and Evolution, 5(2), 27-35. Retrieved from https://ojs.library.queensu.ca/index.php/IEE/article/view/4447

Ayoubi, C., Pezzoni, M., & Visentin, F. (2019). The important thing is not to win, it is to take part: What if scientists benefit from participating in research grant competitions? Research Policy, 48(1), 84-97. doi:https://doi.org/10.1016/j.respol.2018.07.021

Barnett, A., Clarke, P., Vaquette, C., & Graves, N. (2017). Using democracy to award research funding: an observational study. Res Integr Peer Rev, 2(16). doi:https://doi.org/10.1186/s41073-017-0040-0

Barnett, A., Graves, N., Clarke, P., & Herbert, D. (2015). The impact of a streamlined funding application process on application time: two cross-sectional surveys of Australian researchers. BMJ Open, 5, e006912. doi:10.1136/bmjopen-2014-006912

Bianchi, F., Grimaldo, F., Bravo, G., & Squazzoni, F. (2018). The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures. Scientometrics, 116, 1401-1420. doi:https://doi.org/10.1007/s11192-018-2825-4

Bollen, J., Crandall, D., Junk, D., Ding, Y., & Borner, K. (2017). An efficient system to fund science: from proposal review to peer-to-peer distributions. Scientometrics, 110, 521-528. doi:https://doi.org/10.1007/s11192-016-2110-3

Borgstrom, E., Driessen, A., Krawczyk, M., Kirby, E., MacArtney, J., & Almack, K. (2023). Grieving academic grant rejections: Examining funding failure and experiences of loss. The Sociological Review, 72(5). doi:https://doi.org/10.1177/0038026123120719

Box, G. (1976). Science and Statistics. Journal of the American Statistical Association, 71(356), 791-799. doi:https://doi.org/10.1080/01621459.1976.10480949

Dresler, M., Budderberg, E., Endesfelder, U., Haaker, J., Hof, C., Kretschmer, R., . . . Schmidt, F. (2023). Effective or predatory funding? Evaluating the hidden costs of grant applications. Immunology & Cell Biology, 101, 104-111. doi:https://doi.org/10.1111/imcb.12592

DSIT. (2025, June 30). A year in metascience (2025). Retrieved November 26, 2025, from https://www.gov.uk/government/publications/: https://www.gov.uk/government/publications/a-year-in-metascience-2025

Fackrell, K., Church, H., Crane, K., Recio-Saucedo, A., Blatch-Jones, A., & Meadmore, K. (2024). Online survey exploring researcher experiences of research funding processes in the UK: the effort and burden of applying for funding and fulfilling reporting requirements. BMJ Open, 14, e079581. doi:10.1136/bmjopen-2023-079581

Geard, N., & Noble, J. (n.d.). Modelling academic research funding as a resource allocation problem. preprint. Retrieved November 25, 2025, from https://eprints.soton.ac.uk/271374/3/fundingModel.pdf

Grimm, V., Berger, U., Bastiansen, F., Eliassen, S., Ginot, V., Giske, J., . . . al., e. (2006). A standard protocol for describing individual-based and agent-based models. Ecological Modelling, 198, 115-126. doi:10.1016/j.ecolmodel.2006.04.023

Grimm, V., Railsback, S., Vincenot, C., Berger, U., Gallagher, C., DeAngelis, D., . . . al., e. .. (2020). The ODD Protocol for Describing Agent-Based and Other Simulation Models: A Second Update to Improve Clarity, Replication, and Structural Realism. Journal of Artificial Societies and Social Simulation, 23(2), 7. doi:10.18564/jasss.4259

Gross, K., & Bergstrom, C. (2019). Contest models highlight inherent inefficiencies of scientific funding competitions. PLoS Biology, 17(1), 1-15. doi:https://doi.org/10.1371/journal.pbio.3000065

Harnagel, A. (2019). A mid-level approach to modeling scientific communities. Studies in History and Philosophy of Science Part A, 76, 49-59. doi:https://doi.org/10.1016/j.shpsa.2018.12.010

Herbert, D., Barnett, A., Clarke, P., & Graves, N. (2013). On the time spent preparing grant proposals: an observational study of Australian researchers. BMJ Open, 3, e002800. doi:10.1136/bmjopen-2013-002800

Hulkes, A. (2025). Success-volume relationships in funnel plots. doi:https://doi.org/10.31235/osf.io/pumt9_v1

Hussinger, K., & Carvalho, J. N. (2022). The long-term effect of research grants on the scientific output of university professors. Industry and Innovation, 29(4), 463–487. doi:https://doi.org/10.1080/13662716.2021.1990023

Klebel, T., Bianchi, F., Ross-Hellauer, T., & Squazzoni, F. (2025). The paradox of competition: How funding models could undermine the uptake of data sharing practices. Research Policy, 24(10), 105340. doi:https://doi.org/10.1016/j.respol.2025.105340

Luebber, F., Krach, S., Paulus, F., Rademacher, L., & Rahal, R.-M. (2025). Lottery before peer review is associated with increased female representation and reduced estimated economic cost in a German funding line. Nature Communications, 16. doi:https://doi.org/10.1038/s41467-025-65660-9

Mervis, J. (2016, August 5). NSF tries two-step review, drawing praise—and darts. Science, 353(6299), pp. 528-529. doi:10.1126/science.353.6299.528

Morgan, B., Yu, L.-M., Solomon, T., & Ziebland, S. (2020). Assessing health research grant applications: A retrospective comparative review of a one-stage versus a two-stage application assessment process. PLoS ONE, 15(3), e0230118. doi:https://doi.org/10.1371/journal.pone.0230118

Myers, K. (2024). Some Tradeoffs of Competition in Grant Contests. arXiv.org, https://arxiv.org/pdf/2207.02379. Retrieved January 12, 2026, from https://arxiv.org/pdf/2207.02379

Paolucci, M., & Grimaldo, F. (2014). Mechanism change in a simulation of peer review: from junk support to elitism. Scientometrics, 99, 663–688. doi:https://doi.org/10.1007/s11192-014-1239-1

Righi, S., & Takács, K. (2017). The miracle of peer review and development in science: an agent-based model. Scientometrics, 113, 587-607. doi:https://doi.org/10.1007/s11192-017-2244-y

Roebber, P., & Schultz, D. (2011). Peer Review, Program Officers and Science Funding. PLoS ONE, 6(4), e18680. doi:https://doi.org/10.1371/journal.pone.0018680

Seeber, M., Svege, I., & Hesselberg, J. O. (2024). Improving the efficiency of research proposals evaluation: A two-stage procedure. Research Evaluation. doi:https://doi.org/10.1093/reseval/rvae020

Silverman, E., Geard, N., & Wood, I. (2016). Job Insecurity in Academic Research Employment: An Agent- Based Model. Proceedings of the ALIFE 2016, the Fifteenth International Conference on the Synthesis and Simulation of Living Systems. ALIFE 2016, the Fifteenth International Conference on the Synthesis and Simulation of Living Systems, (pp. 460-467). doi:https://doi.org/10.1162/978-0-262-33936-0-ch074

Spiegelhalter, D. (2005). Funnel plots for comparing insitutional performance. Statistics in Medicine, 24(8), 1185-1202. doi:https://doi.org/10.1002/sim.1970

Tickell, A. (2024, February 9). Independent Review of Research Bureaucracy. Retrieved November 25, 2025, from https://www.gov.uk/government/publications/review-of-research-bureaucracy: https://assets.publishing.service.gov.uk/media/62e234da8fa8f5033275fc32/independent-review-research-bureaucracy-final-report.pdf

UKRI. (2025, June 2). Funding modes. Retrieved November 28, 2025, from https://www.ukri.org/: https://www.ukri.org/apply-for-funding/funding-modes/

Annex

Technical description of ABM of outline application

The model description follows the ODD (Overview, Design concepts, Details) protocol for describing individual- and agent-based models (Grimm, et al., 2006) as updated by (Grimm, et al., 2020).

1 Purpose and patterns

The model’s specific purpose is to predict the relative (per person, per award made and per unit of funding) effort required of applicants who prepare proposals in response to one- and two-stage funding processes and so to determine which, if either, of the two tends to require the least effort from applicants.

The patterns used as criteria for evaluating the model’s suitability for its purpose are:

Pattern 1: stability of probability of application across the whole population while preserving variability across individual calls. Overall demand, as summarised by the percentage of agents who apply to a call, should not systematically increase or decrease as the simulation progresses, but it should vary between calls.

Pattern 2: patterns of application that indicate a ‘long tail’ of more frequent applicants, with most agents applying relatively infrequently. This is a feature of real-world application systems. Selectivity of application is necessary to render the model plausible,

Pattern 3: patterns of award that indicate a long but thin tail of applicants who are repeatedly successful. This again is a feature of real-world systems, in which most people are successful in receiving an award only very infrequently, but a very small number are highly successful.

Pattern 4: the average difference between the number of agents applying to each call varies by call type (‘Outline + full’ and ‘Straight-to-full’) and decreases as the level of discounting of effort made by those applying to ‘Outline + full’ calls decreases. The central assumption of the model is that applicants are more likely to apply in response to calls that require less effort to apply to, and that discounting of the effort required to prepare a full proposal after an outline application has been successful makes outline calls inherently more attractive to potential applicants. The connection between discounting and propensity to apply is thus essential.

Pattern 5: success rates of individual applicants should not exhibit excessive and/or implausible variation. While very high individual success rates are possible in real-world systems, these typically will be associated with low application numbers and will reflect only expected natural variation around an average rate. There should be no applicants who both apply frequently and have high success rates.

2 Entities, state variables, and scales

The entities included in the model are applicants, the call and the environment. Applicants are individual agents characterised with a number of state variables which together influence each applicant’s propensity to apply for funding from each call. Any number of applicants can be generated but the model as reported uses 1000. The call is characterised by the level of funding available in total for successful applicants to it, and the level of effort required from applicants to apply to it. Applicants operate in an environment containing them and the call (which induces applicants to apply to it or not). The environment embodies rules and assumptions about application and also includes the ability to force application in cases where no applicants apply in response to the call. Forced application is undesirable and so the environment state variables are balanced in a way that minimises (or, if desired, removes) those events. There is only one environment, so that all applicants are subject to the same rules and assumptions.

There is no interaction between the environment and the call.

Space is not represented. A time scale is represented by the pre-determination of the characteristics of the call that will be offered to applicants in sequence, making this an event-based simulation that proceeds in discrete time steps. A time step involves the submission, in response to the current state of the call, of applications by applicants followed by outcome determination for each application (in the case of outline calls perhaps twice) with applicant state variables being updated in light of outcomes. The model does not allow concurrent application – that is, there only one call state at a time. All outcomes are resolved in relation to the call before the call is updated and next used to provoke a response from applicants. This simplifies the model but omits a key area of uncertainty for real-world applicants (i.e the question of whether they should apply to more than one call for the same, different or related work.) Each simulation involves 240 changes of call state. These changes can loosely be thought of, and are referred to, as ‘months’ but there is no actual temporal dimension to them.

The state variables of the entities (applicants, calls) are shown in tables below. Values describing the environment are all static and so, in the ODD framework, the environment has no state variables. A table of parameters for the environment is given for reference.

3 Process overview and scheduling

The first step in the simulation is the creation of the applicants and the determination of their initial state variables. Following this, the call entity is initialised. Rather than updating the call entity in each step, the full set of calls is created first, defining the duration of the simulation. The call entity’s state variables outline_ind, call_value, outline_effort, full_effort and total_effort are updated simply by reading in the relevant next set of call entity state variables. This approach is taken purely for the convenience of being able to see, before the simulation runs, the full set of call states that will be used.

In the sole process in the simulation applicants are exposed to the call, deciding on the basis of their state variables whether to respond to it by preparing a proposal. Those applicants that do prepare a proposal have a proposal created for them based on that applicant’s state variables. The proposal is ranked in competition with other proposals submitted to that call, in either one or two-stages depending on the call outline_ind. Once the outcome is known, the applicants’ state variables time_to_end, accrued_effort, cycle_funding and p_of_application are updated in light of the outcome. The process repeats until all pre-defined call states have been worked through.

Summaries of the outcomes of this process are recorded and stored once it has completed. These form the basis of the analysis.

4 Design concepts

4.1 Basic principles

This model is intended to provide evidence that research funders might find useful when thinking about a central issue of research funding systems: whether it is, in terms of the resources consumed in application processes, better to use two-stage, ‘Outline + full’ processes or to use ‘Straight-to-full’ processes. It is likely to be impossible to produce experimental evidence relating directly to this question, and there is no such thing as a default preferred answer. While it may be that neither option is inherently more efficient, it seems likely that one or the other will in general be preferable.

The key challenge with determining an answer is that applicant behaviour, in terms of the number of people applying, will clearly be influenced by the level of effort associated with application. While for a given number of applicants it will certainly be the case that an ‘Outline + full’ process will consume the least applicant effort, this will not be the case if application volumes can vary in response to that expectation. It is reasonable to believe that, all other things being equal, a call that starts with a (less demanding) outline stage will attract more applicants than will a call in which applicants have to prepare a full proposal upfront.

For this reason a model in which applicants’ decisions about whether to apply are influenced by the effort required to apply may be useful. Numerous other factors will also determine whether, when faced with a new funding opportunity, an applicant decides to apply, for example how far from the end of their current funding they are, how much funding they currently have (and how this relates to how much they would like to have), how long it is since they applied and recent success rates in similar funding calls. A model which does not include these key factors is unlikely to be either useful or persuasive.

A complete and perfect replica of an application system is unlikely to be feasible. This model aims to incorporate what are likely to be the most significant influences to produce a reasonable proxy that reproduces some significant patterns that reflect real-world systems. It compares the outcomes of interest within the same model between call types. In this way, it is hoped, the inevitable deficiencies, errors and omissions of the model will cancel out to some extent, rendering the comparison more valid. The aim is not to determine the relative importance of the factors incorporated in the model that influence applicant behaviour or to identify new ones.

4.2 Emergence

The ratio of the levels of application to each call type (‘Outline + full’ and ‘Straight-to-full’), or equivalently the percentage of applicants who decide to apply to each call, is considered to be an emergent behaviour of this model. This key indicator emerges in response only to the behavioural rules applied to/through applicants. Patterns of application and success are also emergent behaviours/results, but they are, in the ODD framework, patterns rather than results, validating (to some extent) the results.

The main behaviour driving the differential response to calls is expected to be the level of discounting that applicants do when determining how much effort they might have to consume if they apply. Discounting is embodied in the environment parameters realism_beta_1 and realism_beta_2 which together define a Beta distribution from which applicants’ individual levels of discounting are drawn. These are described as environment parameters rather than applicant state variables as the applicant state variable values derive from the environment. The level of discounting is related to the difference between the level of effort required to prepare a full proposal and that required for an outline proposal. This is determined for each call using a truncated normal distribution centred on the environment parameter full_proposal_ratio. full_proposal_ratio thus has a stochastic influence on an emergent property although it is, in the ODD framework, imposed on the model.

Other key imposed results are the equilibrium of application behaviour, so that levels of application remain relatively stable during the simulation, and the steady level of funding offered in each call. Relatively stable available funding levels are a useful simplification. It would be possible to have strongly varying levels, and to model an applicant behaviour that reflected these, but this is not necessary for the purpose of the model. As variation in funding levels would likely be stochastic, its inclusion would simply create ‘noise’ in the results.

Both the level of discounting, embodied in realism_beta_1 and realism_beta_2, and the relationship between the effort required to prepare an outline and a full proposal, as defined with full_proposal_ratio, can be varied systematically to explore how they affect the outcomes of interest. While the ratio of the levels of application to each call type is emergent, it is intended to be controllable so that what is at heart a predictive model has some ability also to allow (in ODD terminology) theoretical exposition.

4.3 Adaptation

The only adaptive behaviour of agents in the model (that is, the applicants) is whether to apply to the current call. The behaviour is modelled as indirect objective seeking. Applicants determine how much effort they have available to apply to the call with which they are presented by multiplying their current accrued effort by their current probability of application. If the available effort exceeds that required to prepare a proposal (as determined in the relevant call update) they prepare a proposal.

Applicants’ available effort accrues at a rate determined stochastically (Poisson distribution) for each applicant. Their probability of application increases as the time since they last held funding increases, but this behaviour reaches a maximum after 12 call updates. An applicant’s probability of application decreases in relation to that applicant’s accrued total funding. If they have just received new funding, or if they have received more funding than their desired amount, their accrued effort is set to zero and they start accruing new effort at their individual rate.

Probability of application is moderated by the success rate of the call implemented in the most recent update: lower success rates have the effect of deterring applicants from applying. All applicants are assumed to have a non-zero but small probability of application after all these adjustments are made. The minimum application probability for any applicant at the start of a call update is 0.01.

The decision by an applicant as to whether or not to apply is central to the model. As well as applicant-specific factors, it reflects the characteristics of the call. For each call update, the amount of effort required and (in the case of ‘Outline + full’ calls) the balance of effort across the outline and full proposal stages is determined stochastically. The balance of effort between the two-stages of an ‘Outline + full’ call is determined by the environment parameter full_proposal_ratio. This determines what fraction of a putative outline proposal effort is added to that effort to determine the subsequent full proposal effort. The effect of full_proposal_ratio is multiplicative. The additional full proposal effort is determined by taking the outline effort and multiplying it by a value drawn from a random truncated normal with mean full_proposal_ratio. For ‘Straight-to-full’ calls the effort is the sum of the (hypothetical) outline and full efforts, so that the total effort required does not vary across call types: in ‘Outline + full’ calls applicants are required to meet it in two-stages while in ‘Straight-to-full’ calls it is met all at once.

4.4 Objectives

The model does not feature direct objective-seeking and so has no objectives.

4.5 Learning

The model does not make use of learning

4.6 Prediction

The only prediction incorporated in the model is the implicit prediction of applicants that if they prepare a proposal for a call they have a better chance of receiving funding than if they do not. Prediction is modelled in this way because it is a central part of the purpose of the model: we wish to determine how different apparent effort levels result in different effort being expended. More complicated predictions, for example based on the expected value of the return on a proposal, are not included as they are deemed unnecessary for this purpose.

4.7 Sensing

Within their own state variables applicants sense their desire for funding (sampled from a normal distribution with mean mean_cycle_target and standard deviation sd_cycle_target, so that each applicant has its own desire level), their cumulative funding to date, the amount of funding they received in the last call update, the level of effort they have available to prepare a proposal and the success rate of the preceding call.

Applicants also sense a perceived_effort which in the case of a ‘Straight-to-full’ call is the amount of effort required to prepare a proposal and in the case of an ‘Outline + full’ call is the sum of the effort needed to prepare an outline and a discounted level of additional effort needed to prepare a subsequent full proposal. Each applicant has their own unvarying level of discounting, determined for them at initialisation by a draw from a Beta distribution.

Some applicants are more optimistic about the level of effort required to prepare a full proposal (or alternatively some are less able accurately to predict how much effort is required) and some are less optimistic/accurate. This leads to variation and uncertainty in the sensed value of the effort required. This is considered to be a likely feature of real-world applicant behaviour.

4.8 Interaction

Interaction in the model is limited to the process of determining which applicants’ proposals are funded and which are rejected. Applicants responding to a call generate a proposal that has an inherent fundability value in the interval [0,1]. Funding decisions are made simply by ranking these proposals and funding down the list of ranked proposals until the money available in the call runs out. A similar process occurs in the first (outline) stage of ‘Outline + full’ processes except that there the resource being competed for is the right to prepare a full proposal. Applicants only interact with other applicants when those applicants are responding to the same call. Calls do not interact with applicants: their parameters are determined before the simulation starts.

4.9 Stochasticity

The model makes extensive use of stochasticity, in particular in determining the initial values of key applicant and call variables so that the simulation is variable without having to model specific causes of variability and so that no two simulations are exactly the same.

Applicants’ cycle_target (funding level), inherent quality (in the form of a Beta distribution defined by quality_beta_1_val and quality_beta_2_val from which individual proposal quality scores are derived), the rate at which they accrue the effort needed to prepare a proposal (effort_rate), the initial fraction of their target sum of funding each has (initial_target_fraction), the time until their initial funding will run out (time_to_end), their initial accrued_effort and their effort_realism (that is the extent of their discounting of the predicted effort required to prepare a full proposal after a successful outline proposal) are all determined stochastically at initialisation.

A call’s call_value is determined by a normal distribution centred on a value derived from the sum of the initial monthly funding (to help with stability of funding availability and hence application) while the outline_effort and full_effort are determined stochastically based on a Poisson distribution (for outline_effort) and the combination of the outline_effort and a truncated normal distribution centred on the full_proposal_ratio parameter. The overall effect is to ensure that some calls represent a greater effort than do others, that the ratio of outline effort relative to the subsequent full proposal stage is also variable (with some outline processes being very low-effort while others save barely any time at all relative to an equivalent ‘Straight-to-full’ process), but that there is no difference between the total effort required for each call type.

A key parameter of the process for determining how which and how many outline applications to invite to prepare a full proposal is outline_perc. This is the desired success rate for proposals submitted to the full proposal stage in an ‘Outline + full’ process. To match common practice in UKRI it is set at 0.5, meaning that sufficient full proposals will be invited so that on average 50% of them will be funded. Variation around this exact figure is to be expected due to the different sizes of applications.

4.10 Collectives

Meetings are the only collective in the model. A meeting comprises all the proposals submitted by applicants who decide to submit proposals in response to the most recent call update. Meetings emerge from applicant behaviours and are governed by call state variables, as call state variables determine success or invitation rates. Meetings represent the process by which it is determined which applicants receive funding. They also act to consume accrued_effort and so are a key part of the mechanism by which application behaviour is controlled and moderated.

4.11 Observation

A typical simulation run will involve selecting an arbitrary number of ‘months’ (in the simulation a call update is referred to as a month) and allowing the process of updating the state variables in response to updated call variables to occur that number of times. Typically, 240 months is used but in principle the simulation could run for any number of months. The process of running a simulation for the desired number of months can be repeated as many times as required in order to generate a range of outcomes.

For each simulation, the state of the set of applicants is recorded after each call update has completed. For each update (month) several diagnostic values are created. Some of these are useful for tuning or monitoring the model while others are used to extract the results that help support the purpose of the model. These latter values are

Although it is a tuning/monitoring value rather than one that supports the model’s purpose it is worth highlighting the existence of the call_is_unpopular value in the diagnostic results generated by a simulation. It is possible for a call update not to provoke the submission of any proposals, or to provoke only one proposal. When this happens the model forces a stochastically-determined number of applicants to submit proposals. This number is a single Poisson variable with λ = determined_applicants where determined_applicants is a defined parameter. The determined_applicants are not selected at random but instead are the top n_applicants/determined_applicants applicants in terms of their p_of_application (that is, their probability of application). A value of 50 is used, so that when a call_is_unpopular around one fiftieth of all potential applicants, but only those with the highest probability of applying, are forced to apply. Calls which make use of this feature of the model are marked as call_is_unpopular. The extent to which the simulation relies on this feature if it is to be sustained depends on the typical level of application: the higher it is, the less common it is that a call_is_unpopular. There thus exists a practical trade-off when running the model: it will run for extended periods without intervention if the average level of application is high, but this seems less realistic. If the level is lower it may on occasion need an in-simulation forcing instance, generating a call_is_unpopular = 1 value. The extent to which this happens can and should be monitored, as multiple forcing instances indicate that the simulation is not working adequately well. This is particularly an issue as the probability that call_is_unpopular will not be independent of call type: ‘Straight-to-full’ calls will be more likely to be unpopular as they require more resources to prepare a proposal. For the values used in simulations as typically run the rate of call_is_unpopular is usually in the low single digits percent (2 or 3%) indicating that in a typical run a forcing instance will only occur a few times.

5 Initialization

For a simulation a defined number of applicants (n_applicants) is created, typically 1000. Each is assigned an applicant_id from 1 to n_applicants. They are given a cycle_target of funding, which is the amount of funding they would like to have available to them in a month. This is drawn from a truncated normal distribution of defined mean (mean_cycle_target) and standard deviation (sd_cycle_target). A truncated normal is used to prevent the

cycle_target being less than 0. Each applicant is assigned two values (quality_beta_1_val and quality_beta_2_val) as parameters in a Beta distribution that will define the quality of the proposals they submit. Values of quality_beta_1_val and quality_beta_2_val are themselves drawn from a Beta distribution that is defined (with quality_min_beta_1, quality_max_beta_1, quality_min_beta_2, quality_max_beta_2) in a way that gives distributions which will mimic acceptably well those seen in the real-life example given by UKRI. A rate at which effort is accrued by an individual applicant (effort_rate) is determined by a Poisson random variable with λ = effort_accrual_rate. If effort_rate happens to be 0, it is assigned as 1. Applicants are assigned stochastically a initial_target_fraction of their desired funding, sampled from a Beta distribution with parameters target_fraction_beta_1 and target_fraction_beta_2. The initial_target_fraction and the cycle_target are used to determine the applicant’s initial cycle_funding. They are also assigned an initial time_to_end which is the number of call updates they expect to experience until their current funding ceases and their level of funding drops to zero. This is determined stochastically using a uniform distribution in the interval [0,1] and is set to 1 if it is less than 1.

Applicants are also assigned an accrued_effort which is a starting value for the level of effort they have available to use to respond to call updates. This is a function of the environment’s effort_accrual_rate and a uniform distribution in the interval [0,1]. Finally an effort_realism is assigned to each applicant, drawing from a Beta distribution defined by realism_beta_1 and realism_beta_2. accrued_effort, initial_target_fraction, time_to_end and time_to_end_moderator are used to determine an applicant’s initial probability of application (p_of_application).

Calls are initialised by generating the required number of updates (n_calls) each with an identifier, an indicator of whether the call update is an ‘Outline + full’ or a ‘Straight-to-full’ call (with probability defined as outline_perc and 1-outline_perc respectively) and a call_value which is defined stochastically with a normal distribution with mean call_total and standard deviation call_total/5. The way in which the call update’s outline_effort, full_effort and total_effort are derived has been described already.

The meetings at which applicants’ proposals are compared do not exist as collectives before the simulation and are not initialised.

The model is designed to recreate only one simulated ‘world’, in the expectation that multiple ‘worlds’ might easily be created by running the model as may times as desired. For simulating multiple ‘worlds’ the initialisation is kept the same. But the purposes of the model suggest that being able to vary systematically some of the initial state parameters, to explore how the model responds to the initial conditions, would be helpful.

To achieve this two of the initial conditions are varied systematically: realism_beta_1 (which controls, through its contribution to the definition of a Beta distribution, how realistic applicants are, in terms of the extent to which they discount the effort required to prepare a full proposal following an outline) and full_proposal_ratio (which controls the ratio of the effort required to prepare a full proposal following an outline proposal to the effort required to prepare the relevant outline proposal.)

For the main simulation, realism_beta_1 is set to 5. It combines with realism_beta_2 to give a Beta distribution with mean 0.333 and standard deviation 0.118 (that is, quite a broad distribution.) To test the effect of these assumptions on the model, realism_beta_1 is allowed to vary uniformly in the range 1 (mean value of resulting distribution is 0.091) to 100 (mean value 0.909). The effect this has is shown in figure A1 below.

Figure A1. Variation of the difference in the percentage of simulated applicants applying to calls of the two types with variation in applicant ‘realism’. Realism is expressed on the x axis as the percentage of the true effort that applicants perceive. The value used as a central value in the model is shown with a dotted vertical line.

The result is presented as a smoothed regression relationship between the ‘effort pessimism’ (defined as the mean of the relevant Beta distribution defined by realism_beta_1/(realism_beta_1 + realism_beta_2, so that greater pessimism leads to less discounting) and the mean and median difference in the number of applicants applying to ‘Outline + full’ and ‘Straight-to-full’ calls in each of 500 simulations with differing starting values of realism_beta_1. As expected, the more pessimistic (or perhaps realistic) applicants are when it comes to assessing the likely effort required to prepare a full proposal, the smaller the difference in the fraction of them applying to the two types of call is. This indicates that, as required, the relative attractiveness of the two types of call depends on the extent to which applicants are willing/able to ignore the fact that the outline stage is not the end of the effort required.

In an analogous way, the results of varying full_proposal_ratio uniformly between 0.1 and 10 are shown in Figure A2.

Figure A2. Variation of the difference in the percentage of simulated applicants applying to calls of the two types with variation in model full_proposal_ratio. The value used as a central value in the model is shown with a dotted vertical line.

Varying full_proposal_ratio has a weak effect on the relative attractiveness of the two call types. Larger values tend to make ‘Outline + full’ processes relatively more favoured, but the trend is slight. In the absence of any real guide to the best value to use, a value of 1 is used in the simulations, but it is allowed to vary across applicants as already described.

6 Input data

The model does not use input data to represent time-varying processes.

7 Submodels

There are no submodels.