Introduction

Quantitative studies in health and medical research require competent study design and statistical analysis to avoid flawed results. As the eminent statisticians Bland and Altman wrote: “Bad statistics makes bad research, bad research may lead to bad medicine, and bad medicine may cost lives” [1].

Unfortunately, errors are common in the study design and statistical methods of published health and medical research studies, including inadequate or unclear planning [2], inadequate statistical power [3, 4], errors in methods [2, 5–7], and inadequate reporting [7–10]. These errors are an ethical concern because a poorly designed study has little chance of answering its research question, potentially exposes participants to unnecessary risks, and can harm future patients if misleading evidence is translated into practice [11, 12].

Australian health and medical researchers working with human participants require approval from a Human Research Ethics Committee (HREC) for studies involving more than low risk. The Australian National Statement on Ethical Conduct in Human Research (hereafter called the “National Statement”) states that applications to human committees must detail how the study is “designed or developed using methods appropriate for achieving the aims of the proposal” [13, Section 1.1b].

Therefore, in their evaluation of research proposals, Australian ethics committees have the power to prevent poorly designed studies and thereby reduce research waste and prevent unnecessary risk to participants [14, 15]. However, committee members may lack the competency to critique study design, as it is not mandated that ethics committees include a qualified statistician [13, 16]. The relevant membership requirement for human research ethics committees is: “two people with current research experience that is relevant to research proposals to be considered” [13]. This definition could mean a statistician, but could also mean other researchers such as clinicians and epidemiologists.

It is unclear how and whether statistical expertise is integrated into the important work of Australian human research ethics committees. Without statistical expertise, there is the risk that inadequately designed studies are approved. Our aim was to discover whether Australian human research ethics committees have access to qualified statisticians.

We know of two previous national surveys that examined whether statisticians were included in ethics committees: in the UK in 2000 [17] and Canada in 2005 [18]. We believe that an Australian study is worthwhile because practice is likely different in Australia and local data are more likely to inform national policy.

Another reason for an Australian study is the 2023 update of the National Statement, which included the change shown below with the key differences highlighted from the previous 2018 version [19]:

The change in wording from “must see” to “must ensure” implies a stronger scrutiny of study design is required. We were interested in how ethics committees without a statistician would interpret this change.

Study scope

This study considers quantitative research involving human participants in Australia that requires ethical approval.

We excluded animal research because there is no publicly available list of Australian animal ethics committees. We recognise that poor quality studies have been highlighted in preclinical research [20].

Human research ethics committees do not cover all health and medical research. Some research that involves statistical analysis does not require ethical approval, for example, analysis of data that are deidentified and publicly available. Studies approved by an ethics committee may not have fully described their study design or included their statistical analysis plan. Hence, having qualified statisticians available to all ethics review committees would not mean that all medical research would have appropriate statistical scrutiny.

We did not compare the requirements in Australia to those in other countries.

Methods

We aimed to discover whether Human Research Ethics Committees in Australia had access to qualified statisticians. We used a census by approaching all Australian Human Research Ethics Committees registered with the National Health and Medical Research Council (NHMRC).

Our study protocol is available on the Open Science Framework [21]. The study was approved by the Human Research Ethics Committee of the Queensland University of Technology (Date: 29 April 2024, Number: LR 2024-8420-18539).

Census

Australian universities and medical research institutes that have oversight of research must register their ethics committee(s) with the NHMRC. We tried to contact all Human Research Ethics Committees registered with the NHMRC [22]. We used the list of registered committees that was updated on 16 May 2023.

The single inclusion criterion was all current Human Research Ethics Committees in Australia. The exclusion criteria were: Human Research Ethics Committees that have ceased operations and Animal Ethics Committees.

We emailed the chair of each committee. We used the committee’s general contact email if we could not find the chair’s email. The email with a unique link for each committee was sent by investigator Barnett on 20 May 2024 and included the option of responding by telephone.

Reminder emails were sent to non-responders after two weeks, and postal reminders were sent to non-responders after four weeks.

The two previous national surveys that examined whether statisticians were included in ethics committees had response rates of 82% in the UK [17] and 61% in Canada [18]. Therefore, we anticipated a response rate of 70%.

Questions

The questions were designed by the authors (see Appendix Appendix 1 for the full list of questions). Some questions were based on the two previous national surveys [17, 18]. We received feedback on our draft questions from two current chairs of Australian Health Research Ethics Committees, a research office manager, and a statistician who was a current ethics committee member.

Our key question was whether the committee has access to a qualified statistician. We were also interested in how the committees dealt with statistical issues. We asked committees without a statistician why they did not have one and whether they considered that they needed one.

We investigated whether committees had access to qualified statisticians as opposed to health professionals or researchers with some statistical experience. This is because there is a varying degree of statistical expertise in the research community, and whilst experience with statistical analysis is valuable, we do not view this as equivalent to the advanced, theoretical understanding of statistical principles which qualified statisticians typically possess.

We collected the total number of annual applications seen by the committee with quantitative data and/or analysis. The aim was to estimate the national number of applications that may not have received adequate statistical scrutiny.

The questions were delivered online using Qualtrics. Respondents could skip any question except the initial consent question.

Qualitative analysis

We anticipated variance in views and attitudes about the nature and importance of statistical expertise in committees’ work [23]; hence, we included multiple open questions that allowed respondents to explain their involvement with statisticians and their views on the importance of statistical expertise.

Text responses were thematically analysed by inductive and deductive coding. For inductive coding, we drew on the notion of Anthropological Strangeness of Latour and Woolgar [24], to understand the scientific landscape being described from a position of ignorance.

For deductive coding, we relied on Jasanoff’s characterisation of scientific expertise, wherein expertise is understood as “something acquired and deployed within particular historical, political, and cultural contexts” rather than being a “thing” located in the “heads and hands of skilled persons” [25].

Statistical methods

Our analyses are descriptive. The main outcome is the percentage of committees that have a statistician as a full member and provide the statistician’s qualifications. The secondary outcome is the percentage of committees that have access to statistical expertise, either as a full member or non-member, and provide the statistician’s qualifications. To account for the uncertainty due to missing responses, we used a bootstrap procedure that imputed the responses of missing committees using the available responses and calculated 95% bootstrap confidence intervals that accounted for the inflation in variance due to this imputation [26].

We estimated the number of applications reviewed per year without statistical oversight. This used the number of applications including quantitative data and/or analysis that are viewed by committees without access to expertise from a qualified statistician. As committees may not know their application numbers exactly, we allowed respondents to give an upper and lower range, and these ranges and the mean were used in a triangular distribution [27]. We used a bootstrap 95% confidence interval for this figure that included uncertainty in the committee application number estimates, and uncertainty due to missing data after imputing the numbers for committees who did not respond using the available responses [26].

Some large institutions had multiple ethics committees, but we did not adjust for any non-independence as we assumed that all committees are largely autonomous.

Results

Contact details and response rates

The number of committees included and excluded is shown in Figure 1. Five committees were marked closed in the sampling frame and another two were found to be closed after we searched for contact information. Finding contact details was often challenging, and there were 40 web links (21%) in the sampling frame that did not link to the committee’s page.

The web page of one committee promoted an online tool that researchers could use to help them choose a statistical test [28]. Unfortunately, this tool selects methods based on the statistical myth that data often need to be normally distributed [29] and recommends repeated measures ANOVA without warning about the need for balanced data [30].

Out of 187 emails sent, 154 (82%) respondents opened the first page of the online questions that provided the study information (Appendix Appendix 1). However, 33 respondents did not progress beyond this first page and a further 27 clicked through the questions without providing any answers.

The first invitations were sent via email on 20 May 2024, with an email reminder to non-responders on 3 June, and posted reminder letter on 26 June. The cumulative responses over time are plotted in Appendix Appendix 2.

We received responses from 93 committees out of the 187 that were not closed, giving a response rate of 50%. Two committees withdrew. Our response rate was much lower than our anticipated response rate of 70%, and lower than the two related international studies (61% and 82%) [17, 18].

One response was by telephone and one completed a Word version of the questions; the rest were online. The median time to answer the questions was 32 minutes with a 10th to 90th percentile of 4 minutes to over 378 hours. These very long times are due to respondents leaving the online questions open for multiple days. The median time of 32 minutes is surprisingly long and indicates that most of the respondents likely paused their responses to gather information. The number and percent of missing answers for each question are in Appendix Appendix 3.

The majority of respondents (66%) were the chair, the next most being the coordinator or the officer. One response was excluded because it was from the ex-chair.

Figure 1: Flow diagram of committees included in the study. HREC = Human Research Ethics Committee

Answers to questions

Thirty-one committees (33%) reported having a statistician as a full member (Table 1).

Thirty-three committees (35%) reported having a statistician who was not a member but who could be consulted on statistical issues. Our main outcome combined the committees’ reporting of access to a statistician with their reporting of statistical qualifications, and only 25% of committees had a qualified statistician, down from the 33% estimate without considering qualifications.

For committees who reported having a statistician, 12 (23%) did not know the statistician’s qualifications (excluding 4 committees who did not answer this question). Most of the eight committees that ticked “Other qualifications” listed qualifications outside of statistics. Of the fifty-five committees with access to a statistician, 10 (18%) reported that they were not currently employed as a statistician. Ten committees (18%) answered “Other” to the employment question, but only one provided details on their employment as a statistician.

We combined the committees’ access to a qualified statistician with their estimated application numbers. This gave a national estimate of 8,064 applications per year that included quantitative data and/or analysis without the assessment of a qualified statistician (95% bootstrap interval 6,481 to 9,946). This was greater than the estimated 5,233 applications per year from committees with a qualified statistician (95% bootstrap interval 4,377 to 6,186). So, under 40% of applications potentially had an assessment from a qualified statistician. We can partially triangulate our estimate using the national figures gathered by the NHMRC from all committees, which gave 13,284 proposals considered in 2023 [31], which is close to our mean estimated total of 13,297.

The responses from committees that did not have a statistician as a full member are summarised in Table 2. The majority (67%) of the committees without a statistician had never had one. The modal answer to how statistical issues are handled was “Within committee” (33%) and the main reason for not having a statistician was that other committee members could cover statistical questions (see qualitative analysis below). Seventeen percent of the committees without a statistician reported that they needed one. Of the committees who did not think they needed a statistician, only one out of 39 (3%) agreed that the 2023 change to the National Statement wording altered their opinion on the need for a statistician on the committee.

Table 1: Responses to the two questions on the committees’ access to statisticians, with separate columns with and without considering statistical qualifications (n = 93). The cells show the number and percent with a 95% confidence interval. The main outcome is shown in bold and the secondary outcome in italics. Due to rounding, not all percentages add to 100%.

For the committees who answered that they had access to a statistician as a full member or who could be consulted, we asked them how statistical issues are dealt with (Table 3). If the statistician was concerned about the data collection and/or analysis, then these concerns were taken seriously by almost all committees, as 84% would always discuss the concerns and 71% would always send the application back to the researchers to address the statistician’s concerns.

Table 2: Numbers and percentages for the questions to committees who did not have a statistician as a full member of the committee (n = 52). a Respondents could tick multiple responses for this question; hence, the percentages do not add to 100%.

Table 3: Two questions on how statistical issues are dealt with for those committees who said they had access to a qualified statistician (n = 56). The scenario was: “Consider a hypothetical application. If the statistician advised the committee that the proposed method of data collection and/or analysis were not appropriate to answer the research question(s) then what actions are taken.”

Qualitative analysis of text responses

Four questions asked for a free-text response (views on implications of the National Statement wording; comment on the role of statisticians within ethics committees; general comment in relation to the topic; and specific feedback on the questions; see Appendix Appendix 1). Five additional questions allowed a free text response (‘other’), and these data were combined with the four open questions for qualitative analysis. The analysis focused on unpacking the ways respondents understand and perceive the role of statisticians in the work of ethics committees, according to the results in Table 2.

Seven themes span all questions, discussed below in two groups: i) beliefs and views about the role, scope, and form of ethics committees, and ii) beliefs and views about statisticians and the nature of statistical expertise.

i)   The role, scope, and form of the Ethics Committee

Perhaps not surprisingly, beliefs about the value and potential contribution of statisticians on ethics committees depended transparently on perceptions of the committee’s role and scope. The belief that statisticians are not essential for this work seemed to be underpinned principally by three beliefs held by some respondents.

1. Scope of committee responsibility

Not all respondents believe that the scrutiny of statistical methods falls within the scope of the duties of an ethics committee. Several respondents stated this directly, as in the following three examples:

“Detailed Statistical analysis – as part of ethics approval – is not assessed.”

“It is NOT up to the committee to design the research.” (their emphasis)

“Our decisions must be limited to the ethical issues.”

Others presented a contrasting view, that statistical methods and study design are ethical issues requiring careful and routine consideration:

“It would be unethical to involve participants and expend resources on research that was not able to produce reliable results.”

Others articulated a half-way position, indicating that method and design may sometimes (but not always) require comment from the committee:

“We do not comment on design and method unless it raises an ethical risk or concern.” (emphasis added)

These responses suggest marked diversity in the way that committee members understand the scope of their committee’s responsibility when it comes to reviewing statistical details of planned studies, with obvious consequences: If statistical details are considered out of scope, the presence of a statistician member is not warranted. This was also related to the perception of the researcher’s own responsibility and the committee members’ trust in the researchers.

2. Trust in researchers

Some respondents indicated that, in general, they trust researchers to ensure that the design of proposed studies is appropriate. As one respondent said, applicants “have it worked out”.

One respondent stated that they “relied on researchers to arrange appropriately powered treatment groups”, others noted that “it is expected that researchers consult with statisticians [regarding] data collection and analysis”. Comments such as these were often the prelude to an expression of the view that studies requiring review by a qualified statistician are “rare”.

Additionally, the expectation that researchers should consult a statistician before submitting their ethics application seemed disconnected from a widely expressed belief that accessing statisticians is difficult, discussed below.

3. Diversity of processes

Some respondents noted that, while their committee may not have a statistician member, structures and processes are in place to ensure the quality of approved studies. The complete set of mechanisms is shown in Table 4, ranging from the most formal to the expressly informal.

The diversity of these structures and processes is significant, highlighting the varied ways of working among committees across Australia. Any effort to increase the participation of qualified statisticians in the work of ethics committees must be tuned to this wide-ranging set of existing processes.

ii) Beliefs about Statisticians and Statistical Knowledge

Beliefs about the value and potential contribution of statisticians on ethics committees seemed to depend on and reflect the beliefs of members about statisticians and statistical knowledge. The belief that statisticians are not essential committee members seemed often to be underpinned by one or more of the following beliefs:

• Members with research experience have expertise which is equivalent to a statistician
• Statisticians are too hard to recruit, and/or
• Having a statistician on a human research ethics committee is of limited value Contrasting views on all three topics were presented, which are explored below. Applied research experience as equivalent to formal statistical expertise

Several respondents who do not have statisticians on their committees indicated that scientific training in applied disciplines (e.g. psychology, epidemiology, medicine) or general “research experience” is equivalent to the expertise of a qualified statistician. When explaining why statisticians are not members of their committee or are not consulted, there were abundant references to “members who use statistics as a method of data analysis”, “highly numerate” members, “researchers who are familiar with stats” or members with “a very good grounding in statistics”. For these respondents, having experienced researchers on the committee was sufficient:

“Many researchers have a very good grounding in statistics [. . . ] As long as there is expertise in the [study] area then a ‘statistician’ as such is not necessarily needed.”

“It has never felt like a gap in the committee’s collective knowledge, perhaps because we have a number of quant researchers with sufficient statistical expertise to assess relevant applications.”

This last response reflects the informal way that these issues appear to be navigated on a committee-by-committee basis. An individual chair’s impression of what their committee reviews, requires, and lacks was obviously key to decision making about the participation of statisticians, and the decision to engage a qualified statistician depends on the members’ perception of their own statistical literacy. These data align with Jasanoff’s model of scientific expertise [25], where the label “expert” is flexible, relative and contextually dependent. As statisticians, we disagree with the idea that research experience and advanced statistical training are equivalent; however, this is of little importance, as our data suggest this belief is reasonably widespread and substantively underpins decision-making around how and whether to involve statisticians in ethical review of research proposals.

Issues of access

Several respondents felt that recruiting a statistician to their committee was impractical, due to the perceived challenges of accessing statisticians and statistical expertise [32]. The following two quotations are illustrative:

“It would be lovely to have a statistician but they are as rare as hens’ teeth.”

“It would be nice to have a statistician on / available to the committee but availability and cost would be factors”

Clearly, some committees face practical barriers when recruiting statisticians. However, these qualitative comments are somewhat in contrast to the finding that only five committees without a statistician said they could not recruit one (Table 2).

There is no current national mechanism that enables committees to contact a pool of willing statisticians, but some respondents directly expressed a wish for such a mechanism:

“Having statisticians as available experts when needed, who could assist any HREC rather than being attached to just one, would be more beneficial [than statisticians serving as full members on a single committee].” (emphasis added)

“They [statisticians] may not best serve the broader community if they spread themselves too thinly, by taking up membership on ethics committees. They may, however, inform HRECs of their willingness to consult on a needs basis. I suspect that at least some HRECs would welcome this.”

Therefore, an existing pool of ‘willing’ statisticians could fill an important gap. However, evidently not all committees would welcome such a mechanism, as respondents did not unanimously believe that the input of a statistician is valuable.

Sense of a statistician’s value

There is substantial heterogeneity of opinion between committees regarding the value of advice from a qualified statistician. At one extreme, several respondents described access to qualified statisticians as “critical”, “crucial”, and “vitally important” for the activity of the Committee. The following quotations are illustrative of this view:

“In my experience their involvement is critical. While I have some statistical expertise myself as an experienced researcher, the statistician’s expertise has been important in identifying some flaws in a proposal on many occasions that would otherwise have been missed.”

“[They are] important for HREC, as it [having a statistician] does provide you with a level of confidence, as Chair, that the application is rigorously and fairly reviewed.”

Sharply contrasting views were also expressed. One committee chair indicated that a statistician is “not needed for 99.9% of applications” and this committee reported 250 annual applications that included quantitative data and/or analysis. Others stated that statistical input is “not a necessity”, “rarely” a necessity, or that they “do not see the need for a statistician for the work that comes before [our] committee”.

If some committees view statisticians as indispensable, and others view statisticians as fairly useless, then this is an interesting state of affairs suggestive of deep heterogeneity in Chairs’ understandings of how often, and in what ways, statisticians might contribute to the work of an ethics committee.

A quantitative taxonomy? Sense of proposals that do and do not require statistical scrutiny

Several respondents expressed a strong belief that their committee only deals with proposals that do not require close statistical scrutiny. In this way, the perceived low value of a statistician was directly and obviously related to a deeper set of beliefs about whether all quantitative research proposals require statistical review – since it is assumed that there are ‘kinds’ of proposals, and not all kinds are understood to require the input of a statistician. This was related to perception of the existence of tiers, or types of quantitative research projects that merit statistical review.

The projects that needed statistical review were those with “high level analysis” or “data intensive research”, while those that did not need statistical review were those using “basic statistics” or large clinical trials with in-house statisticians.

Reviewing this proposed taxonomy, it does seem sensible that respondents assume internationally-sponsored trials with internal statistical teams to have sound research designs. However, other respondents implied that, in general, ethical approval of randomised controlled trials should include statistical review:

“Our committee does not typically see many clinical trials [. . . ] where the services of a qualified statistical expert is necessary.”

“In HRECs dealing with interventional research a statistical member is essential.”

We also note a recent review of published clinical trials which found that only 8% were at low risk of bias [33].

The view (sometimes stated directly and sometimes implied) that observational studies do not pose statistical difficulties or require the input of a statistician was surprising and concerning to us. The three authors have a combined 52 years of experience of supporting clinicians in designing and conducting research, we know that observational studies are often the most challenging to design and analyse. There are many papers warning of common errors in observational studies, for example, [29, 34–38], and the view that observational studies do not need statistical input conflicts with the consensus among the statistical community.

Similarly, there were references to “large scale” projects, understood to exist as a category and to require statistical review, implying that “small scale” projects do not. These large-scale projects might be real or hypothetical:

“They [statistician] only attends when large-scale, data intensive research is being undertaken.”

“If the research community at our institution was engaged in more large-scale projects, we would certainly reconsider this stance [not having a statistician].’

In our experience, there is no agreement among researchers on what constitutes a “large” study and this will also vary by field. Even if there were agreement, small studies often require substantial skill to design and analyse, to carefully balance the desire to fully interrogate a question and the weaknesses of a small sample. The view that a statistical review is not required for small studies is therefore concerning and suggests that educational resources on this point would be valuable.

References to studies employing “basic”, “high level” or “advanced” statistical methods were numerous, implying that these categories of studies exist and that studies using simple statistical methods cannot be poorly or inappropriately designed. As statisticians, we strongly oppose this idea on the basis of substantial experience. Further investigation of what kinds of methods are understood to be “basic” or “advanced” would be illuminating, as it seems highly unlikely that researchers, committee members, and statisticians are in agreement on this point.

One respondent indicated that projects that disclose an intention to publish results in peer-reviewed journals could be safely assumed to be of sound study design. This was surprising as studies with flawed designs are routinely published [33, 39], sometimes in prestigious journals [40–43].

If quantitative research proposals are presupposed to be easily sortable into these different categories: large and small; basic and advanced; interventional and observational; destined for publication and not; and it is sincerely believed that not all categories pose statistical risks, then this provides important context and explanation for comments indicating that statistical input is rarely required or useful for some committees. It appears that several respondents believe that the input of a statistician is only necessary or valuable for a certain type of quantitative project. If statisticians or national bodies wish to see a greater engagement with statistical experts by committees, they should begin by seeking to modify this belief.

Limiting statisticians to statistical power

Statistical expertise was often closely associated with, or understood to be limited to, issues of sample size and statistical power. Not encountering issues with power and sample size was presented as an explanation for the absence of statisticians on the committee by more than one respondent:

“Many studies [we consider] employ established techniques where numbers of participants have been established by long-standing practices.”

Responses such as these further contributed to the sense that some committee members have a narrow view of how statisticians might contribute to the work of an ethics committee. Although, when viewed alongside the argument that small studies do not require statistical scrutiny, there is some disconnect here that merits further investigation.

The National Statement

How were these issues expressed and reflected in comments about the change to the National Statement? With one exception, discussion of changes to the National Statement seemed to recapitulate the above issues. With the following extract neatly summarising multiple key results already discussed:

“It [adding statisticians to the committee] is not a priority for us at the moment. We have experienced researchers on the Committee. Most research we review use relatively basic statistics.”

The equivalence of research experience and statistical expertise was again clearly expressed, with several respondents arguing that no changes are required on the basis of the new wording, due to the “depth of knowledge” of their members, and expressing the view that “experienced researchers on committee [. . . ] enable evaluation of project design”. Other respondents noted that only “a minority of projects need a stats opinion”, so no substantive change was required.

Other respondents noted that there are various ways to “ensure” that research is well designed, and while having a statistician member might be ideal, this is not made explicit in the statement, so other strategies and processes for accessing if statistical advice are sufficient. Other respondents indicated that they had always interpreted the 2018 statement to imply what is now explicit in the 2023 statement, so no change was necessary.

However, discussions of the updated wording in the National Statement did contain a new finding; namely that some committee members understand the responsibility as relating exclusively to institutions, rather than ethics committees:

“This to me reads as an institutional responsibility rather than a Committee responsibility, aligned with the a purpose of updating the National Statement which, if I recall correctly, was to refine institutional responsibilities.”

“The statement “Institutions must” is about the institution’s responsibilities, and does not directly need to be discharge [sic] through a HREC or subcommittee membership.”

Therefore, in addition to believing that responsibility for the design of a sound study lies with the researcher or a trial sponsor, some respondents believe that this responsibility ultimately lies with the institutions. However, some institutions may believe that they are fulfilling this responsibility via the ethics committee. Hence, the important task of examining the study design may be overlooked due to shifting responsibilities.

In summation, the act of engaging a qualified statistician to assist a human research ethics committee clearly depends on deeper attitudes. These attitudes are related to the scope of the work of a committee and the location of responsibility for rigorous study design. The substantive pre-requisites for engaging a statistician are already having access to a qualified statistician, belief that statisticians bring expertise beyond those with applied research experience, and a sense that a statistician can contribute something of value. Committees reported richly diverse processes for accessing statistical advice (Table 4), which require consideration for any efforts to make change. Finally, the view that not all quantitative projects require statistical scrutiny – a view with which most statisticians would disagree – seems fairly widespread and was a common justification for the absence of a statistician.

Discussion

We found that 25% of Australian Human Research Ethics Committees included a qualified statistician member, and we estimate that approximately 40% of applications receive qualified statistical scrutiny. These are concerning results given the centrality of quantitative methods in health and medical research.

Three-quarters of surveyed committee chairs without a statistician member reported they do not feel they need one, and reasons for this belief were diverse. Some committees, mostly based within large health services or universities, reported separate, formal structures and processes which facilitate access to statistical advice. For others, the Chair’s personal networks were engaged on an as-need basis. Others reported that the committee never engages statistical advice. Qualitative data revealed that these latter groups operate via an implicit hierarchy of studies which do and do not require statistical review.

Such a hierarchy is not detailed in the National Statement. This informal categorisation mechanism – never before described – appears to be driving inconsistent practice among Australian committees. Future studies could explicitly map this typology and evaluate it against evidence of where design flaws actually occur. It has long been established that statistical errors are not limited to large or complex studies [38, 44, 45], and include those that use relatively simple methods such as linear regression [6, 46].

The diversity we describe in access, mechanisms (Table 4), and beliefs represent compelling evidence of systemic inconsistency. Without standardisation, committees are making idiosyncratic judgments about what needs statistical review, and what counts as statistical review. This variability also reflects fundamentally different interpretations of the ethics committee’s role in evaluating research, and raises questions about whether current practices consistently align with the National Statement’s expectations regarding research quality.

Our findings have implications for research quality and participant protection, because poorly designed studies expose participants to risk without generating reliable knowledge [11].

Statistically unsound research is also inefficient, as flawed studies waste resources and participant time. The cumulative effect of these issues in Australia is unknown, but is likely significant if 60% of applications lack qualified statistical review.

Comparisons with previous studies

Our estimate of 25% of committees with a qualified statistician member is higher than previous international reports of 15% for human ethics committees in 1997 in the UK [17] and 22% for human and animal ethics committees in 2003 in Canada [18].

For committees without a statistician, dealing with statistical issues “within committee” was the most common response for our study and for the two previous national studies of ethics committees, but with higher percentages of 51% in Canada and 67% in the UK compared to our finding of 33% [17, 18].

In terms of committee scope, a qualitative study in the USA similarly found that some members of the Institutional Review Board saw the scientific merit of a study as outside their scope, despite regulations stating that a scientific merit evaluation is required [47]. A related Canadian study also encountered a similar opinion: “Ethical decisions do not depend on such detailed scrutiny” [18]. An Australian study that examined committee member roles quoted one member who reported being unwilling to “tinker” with methods while also stating, “You don’t allow bad research through” [48]. This study also found that some committee members had not read the National Statement and therefore may not be aware of their role in reducing the potential harms to the wider community from flawed research, and instead make decisions based on the risks to participants and their institution [48].

Similarly to the variance in practice that we found, a study of the standard operating procedures of Institutional Review Boards in the USA found a variance in the coverage of scientific merit, with significant gaps in areas critical to the evaluation of study quality [49].

Previous studies have discussed how some colleagues in health and medical research have a narrow view of the value of statisticians as colleagues “who can do computers” [50]. This discussion has spanned decades, with a 1974 paper debating whether statisticians are fellow scientists or “shoe clerks” who simply provide a service [51].

A recent study examined whether ethics committees requested sample size calculations, as recruiting too few participants is a common cause of research waste [52]. They found that only 12 out of 20 committees requested that researchers justify their sample size.

Statistics is not the only area where human research ethics committees can lack expertise. A recent study of all Australian human research ethics committees found a low to moderate confidence in committee members when examining applications concerning genetics [53].

In 2000, The Netherlands required that all institutional ethics committees include a methodologist, which led to the closure of some small hospital committees as they could not find a suitable researcher [54].

Our recommendations

More oversight from qualified statisticians is needed at the ethical review stage. Scrutinising studies before they begin could minimise research waste by highlighting design errors when they can still be corrected. We support a previous recommendation, aimed at reducing research waste, that randomised trials should not receive ethical approval unless the trial team contains methodological and statistical expertise [33]. We extend this recommendation to clinical prediction models, as these are a commonly used study design where avoidable errors frequently undermine the value of the study and potentially cause harm if poorly developed models are used in clinical practice [55–57].

This recommendation is in accordance with a 2024 change to the Declaration of Helsinki [58]:

“Medical research involving human participants must have a scientifically sound and rigorous design and execution that are likely to produce reliable, valid, and valuable knowledge and avoid research waste.”

We welcome this change, which recognises that the problems of research waste are an ethical issue. We note that this change occurred after we completed our data collection and so we could not examine how committees reacted to this change.

We strongly recommend that the current variance in practice between Australian human research ethics committees is reduced. This could start with a national discussion to determine whether study design should be part of a committee’s deliberations. If there is agreement that study design is not the remit of human research ethics committees, then many institutions would need to provide greater statistical support, and many current application forms could be shortened to remove questions on study design. In contrast, if design is considered part of the process, then many committees would need to change practice to maintain national standards.

Before any standardisation, the current variance in practice between committees should be transparent. All committees should make a public statement about whether they consider the study design. Institutions should be aware of whether the committees their researchers use are evaluating the study design, as they may need to create systems outside the ethics committee to support their researchers to create sound study designs that can answer their research questions.

We support the recommendation to reduce the number of ethics committees in Australia and concentrate expertise in fewer committees [59], as this may help more committees include statistical expertise. We also recommend examining whether committee members and external experts should be paid, as this may help committees recruit statisticians. We note that some Australian committees already pay their members, which is another variance in practice. Some researchers and institutions may not want to pay for a service that has long been free, but the current system is potentially creating enormous downstream costs. For example, the costs of poorly designed trials published in one year were conservatively estimated at £726 million [33]. Money spent preventing poorly designed studies at source could therefore quickly pay for itself.

The ability of committees to spot serious statistical flaws could be examined using a “mystery shopper” exercise, in which a mock application is written and submitted to multiple committees [60]. The “mystery shopper” approach, and shared ethical debates, revealed a large variation between committees in how they evaluated applications [60]. This approach could reveal the large variation in statistical oversight and may provide first-hand evidence to committees about the value of statistical expertise.

Limitations

Contacting every committee was difficult, and there are likely committees that never received our email approach.

Our response rate of 50% was lower than we expected. There could be some non-response bias as committees without a statistician may have been more reluctant to respond, which would mean our results have overestimated the availability of statisticians.

Our approach emails received many automated responses stating that the committee was currently very busy, and hence some non-response may be due to committees being swamped with their regular work.

Some respondents reported difficulty in answering the questions, as multiple people from the same committee needed to access the questions, but our online system was not set up for this. This need for multiple access was because some questions, such as the annual number of applications, may have been better answered by the secretariat, whereas other questions were better answered by the chair.

It is interesting that the questions took so long to complete and that many committees clicked through the questions without providing answers. It is possible that our questions were difficult to complete as they involved searching for information and contacting multiple committee members.

We appreciate that members of human research ethics committees devote their time to considering research and the safety of research participants, and this is a time-consuming and often difficult task. Our critiques are intended to improve the system as we see it and are not directed at individuals or committees [61].

Conclusion

We have found a large variation in practice in a crucial part of the research process in Australia. Some ethics committees view statistical expertise as essential, whilst others are either not examining the study design or relying on non-statistician researchers to provide statistical advice. This situation risks approving studies that will in the best case waste resources and in the worst case create harms due to flawed evidence.

Data availability statement: All quantitative data are available in a GitHub repository [62]. We could not share qualitative data because it may be possible to identify individuals.

Competing interests: The three authors are statisticians and current members of the Statistical Society of Australia; the society was not involved in this study. We acknowledge a potential professional conflict of interest as we could be seen to be promoting our own field and seeking to gain an advantage.

Adrian Barnett is a current member of the national Ethics Committee Advisory Group. This research was carried out independently of this work.

Author contributions statement: A.B., T.C., and N.W. designed the questions. A.B. deployed the questions and managed the data. A.B. analysed the quantitative results, and T.C. analysed the qualitative results. A.B., T.C., and N.W. wrote and reviewed the manuscript.

Funding: None.

Acronyms and initialisms: ANOVA = Analysis of variance; HREC = Human Research Ethics Committee; NHMRC = National Health and Medical Research Council.

Acknowledgments: Thanks to the two committee chairs, research officer, and statistician who provided feedback on our questionnaire.

Appendices

Appendix 1     Participant information and questions

The participant information sheet is available on the Open Science Framework : https://osf.io/steyx.

A PDF version of the questions is available on the Open Science Framework : https://osf.io/9we7r. Participants completed an online version.

Appendix 2    Responses over time

Figure A.1: Cumulative responses from the human research ethics committees over calendar time. We sent one email reminder and one postal reminder to non-responders.

Appendix 3 Item-missing data

Table A.1: Number and percentage of missing responses for each question. The questions are ordered by the maximum number of responses and then the percent missing. The first column is the question number. Questions 3.1 and 3.2 were two alternatives to provide the same information and hence we expected some missing answers here.

The table shows the number and percentage of missing responses per question for all 16 questions. The questions that people saw depending on their previous answers, hence we also show the maximum number of responses. The table is ordered by the maximum number of responses and then the percent missing. The first column is the question number on the online questionnaire.

References

[1]          Bland, J. M. and Altman, D. G. “Misleading Statistics: Errors in Textbooks, Software and Manuals”. International Journal of Epidemiology 17.2 (1988), pp. 245–247. DOI: 10.1093/ije/17.2.245.

[2]          Van Calster, B. et al. “Methodology over metrics: current scientific standards are a disservice to patients and society”. Journal of Clinical Epidemiology 138 (Oct. 2021), pp.  219–226.  DOI: 10.1016/j.jclinepi.2021.05.018.

[3]          Lamberink, H. J. et al. “Statistical power of clinical trials increased while effect size remained stable: an empirical analysis of 136,212 clinical trials between 1975 and 2014”. Journal of Clinical Epidemiology 102 (2018), pp. 123–128. DOI: https://doi.org/10.1016/j.jclinepi.2018.06.014.

[4]          Zwet, E. W. van and Goodman, S. N. “How large should the next study be? Predictive power and sample size requirements for replication studies”. Statistics in Medicine 41.16 (2022), pp. 3090–3101. DOI: https://doi.org/10.1002/sim.9406.

[5]          Stark, P. B. and Saltelli, A. “Cargo-cult Statistics and Scientific Crisis”. Significance 15.4 (July 2018), pp. 40–43. DOI: 10.1111/j.1740-9713.2018.01174.x.

[6]          Jones, L., Barnett, A., and Vagenas, D. “Linear regression reporting practices for health researchers, a cross-sectional meta-research study”. medRxiv (May 2024). DOI: 10.1101/2024.05.28.24308029.

[7]          Riley, R. D. et al. “On the 12th Day of Christmas, a Statistician Sent to Me . . .” BMJ 379 (2022). DOI: 10.1136/bmj-2022-072883.

[8]          Lang, T. A. and Altman, D. G. “Basic statistical reporting for articles published in Biomedical Journals: The “Statistical Analyses and Methods in the Published Literature” or the SAMPL Guidelines”. International Journal of Nursing Studies 52.1 (2015), pp. 5–9. DOI: https://doi.org/10.1016/j.ijnurstu.2014.09.006.

[9]          White, N. M. et al. “An observational analysis of the trope “A p-value of < 0.05 was considered statistically significant” and other cut-and-paste statistical methods”. PLOS ONE 17.3 (Mar. 2022), pp. 1–15. DOI: 10.1371/journal.pone.0264360.

[10]       Héroux, M. et al. “Poor statistical reporting, inadequate data presentation and spin persist despite Journal awareness and updated Information for Authors”. F1000Research 12 (Nov. 2023), p. 1483. DOI: 10.12688/f1000research.142841.1.

[11]       Altman, D. G. “Statistics and ethics in medical research: Misuse of statistics is unethical”. BMJ 281.6249 (Nov. 1980), pp. 1182–1184. DOI: 10.1136/bmj.281.6249.1182.

[12]       Gaines Das, R. E. ““Statistics” is not a sausage machine: a statistician’s viewpoint and some comments on experimental design”. Alternatives to laboratory animals: ATLA 32 Suppl 2 (2004), pp. 5–8. URL: https://api.semanticscholar.org/CorpusID:20687699.

[13]       National Health and Medical Research Council, Australian Research Council and Universities Australia. National Statement on Ethical Conduct in Human Research 2023. Reference number: E72C. 2023. URL: https://www.nhmrc.gov.au/about-us/publications/national-statement-ethical-conduct-human-research-2023     (visited on 11/09/2024).

[14]       Vail, A. “Experiences of a biostatistician on a U.K. Research Ethics Committee”. Statistics in Medicine 17.24 (Dec. 1998), pp. 2811–2814. DOI: 10.1002/(sici)1097-0258(19981230)17:24<2811::aid-sim22>3.0.co;2-1.

[15]       Altman, D. G. “Poor-Quality Medical Research: What Can Journals Do?” JAMA 287.21 (June 2002), p. 2765. DOI: 10.1001/jama.287.21.2765.

[16]       National Health and Medical Research Council. Australian code for the care and use of animals for scientific purposes (8th edition). Reference number: EA28. 2013. URL: https://www.nhmrc.gov.au/about-us/publications/australian-code-care-and-use-animals-scientific-purposes (visited on 11/09/2024).

[17]       Williamson, P. et al. “Statistical Review by Research Ethics Committees”. Journal of the Royal Statistical Society. Series A (Statistics in Society) 163.1 (2000), pp. 5–13.

[18]       Thabane, L., Childs, A., and Lafontaine, A. “Determining the Level of Statistician Participation on Canadian-Based Research Ethics Boards”. IRB: Ethics and Human Research 27.2 (Mar. 2005), p. 11. DOI: 10.2307/3564481.

[19]       National Health and Medical Research Council, Australian Research Council and Universities Australia. National Statement on Ethical Conduct in Human Research (2007) – Updated 2018. Reference number: E72. 2018. URL: https://www.nhmrc.gov.au/about-us/publications/national-statement-ethical-conduct-human-research-2007-updated-2018 (visited on 11/09/2024).

[20]       Wilson, E. et al. “Designing, conducting, and reporting reproducible animal experiments”. Journal of Endocrinology 258.1 (Apr. 2023). DOI: 10.1530/joe-22-0330.

[21]       Barnett, A., White, N., and Collyer, T. Study protocol: What is the national prevalence of statisticians as full members of human research ethics committees in Australia? Apr. 2024. URL: https://osf.io/gx5qa (visited on 11/09/2024).

[22]       National Health and Medical Research Council. List of Human Research Ethics Committees registered with NHMRC. May 2023. URL: https://www.nhmrc.gov.au/research-policy/ethics/human-research-ethics-committees  (visited  on  11/09/2024).

[23]       Collyer, T. A. “The eye of the beholder: how do public health researchers interpret regression coefficients? A qualitative study”. BMC Public Health 24.1 (Jan. 2024). DOI: 10.1186/s12889-023-17541-3.

[24]       Latour, B. and Woolgar, S. Laboratory Life: the construction of scientific facts. Princeton University Press, 1986.

[25]       Jasanoff, S. “Breaking the Waves in Science Studies: Comment on H.M. Collins and Robert Evans, ’The Third Wave of Science Studies’”. Social Studies of Science 33.3 (2003), pp. 389–400. ISSN: 03063127.

[26]       Shao, J. and Sitter, R. R. “Bootstrap for Imputed Survey Data”. Journal of the American Statistical Association 91.435 (1996), pp. 1278–1288. DOI: 10.1080/01621459.1996.10476997.

[27]       Wikipedia. Triangular distribution. 2025. URL: https://en.wikipedia.org/wiki/Triangular_distribution (visited on 01/08/2025).

[28]       Liew, S. et al. statoolio: statistical tool guide. 2014. URL: https://datalibrary-rc.health.wa.gov.au/surveys/?s=79F8WXMP9A (visited on 11/09/2024).

[29]       Lumley, T. et al. “The Importance of the Normality Assumption in Large Public Health Data Sets”. Annual Review of Public Health 23.1 (May 2002), pp. 151–169. DOI: 10.1146/annurev.publhealth.23.100901.140546.

[30]       Diggle, P. et al. Analysis of Longitudinal Data. Oxford Statistical Science Series. OUP Oxford, 2002. ISBN: 9780198524847.

[31]       National Health and Medical Research Council. Report on the Activity of Human Research Ethics Committees and Certified Institutions for the period: 1 January 2023 to 31 December 2023. Apr. 2025. URL: https://www.nhmrc.gov.au/file/23347/download?token=rK3KESRC (visited on 08/19/2025).

[32]       Cameron, C. et al. “More than just numbers: challenges for professional statisticians”. Statistics Education Research Journal 16.2 (2017), pp. 362–375.

[33]       Pirosca, S. et al. “Tolerating bad health research: the continuing scandal”. Trials 23.1 (June 2022). DOI: 10.1186/s13063-022-06415-5.

[34]       Greenland, S. “Avoiding Power Loss Associated with Categorization and Ordinal Scores in Dose-Response and Trend Analysis”. Epidemiology 6.4 (July 1995), pp. 450–454. DOI: 10.1097/00001648-199507000-00025.

[35]       Altman, D. G. and Bland, J. M. “Statistics Notes: Comparing several groups using analysis of variance”. BMJ 312.7044 (June 1996), pp. 1472–1473. DOI: 10.1136/bmj.312.7044.1472.

[36]       Altman, D. G. and Bland, J. M. “Standard deviations and standard errors”. BMJ 331.7521 (Oct. 2005), p. 903. DOI: 10.1136/bmj.331.7521.903.

[37]       Wolkewitz, M. et al. “Two Pitfalls in Survival Analyses of Time-Dependent Exposure: A Case Study in a Cohort of Oscar Nominees”. The American Statistician 64.3 (2010), pp. 205–211.

[38]       Wolkewitz, M. et al. “Statistical Analysis of Clinical COVID-19 Data: A Concise Overview of Lessons Learned, Common Errors and How to Avoid Them”. Clinical Epidemiology 12 (Sept. 2020), pp. 925–928. DOI: 10.2147/clep.s256735.

[39]       Glasziou, P. and Chalmers, I. “Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers”. BMJ (Nov. 2018), k4645. DOI: 10.1136/bmj.k4645.

[40]       McKinney, W. P. “The Inexact Use of Fisher’s Exact Test in Six Major Medical Journals”. JAMA 261.23 (June 1989), p. 3430. DOI: 10.1001/jama.1989.03420230084031.

[41]       Sylvestre, M.-P., Huszti, E., and Hanley, J. A. “Do Oscar Winners Live Longer than Less Successful Peers? A Reanalysis of the Evidence”. Annals of Internal Medicine 145.5 (Sept. 2006), p. 361. DOI: 10.7326/0003-4819-145-5-200609050-00009.

[42]       Strasak, A. M. et al. “The Use of Statistics in Medical Research: A Comparison of The New England Journal of Medicine and Nature Medicine”. The American Statistician 61.1 (Feb. 2007), pp. 47–55. DOI: 10.1198/000313007×170242.

[43]       de Haas, B. et al. “Retraction Notice to: Perceptual load affects spatial tuning of neuronal populations in human early visual cortex”. Current Biology 30 (2020). DOI: 10.1016/j.cub.2020.11.015.

[44]       Leek, J. et al. “Five ways to fix statistics”. Nature 551.7682 (Nov. 2017), pp. 557–559. DOI: 10.1038/d41586-017-07522-z.

[45]       Yamada, T. et al. “On Pitfalls in Statistical Analysis for Risk Assessment of COVID-19”. Statistical Modelling and Risk Analysis. Springer International Publishing, 2023, pp. 215–229. ISBN: 9783031398643. DOI: 10.1007/978-3-031-39864-3_17.

[46]       Jones, L., Barnett, A., and Vagenas, D. “Common misconceptions held by health researchers when interpreting linear regression assumptions, a cross-sectional study” (Feb. 2024). DOI: 10.1101/2024.02.15.24302870.

[47]       Walker Morrell Luke Gelinas, D. Z. and Bierer, B. E. “Ensuring the Scientific Value and Feasibility of Clinical Trials: A Qualitative Interview Study”. AJOB Empirical Bioethics 14.2 (2023), pp. 99–110. DOI: 10.1080/23294515.2022.2160510.

[48]       Guillemin, M. et al. “Human Research Ethics Committees: Examining Their Roles and Practices”. Journal of Empirical Research on Human Research Ethics 7.3 (2012), pp. 38–49. DOI: 10.1525/jer.2012.7.3.38.

[49]       Shum, C. et al. “Flying Blind: How Thorough are IRBs when Assessing Scientific Value?” Journal of General Internal Medicine (Dec. 2024). DOI: 10.1007/s11606-024-09286-5.

[50]       Bunce, C. et al. “Medical statisticians: Always on tap but never on top?” Significance 21.2 (Apr. 2024), pp. 21–23. DOI: 10.1093/jrssig/qmae026.

[51]       Bross, I. D. J. “The Role of the Statistician: Scientist or Shoe Clerk”. The American Statistician 28.4 (1974), pp. 126–127. URL: http://www.jstor.org/stable/2683335 (visited on 03/12/2024).

[52]       Papalouka, S. et al. “Research Ethics Committees as an intervention point to promote a priori sample size calculations” (Nov. 2023). DOI: 10.31222/osf.io/5sehd.

[53]       Pysar, R. et al. “Australian human research ethics committee members’ confidence in reviewing genomic research applications”. European Journal of Human Genetics 29.12 (Aug. 2021), pp. 1811–1818. DOI: 10.1038/s41431-021-00951-5.

[54]       Kloot Meijburg, H. H. van der and Meulen, R. H. J. ter. “Developing standards for institutional ethics committees: lessons from the Netherlands”. Journal of Medical Ethics 27.suppl 1 (2001), pp. i36–i40. DOI: 10.1136/jme.27.suppl_1.i36.

[55]       Wynants, L. et al. “Prediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal”. BMJ 369 (2020). DOI: 10.1136/bmj.m1328.

[56]       Riley, R. D. et al. “Clinical prediction models and the multiverse of madness”. BMC Medicine 21.1 (Dec. 2023). DOI: 10.1186/s12916-023-03212-y.

[57]       Andaur Navarro, C. L. et al. “Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models”. Journal of Clinical Epidemiology 154 (2023), pp. 8–22. DOI: https://doi.org/10.1016/j.jclinepi.2022.11.015.

[58]       World Medical Association. Declaration of Helsinki – Ethical principles for medical research involving human participants. Oct. 2024. URL: https://www.wma.net/policies-post/wma-declaration-of-helsinki/ (visited on 11/09/2024).

[59]       Australian Commission on Safety and Quality in Health Care. Literature Review: Quality standards and accreditation schemes for human research ethics committees. Sydney, Mar. 2023. URL: https://www.safetyandquality.gov.au/our-work/health-and-human-research/national-mutual-acceptance-scheme-ethical-and-scientific-review-multi-centre-research (visited on 11/09/2024).

[60]       Trace, S. and Kolstoe, S. E. “Measuring inconsistency in research ethics committee review”. BMC Medical Ethics 18.1 (Nov. 2017). DOI: 10.1186/s12910-017-0224-7.

[61]       Hickey, A. et al. “Beyond Criticism of Ethics Review Boards: Strategies for Engaging Research Communities and Enhancing Ethical Review Processes”. Journal of Academic Ethics 20.4 (July 2021), pp. 549–567. DOI: 10.1007/s10805-021-09430-4.

[62]       Barnett, A. Code and data for the study of statisticians on human research ethics committees. 2025. DOI: 10.5281/zenodo.14759948. URL: https://github.com/agbarnett/ethics_statisticians  (visited  on  01/29/2025).