Published at MetaROR
July 7, 2025
Table of contents
Reproducibility and replicability of qualitative research: an integrative review of concepts, barriers and enablers
Nicki Lisa Cole1 , Sven Ulpts2
, Agata Bochynska3
, Eva Kormann4
, Matthew Good3
, Barbara Leitner5
, Tony Ross-Hellauer1
1. KnowCenter Research GmbH
2. Aarhus University
3. University of Oslo
4. Graz University of Technology
5. Amsterdam University Medical Center
Originally published on December 19, 2024 at:
Abstract
The integrative review presented here examines how reproducibility and replicability are conceptualized and discussed in relation to qualitative research, and which factors and practices enable or undermine them. Both peer-reviewed and grey English-language literature that address reproducibility and/or Open Science in relation to qualitative research were eligible for inclusion. Initial searches were conducted in Scopus, Web of Science, Dimensions, PubMed, APA PsychInfo, and JSTOR, and followed with snowball sampling from included literature. Studies were screened and both quantitative and qualitative data were extracted using the SyRF online platform, with 248 papers included. We found that conceptualizations that stem from quantitative standpoints are overwhelmingly framed as inappropriate practices and epistemic criteria for (most) qualitative research. When conceptualized in alternative ways that are adapted to the epistemic conditions, aims and practices of qualitative research, they can be both applicable and appropriate. Key barriers include the ontological and epistemological misalignment of reproducibility, replicability and Open Science and qualitative research, and ethical and practical concerns surrounding data sharing and reuse. Key enablers include practices that respond to ethical and practical concerns around data sharing and reuse (anonymization, ethical consent practices, context documentation, and ethical access management), adapting expectations and norms of openness, and established qualitative practices including documentation, reflexivity, and considering positionality. We conclude that reproducibility, replicability and Open Science practices must be adapted to the aims and epistemic conditions of qualitative research for them to be applicable and feasible, and that they will not always be both for all qualitative research.
1. Introduction
In recent years perceptions of a “reproducibility crisis” within some disciplines (Baker, 2016; Begley & Ellis, 2012; Open Science Collaboration, 2015) have prompted science policymakers, funders and publishers to push for more transparency in the scientific process and the reproducibility and replicability of its outputs through policy1, funding2 and publishing requirements3, and specific funding instruments4. While the concepts of reproducibility and replication have many definitions, across research contexts (Plesser, 2018), there are two distinct, but related components of conceptualizations of reproducibility and replication: the practices involved (see e.g., Gundersen, 2021) and the epistemic functions (see e.g., Matarese, 2022). The practices can be further divided into practices of redoing the whole or parts of a study and practices that enable redoing or more general forms of intersubjective accountability (Ulpts & Schneider, 2024) Suspected drivers of the perceived reproducibility crisis are a lack of transparency, a lack of incentives for reproduction or replication, a bias toward publishing positive results only, and various other questionable research practices (Atmanspacher & Maasen, 2015). The responses to concerns about reproducibility and replication described above typically require the use of certain Open Science practices, like open/FAIR data (findable, accessible, interoperable and reusable), open code and software, and Open Access publishing.
This reflects a normative shift towards evaluation, assessment and reward of research in accordance with the demonstration of Open Science practices, which are assumed to enable replication and reproducibility (see, e.g., Drummond, 2019; Penders et al., 2019). Hence, we understand these trends as demonstrating a focus on practices of redoing research, and on the transparent, open practices that potentially enable redoing, in keeping with the conceptualization offered by Ulpts & Schneider (2024).
The solutions proposed above are often expected to be applied across the full spectrum of research and scholarship, as indicated by the policies and requirements referenced above (unless researchers can justify exceptions). However, the framing of the crisis and policy responses to it are rooted in specific quantitative, positivist ontological and epistemological perspectives (Bazzoli, 2022; Bennett, 2021; Perrier et al., 2017; Wilkinson et al., 2016).5 An oft-cited survey published in Nature (Baker, 2016), that established the perception of a reproducibility crisis among some, was answered primarily by biologists and researchers from other STEM fields. In contrast, a survey conducted with a more diverse group of Horizon 2020 funding beneficiaries found that the perception of a significant crisis was most prevalent in medical and health sciences and least prevalent in humanities and social sciences (among other disciplines) (Athena RC et al., 2022). Additionally, just 24 percent of social scientists surveyed believe that reproducibility is “very important” to their discipline, compared with 67 percent in medical and health sciences (Athena RC et al., 2022). These findings suggest that concerns about reproducibility are far from universal across the scientific research landscape and vary based on the conditions and aims of the research.
Therefore, while the use of some Open Science practices appears to be on the rise in some circles – the proportion of publications that are Open Access (Directorate-General for Research and Innovation, 2024a), protocol preregistration (Serghiou et al., 2021) and data sharing within biomedical fields (Serghiou et al., 2021; Wallach et al., 2018) have all steadily risen since the 2000s, and Bochynska et al. (2023) found increases within linguistics from 2008–2019 – there have also been critiques from researchers highlighting some negative implications of these practices. Some have noted the high cost, in terms of time and resources of practicing Open Science, including preregistering study protocols, making data open and FAIR (see, e.g., Evans et al., 2023; Fecher et al., 2015), and publishing analysis code (Hostler, 2023). Additionally, qualitative social scientists, and scholars of humanities and arts have pushed back conceptually on concerns with reproducibility and replication, arguing that both concerns about and responses to them are based on quantitative, positivist ontologies and epistemologies, and are therefore neither applicable to nor appropriate for their research (Bazzoli, 2022; Bennett, 2021; Drummond, 2019; Penders et al., 2019; Ulpts & Schneider, 2023). For example, qualitative researchers have noted a mismatch between their ontologies, epistemologies, research designs, methods and data, and the requirements of standardized data management plans (Karcher et al., 2016), preregistration templates and practices (Haven et al., 2020, 2020; Haven & Van Grootel, 2019), and data sharing (Feldman & Shaw, 2019; Freese et al., 2022; Joyce et al., 2022; Lorenz & Holland, 2020; Mauthner et al., 1998; Prosser et al., 2022). Some have pointed out that the time-cost for making data open and FAIR is especially high for qualitative researchers, due to the context-specific and often personalized and researcher-dependent nature of the data (Dienlin et al., 2021; Field et al., 2021; Karcher et al., 2016; Mozersky et al., 2021; Tonnesson, 2012).
Others fear that these aspects of science reform may have homogenizing effects on research processes and methods (Malich & Rehmann-Sutter, 2022) and exclusionary effects on what is considered “good”, publishable research (Drummond, 2019). Critics fear that normalizing Open Science practices within science policy, funding and publishing, in pursuit of reproducibility and replicability, will foster (further) epistemic exclusion and marginalization of researchers and scholars working outside of quantitative, positivist research paradigms. As Penders et al. (2019) put it, “If fields of research exist for which replication is an unreasonable epistemic expectation, then policies for research that universalize the replication drive will perpetrate (some might say perpetuate) an epistemic injustice, ghettoizing the humanities and hermeneutic social sciences as either inferior research or not really research at all.” Requiring Open Science practices could even impede the ability of qualitative researchers to carry out their work, due to ethical, practical, and epistemic dependencies of qualitative approaches (Bazzoli, 2022; Bennett, 2021).
Yet, at the same time, many qualitative research traditions have a long history of practicing transparency and reflexivity, achieved through extensive documentation of the research process, its sites, samples, and the observations and interactions that provide the basis for much qualitative data (Coombs, 2017; Friedhoff et al., 2013; Jesser, 2011; Karcher et al., 2021; Karhulahti et al., 2022; Kern & Mustasilta, 2023), and reflecting on the researcher’s positionality while doing so (Bennett, 2021; Doyle et al., 2020; Thoresen & Öhlén, 2015). And, despite criticisms, the idea of open qualitative research is gaining traction, with a growing community of scholars, service providers, and IT and research support staff working to establish specialized tools, templates, guidance and platforms for open qualitative methods, analysis, and data.6 Additionally, some within the qualitative research community are working to illuminate the complexities in the relationship between reproducibility or replication and qualitative research, and to define them in ways relevant to and feasible for qualitative research (Büthe & Jacobs, 2015; Talkad Sukumar et al., 2020; Tuval-Mashiach, 2021). Others have conducted investigations into researchers’ perceived applicability of reproducibility to qualitative approaches (Reischer & Cowan, 2020).
We therefore conceived this integrative review to gather the disparate and ongoing discourse on reproducibility and replication (redoing) and Open Science (enabling) across diverse qualitative research disciplines, fields, and areas, and to synthesize the debates surrounding and positions toward them within the broader qualitative research community. We aim to consolidate perceptions of relevance and feasibility, as well as barriers and enablers of reproducibility, replication, and Open Science practices for qualitative research. We do so to foster informed, epistemically just development of science reform, expectations, policies, and practices.
We pose the following two research questions:
-
How are reproducibility and replication conceptualized and discussed in relation to qualitative research?
-
Which factors and practices enable, and which are barriers to, the potential reproducibility or replication of qualitative research?
2. Materials and Methods
This review follows the rationale for conducting integrative reviews originally developed by Whittemore and Knafl (2005) and expanded upon by Torraco (2016) and Toronto and Remington (2020). An integrative review provides a broad synthesis of literature from both empirical studies of assorted designs and theoretical evidence. There is no extension to the PRISMA guidelines specifically for integrative reviews, therefore, we follow the PRSIMA-Scr checklist for scoping reviews (Tricco et al., 2018).
2.1. Protocol and registration
This study was pre-registered through the Open Science Framework under https://doi.org/10.17605/OSF.IO/Q4XWK on 13th July 2023 (Cole et al., 2023). The preregistration includes additional details on study and procedure. Deviations from the preregistered protocol are shared in Supplement 2.
2.2. Positionality of the research team
Our interdisciplinary, mixed-gender, Europe-based team includes those trained in psychology, sociology, philosophy, information studies, and innovation and entrepreneurship, with a mix of qualitative and quantitative methodological expertise. We are diverse in terms of academic age and seniority, and composed of researchers and research support providers, all with experience with and expertise in open and reproducible research. We all hold positive views of research openness, transparency, and reproducibility as guiding values and aims, yet collectively recognize that expectations of these are not appropriate for all research projects or settings, and that sometimes science reform policies have unintended negative consequences. For a collection of in-depth, individualized positionality statements from each member of our team, see Supplement 3.
2.3. Eligibility criteria
Our review includes English-language peer-reviewed papers, preprints, white papers and grey literature that discuss reproducibility and replication in the context of or in relation to qualitative research, qualitative methods or “mixed methods” (so as not to miss discussions relevant to qualitative research that might be situated within this literature rather than that focused exclusively on qualitative research). Because of their close conceptual link to reproducibility and replication and their prominence within qualitative research methods, we also considered eligible sources that discuss transparency and accountability in this context. Additionally, due to the assumed link between replication and reproducibility with Open Science, literature discussing or investigating the use of Open Science practices in qualitative and mixed methods research was included. For search string development, we included Open Science generally, and specific practices linked to efforts to promote and increase reproducibility and replication, including open data, open methods, and pre-registration (as opposed to other Open Science practices, like Open Access publishing, Citizen Science, and open evaluation, for example). While these criteria shaped the development of search strings, they did not serve as firm exclusion criteria during our review. For example, other relevant practices present in the retrieved literature, like Open Analysis and registered reports, were documented during our data extraction process. No limitations were set in terms of publication date but inclusion required the availability of the full text.
2.4. Information sources
Academic databases including Scopus, Web of Science (WoS), Dimensions, PubMed, APA PsychInfo, and JSTOR were initially searched. Simultaneously, we conducted a grey literature search in various online sources based on our knowledge of key stakeholders in the science reform and policy landscape, including preprint archives (more detail is provided in our pre- registered protocol). During data extraction of included academic and grey literature, reviewers could mark a record as relevant for the snowball search. Reference lists for these records were visually scanned for additional relevant sources. The study team members also had the option to add literature manually for snowball screening.
2.5. Search
We developed search strings based on a set of keywords and adapted for each database and grey literature source. The list of keywords included reproducibility, replicability, open data, open science, accountability, transparency, preregistration, qualitative research, and mixed methods research. Search strings for all databases are listed in our preregistered protocol, with an example provided in Table 1. We validated our search strings by piloting in Scopus to check whether sources identified through background research were present within the search results, and by testing and developing search strings in each database to ensure maximal inclusion of relevant results and minimal inclusion of irrelevant ones. NLC and AB developed search strings collaboratively, with additional expertise offered by research librarians at the University of Oslo library. We carried out the initial academic database search on 13th July 2023.

We conducted the grey literature search between 7th and 22nd August 2023 using the same keywords identified above and used Google search functionality to carry out site-specific searches on most sites (NLC and SU conducted this search). Details on the grey literature search and its outcomes are included in the relevant spreadsheet in our shared datasets (Kormann et al., 2024).
2.6. Selection of sources of evidence
All academic search results were first deduplicated using the ASySD tool (Hair et al., 2021) and then screened by two independent reviewers in Systematic Review Facility (SyRF) (Bahor et al., 2021), an online screening platform for literature reviews. Grey literature search results were initially abstract screened (by NLC and SU) during the search itself with only sources deemed relevant advanced to full-text screening. Sources were excluded if they contained neither a focus on reproducibility/replicability, transparency, or accountability of qualitative research, nor a focus on Open Science practices as they pertain to qualitative research. Detailed exclusion criteria are provided in our preregistered protocol. Following this, all included texts were full text screened by a single reviewer. Included full-texts were then used as a basis for a snowball search which advanced additional texts to full-text screening (see Figure 1 for the full process of selecting sources). A more detailed description of the full selection process is provided in Supplement 4.

Figure 1. PRISMA-P flow diagram.
*Indicates that there is a data set shared for this step of the process at https://doi.org/10.17605/OSF.IO/JAVZ2 (Kormann et al., 2024).
2.7. Data extraction process and quality appraisal
Data was extracted within SyRF by a single reviewer (the person making the decision on inclusion during full-text screening) using a purpose-built data extraction form. Items included in the form were developed collaboratively during the preparation of the study protocol to collect descriptive and general data from all sources and data on reproducibility, replicability, accountability, transparency, and Open Science practices (see Supplement 5). Additional comments could be entered for each question in the data extraction form if required. The form was designed to allow us to generate descriptive data about our sample and to facilitate responses to the research questions. The extraction form and process were piloted and adapted by NLC and AB during development of the study protocol. Quality appraisal criteria were developed by NLC based on existing criteria and formulated separately for distinct types of sources (theoretical and conceptual, empirical and reviews, methodological and guidance). Extraction instructions and questions, and quality assessment criteria (and their bases) are described in detail in our preregistered protocol.
2.8. Sample
The final set of 248 included sources is diverse in terms of authorship and publication outlet, with more than 700 authors and 130 outlets present (see Figure 2A).7 The year of publication ranges from 1997 to 2023, with a pronounced increase within the decade prior to our search (see Figure 2B). The median publication year was 2018, meaning that only half of all included sources were published over the two decades between 1997 and 2018. In terms of field, discipline or research area associated with the paper, more than a third of sources could not be allocated to one specific discipline but were coded as referring to General Qualitative Research (35.5% of included sources) (see Figure 2C). Where a more direct focus could be identified, the sources mainly related to medical and health sciences (16.9% of included sources) or psychology (13.3% of included sources). All sources included in this review are published and archived in an open Zotero library, though not all are cited in this paper.8

Figure 2. Sample description of included sources. (A) Overview. (B) Number by publication year with median line indicated. (C) Sources as classified by their disciplinary orientation.
2.9. Analysis and synthesis of results
The primary aim of our analysis and synthesis processes was to identify trends in the literature to respond to our research questions with depth and nuance. As part of this, we aimed to identify attitudes toward, perceptions of, and evidence demonstrating how reproducibility, replication, and Open Science relate to qualitative research, as documented by the literature included in our sample. Our coding process for extracted qualitative data derived from open-ended questions began by creating a structure for the data (see Deterding & Waters, 2021), then moved into an inductive process (see Supplement 6 for our preliminary codebook), before transitioning to a (mostly) deductive process (see Supplement 7 for our final codebook). Coded data were then analyzed by specific members of the team to synthesize the trends in the data and respond to the research questions. A detailed description of these steps is provided in Supplement 8 and the full NVivo project is shared as part of our shared data package (Windows and REFI-QDA versions). Additionally, descriptive quantitative analysis of data derived from closed-ended questions was conducted in Python to generate frequencies, proportions, crosstabs, and figures (Kormann, 2024).
3. Results
Of the 248 sources included in this study, 67.3% were identified as addressing reproducibility broadly (including discussions of replicability, accountability, and transparency) and 81.2% as addressing Open Science (including openness generally). Almost half of the sources were coded as discussing both (see Figure 3A). Open Data (including Open Data itself, data sharing, data availability, and data reuse) was the most discussed type of OS practices within our sample, with more than half of all sources discussing it (57.3%). This was followed by practices referring to open processes, such as Open Methods (35.5%), Open Analysis (19.0%), and Preregistration (including registered reports, 14.1%) (see Figure 3B). In the majority of our sample, no specific data types or methods were mentioned as the focus within the sources (see Figures 3C and 3D). Where a direct focus could be identified, text data was mentioned most often (33.1%), with interviews as the most specified method (36.3%).

In what follows, we present data in response to RQ 1 in section 3.1, and in response to RQ 2 in sections 3.2 and 3.3.
3.1. Conceptualizations of and discourse surrounding reproducibility and replication
In our sample 99 sources directly discuss reproducibility or replication (a subset of those indicated in Figure A). While there seems to be a balance between positive and critical attitudes (44 critical sources, 43 positive ones, and 12 neutral), what each means when they refer to replication or reproducibility varies. In this section we first present how the literature relates practices of reproducibility and replication to qualitative research, and then how it relates their assigned epistemic functions to qualitative research.
Often the meanings of replication and reproducibility are mentioned to discuss which practices of redoing and enabling are applicable to or appropriate for qualitative research, and which are not (Bienefeld et al., 2020; Tsai et al., 2016; Tuval-Mashiach, 2021). The conceptualizations focused on practices tend to be about various forms of redoing something, like parts or the whole of a study again to obtain findings or interpretations that are the same or similar to the original (see e.g., Aguinis & Solarino, 2019; Branney et al., 2019; Dubois & Gadde, 2014; Frohwirth et al., 2023; Huma & Joyce, 2022; Kern & Mustasilta, 2023; Lui et al., 2022; Moravcsik, 2014b; Noret et al., 2022; Reischer & Cowan, 2020; Tonnesson, 2012; Tuval-Mashiach, 2021). Different types of redoing are presented, like conceptual, exact, partial, empirical, and direct replication (Aguinis & Solarino, 2019; Bienefeld et al., 2020; Hoogeveen & Elk, 2021; Tuval-Mashiach, 2021). Some argue that certain types of replication that allow for variation or change in the redoing of previous research, like conceptual replication (Bienefeld et al., 2020; Hoogeveen & Elk, 2021; Tuval-Mashiach, 2021), can be appropriate for qualitative research. While Buckley et al. (2022) argue for methodological repeatability, Tsai et al. (2016) reject replication as verification for numerous reasons, but support Clemens’ (2017) definition of “reproduction”, which requires use of identical methods as an original study on a different sample from the same population; and Goodman et al.’s (2016) “result reproducibility”, which means conducting an independent study using procedures as similar to an original study as possible and obtaining the same result (Tsai et al., 2016). However, some also outright reject the notion of replication as redoing because it is seen as foreign to their forms of qualitative research. Büthe and Jacobs (2015), for instance, argue that for immersive and hermeneutic research, it is neither a relevant goal nor an appropriate evaluative standard. Another class of definitions in the literature refers to practices that can enable others to redo the analysis of a study (Makel et al., 2022; Steltenpohl et al., 2023), or enable more general forms of intersubjective accountability by allowing others to trace, understand, and evaluate what was done in the research in its local conditions, both of which can mostly be achieved by providing enough depth and richness of detail in writing (Büthe & Jacobs, 2015; Frohwirth et al., 2023; Hall, 2016).
Beyond these references to practices of reproducibility and replication in the quantitative domains and how they relate to qualitative research, there are also attempts within the literature to adapt practices of replication and reproducibility to the practices, conditions, and aims of qualitative research (see section 3.3.3. on flexibility of open research practices). This reflects a perceived need to do so from various authors (Freese & Peterson, 2017; Huma & Joyce, 2022; Talkad Sukumar & Metoyer, 2019; Vuckovic Juros, 2022). For example, Büthe and Jacobs (2015), in an attempt to adapt replication, and make it applicable to and relevant for qualitative research, define replication-in-thought as:
“…the provision of sufficient information to allow readers to trace the reasoning and analytic steps leading from observation to conclusions, and think through the processes of observation or engagement. Replication-in-thought involves the reader asking questions such as: Could I in principle imagine employing the same procedures and getting the same results? If I looked at the evidence as presented by the author, could I reason my way to the same conclusions? Replication-in-thought also allows a reader to assess how the researcher’s choices or starting assumptions might have shaped her conclusions.” (Büthe & Jacobs, 2015)
This is similar to Pratt et al.’s (2020) decoupling of transparency from replication and not about redoing or enabling acts of redoing, but about enabling more general forms of intersubjective accountability by presenting the research in a way that allows the reader to trace what happened and comprehend the logic of a study.
In a similar attempt to adapt the practices of replication and reproducibility to qualitative research, Talkad Sukumar and Metoyer (2019) claim that for sociolinguistic research, “reproducibility in the constructivist sense as the alternative analyses of the gathered data and also including interpretive comparisons of the findings” (Talkad Sukumar & Metoyer, 2019) would be suited. Furthermore, they define qualitative replication as:
“… a (novel) qualitative study conducted by independent researchers replicating one or more aspects (such as study design, research questions, context, methods, and participants) of an earlier qualitative study and embedding within its findings an interpretive comparison with a view to corroborate, elaborate, contrast, or clarify the elements corresponding to the replicated aspects with those of the earlier study.” (Talkad Sukumar & Metoyer, 2019)
This quote illustrates that this form of replication in qualitative research does not aim for sameness or similarity of results, but it is broader and more flexible. Talkad Sukumar and Metoyer (2019) state that their conceptualization of replication is similar to the established qualitative research practices of triangulation and crystallization. While others argue that conceptual replication is like triangulation (Hoogeveen & Elk, 2021), Pownall (2022) suggests that replication, or the label, might be redundant or unnecessary for qualitative research, because similar practices and principles, like triangulation and crystallization, already exist and are more adapted to the ontologies of qualitative research.
Conceptions of the epistemic functions of reproducibility and replication in quantitative research, which are put in relation to qualitative research, tend to vary in the literature. Therefore, Tuval- Mashiach (2021) lists the aim or function of replication as one of four criteria that determine the relevance of it for qualitative research (Tuval-Mashiach, 2021). Among the functions mentioned or taken from rather quantitative and positivist research traditions in our sample are establishing generalizability (Bienefeld et al., 2020; Borgstede & Scholz, 2021; Dienlin et al., 2021; Hoogeveen & Elk, 2021; Mozersky, Walsh, et al., 2020; Tucker, 2016), credibility (Lui et al., 2022), verification (Tsai et al., 2016), validity (Buckley et al., 2022; Tuval-Mashiach, 2021), trustworthiness (Noret et al., 2022), truthfulness (Reischer & Cowan, 2020), and reliability (Roberts et al., 2019; Vuckovic Juros, 2022).
Some authors (Talkad Sukumar & Metoyer, 2019; Tuval-Mashiach, 2021) argue that the epistemic functions of reproducibility and replication are inherently linked to or even dependent upon the different epistemological positions underlying the qualitative research at hand. This, in combination with the plurality of epistemologies that can underlie qualitative research, might explain the diversity and disagreement we find in the literature regarding opinions about the epistemic functions of reproducibility and replication. However, some scholars also argue that qualitative research has its own epistemic crisis because it lacks evaluative standards, while practices and criteria related to reproducibility and replication could function as such (Befani, 2020; Moravcsik, 2010). For instance, generalizability is one of the epistemic functions of replication in qualitative research mentioned in the literature (Aguinis & Solarino, 2019; Bienefeld et al., 2020; Moravcsik, 2014b). Aguinis and Solarino (2019) frame empirical replication as redoing the same procedure with a different population to test generalizability in positivist qualitative research. Related to generalizability, Tucker (2016) argues that replication could help extend local and limited theories of grounded theory approaches. However, this function of replication is also mentioned by others as not applicable and one of the reasons for rejecting a role for replication in qualitative research (Dienlin et al., 2021; Doyle et al., 2020; Hoogeveen & Elk, 2021; Høyland et al., 2017; Lui et al., 2022; Reischer & Cowan, 2020; Talkad Sukumar & Metoyer, 2019; Tuval- Mashiach, 2021). Dienlin et al. (2021), for instance, argues that qualitative research cannot have a replication crisis because it does not aim for generalizability (also see section 3.2.1.2. on epistemological barriers).
Some also attempt to adapt the epistemic functions of reproducibility and replication to qualitative research. In these attempts, scholars try to align the epistemic functions of reproducibility and replication with the ways knowledge is produced and assessed in qualitative research. Some argue that since non-positivist qualitative research builds on different ways of knowing compared to quantitative research, and aims to achieve distinct epistemic functions, to make replication and reproducibility applicable, they can and have to be redefined or adapted to the aims and conditions of qualitative research if they are to be used within qualitative research (Buckley et al., 2022; Talkad Sukumar & Metoyer, 2019; Tuval-Mashiach, 2021).
In attempts to move replication into qualitative research and adapt it to the evaluative standards of qualitative research, transferability is a frequently stated function of replication (Buckley et al., 2022; Hoogeveen & Elk, 2021; Porte & Richards, 2012; Reeping & Edwards, 2020; Tuval-Mashiach, 2021). However, Pownall (2022) notes in a commentary on Makel and colleagues (2022) referring to Tuval-Mashiach (2021), that while replicability and transferability are similar, they are also quite different. Tuval-Mashiach (2021) states that while transferability is about the ability to apply results of an original study to a new context, replication is about redoing an aspect or specific aspects of an original study in a new study. Consequently, Pownall (2022), in reference to Pratt et al. (2020), argues that equating transferability with replicability contributes to an “ongoing inappropriate transfer of quantitative practices and tools to qualitative research. This perpetuates the notion that quantitative practices are ‘gold standard’, and that qualitative work must mould itself around these standards” (Pownall, 2022). Additionally, concretizing how the appropriate functions of replication depend on the specific qualitative research approach and epistemology, Talkad Sukumar and Metoyer (2019) argue that for phenomenological qualitative research, replication can help in capturing the “essence” of an experience. They also claim that for critical qualitative research, replication can uncover and compare biases, prejudgments and thought processes within a study (Talkad Sukumar & Metoyer, 2019).
In sum, we find that while practices that enable others to trace, understand and evaluate research are mostly seen as suitable for qualitative research, replication as practices of redoing to get the same result are mostly seen as foreign practices or epistemic criteria to qualitative research. However, once replication and reproducibility are conceptualized more flexibly or adapted to the conditions, aims and practices of (non-positivist) qualitative research, some argue that there can be a place for them, with various functions depending on the underlying epistemology. Hence, the literature, and the diversity of opinions it contains, suggests that the place, form and function of replication are not uniform, but depend on the specific conditions and aims of the qualitative research in question. The subsequent sections elaborate on barriers and enablers of reproducibility, replication and Open Science practices in qualitative research, however, as this section indicates it is important to keep in mind that while the sources referred to in the following sections often use the same terms, they might be addressing different things.
3.2. Barriers to reproducibility, replication, and Open Science
Key barriers to reproducibility, replication, and Open Science (see Table 2) include the ontological and epistemological disparities between quantitative and qualitative research (found in 36.29% of sources), including specifically the context-based nature of much qualitative research (27.82%) and the role of the researcher (21.37%) in qualitative research methodologies in which researcher subjectivity plays a central role. Other key barriers include those that are in direct relationship to the belief that open data is a requirement for reproducible/replicable research; namely, the ethical issues associated with qualitative research that involves human participants (31.85%), and related to this, participant anonymity (27.82%) and informed consent (18.15%). Indeed, most barriers articulated to Open Science practices (see Table 3) are specific to data sharing (42.74%) and reusability (25.00%). In what follows, we provide detailed findings on the key barriers described here.


Table 3. Coding references and sources coded to Barriers and OS Practices
3.2.1. Ontological and epistemological barriers to reproducibility, replication, transparency and Open Science
3.2.1.1. Ontological barriers
As discussed in detail in section 3.1, a key theme in our included literature is that reproducibility, replicability, and linked issues like transparency, reliability, and validity, developed within an objective ontological stance, are often perceived as in ontological and/or epistemological conflict with qualitative research. The key reason for these perceptions presented in our sample is that the ontological assumption of objectively measurable phenomena in pursuit of a singular, knowable truth does not apply to qualitative research (Haven & Van Grootel, 2019; Nixon & Power, 2007). In contrast, much qualitative research is constructivist or interpretivist and therefore subjective in nature, which means that it “has an ontological assumption that phenomena are understood differently by individuals with such understandings being socially and historically influenced (Creswell & Creswell, 2018; Kivunja & Kuyini, 2017)” (Buckley et al., 2022). In some cases, even defining the subject of a qualitative study may be difficult, given an assumption of socially constructed meaning (e.g., when a person or community’s religious experience is the phenomenon under study) (Anczyk et al., 2019). Under such conditions, Anczyk et al. (2019) claim that replicability is not possible. Therefore, applying a positivist conception of replicability to qualitative research has been framed as both “ontologically problematic and potentially harmful (Pratt et al., 2020)” (Buckley et al., 2022). Some authors offer similar accounts of the relationship between Open Science and qualitative research, asserting that Open Science is conceived based on and organized around quantitative, positivist logics and quality criteria of knowledge production (Bazzoli, 2022; Bennett, 2021; Branney et al., 2023; Chin et al., 2020; Huma & Joyce, 2022; Perreault & Dienlin, 2022; Pownall et al., 2021; Riley et al., 2019; Rubin, 2023; Steltenpohl et al., 2023), and that it is therefore designed to solve problems that exist within quantitative research (Bennett, 2021; Huma & Joyce, 2022) by focusing on issues that are not considered pressing or present in qualitative research, like statistical significance, replication and hypothesis testing (Bennett, 2021). Antonio et al. (2019) extends this critique to standard data management plans, while others suggest that established practices of data sharing (Camfield, 2019; McLeod & O’Connor, 2021; Pownall et al., 2022) and preregistration (Branney et al., 2023; Haven & Van Grootel, 2019; Perreault & Dienlin, 2022) are an ontological mismatch with qualitative research.
3.2.1.2. Epistemological barriers
Building on these ontological barriers, many assert that epistemological differences between quantitative and qualitative research, and within qualitative research, are barriers to reproducibility, replication, and Open Science (Bazzoli, 2022; Branney et al., 2023; Pownall et al., 2021, 2023; Rubin, 2023) (see Table 2 and Table 3). Central to this argument are the constructivist (Buckley et al., 2022; Campbell et al., 2022; Rubin, 2023; Talkad Sukumar & Metoyer, 2019; Tuval- Mashiach, 2021) and interpretive nature of qualitative research (Haven & Van Grootel, 2019; Jacob et al., 2021; Karhulahti et al., 2022; Kern & Mustasilta, 2023; Tuval-Mashiach, 2021; Yardley, 2000), particularly participatory and ethnographic research (Dubois & Gadde, 2014; Tuval-Mashiach, 2021), and the often open-ended, dynamic and exploratory nature of it (Bennett, 2021; Bienefeld et al., 2020; Chin et al., 2020; Pownall et al., 2021). Some frame this barrier as a distinction between inductive and abductive theory generation (qualitative), which are constructivist in nature, versus deductive theory testing (quantitative) (Buckley et al., 2022). Pratt et al. (2020) states that replication is not appropriate for qualitative research because it is not an aim of inductive research, while Silverman et al. (2002) states, “…for social constructionist approaches, the very notion of wanting to replicate findings … is erroneous.”
Other aspects of qualitative research epistemologies and research designs that are framed as barriers to replicability include the differing circumstances in which research is set (Buckley et al., 2022; Camfield, 2019; Dubois & Gadde, 2014; Hammersley, 1997; Makel et al., 2022), that situations studied are often fluid and dynamic (Camfield, 2019; Yardley, 2000), and that they are bound by history, context, time, and composed of individual relationships (Campbell et al., 2022). Despite the best efforts of researchers, Hammersley (1997) points out that there are limits to how much context one can convey in study reports. Summing this up and framing replication as suspicious, Pratt et al. (2020) wrote, “In the transcendental realist world of open, fluid, and complex systems, exact replications would not be acclaimed as contributions to cumulative knowledge; they would be anomalies needing to be explained. From this view, replication would, or should, evoke skepticism rather than confidence.”
Some authors make similar arguments against certain definitions of transparency (Abramson & Dohan, 2015; Freese et al., 2022; McGrath & Nilsonne, 2018; Pratt et al., 2020; Tamminen et al., 2021; Thoresen & Öhlén, 2015), rigor (Abramson & Dohan, 2015; Kern & Mustasilta, 2023; Roberts et al., 2019), integrity (Hall, 2016), reliability and validity (Nixon & Power, 2007) and generalizability (Lui et al., 2022; Mozersky, Walsh, et al., 2020; Reischer & Cowan, 2020; Tuval- Mashiach, 2021), for how they are associated with what reproducibility and replicability mean in the context of quantitative science, as described in section 3.1. For example, inter-coder reliability, often considered a standard of rigor and integrity in qualitative research, will not always be achievable due to the subjective nature of interpretive research (Roberts et al., 2019), while Freese and Peterson (2017) fear that attempting to make qualitative coding more “scientific” through practices aimed at increasing rigor may obscure the process of interpretive work (citing Biernacki, 2012).The context-based nature of qualitative research (Lui et al., 2022), its in-depth analysis of specific objects of study (Hoogeveen & Elk, 2021) and its use for “local inference” (Reischer & Cowan, 2020) are viewed as barriers to generalizability. When research is focused on “rare phenomena,” generalizability would not be an aim (Tuval-Mashiach, 2021). And, because it is “rarely meant to be representative or generalizable”, Mozersky et al. (2020) state that replicability is not an appropriate expectation for qualitative research. Regarding transparency, Pratt et al. (2020) state the “author’s ontological and epistemological assumptions about research” determine which aspects should be transparent. Therefore, associating transparency with replication, reproducibility or trustworthiness is problematic for qualitative research (Pratt et al., 2020). As Tamminen et al. (2021) put it, transparency is “in conflict with the underlying philosophies used in qualitative research.” Jacobs et al. (2021) take a firmer stance against it, by stating that transparency is “an intellectually incoherent notion grounded in a narrow and questionable set of presumptions about how knowledge is produced” which ignores the diversity of methods and settings within qualitative research. They also argue that, despite the provision of context information and the practice of reflexivity being normal facets of qualitative research, one cannot evaluate (interpretive) qualitative research on the basis of its transparency or lack thereof, because “we never have full, conscious access to the deep theoretical constructs that structure our perceptions and understandings” (Jacobs et al., 2021).
Some argue that the epistemological barriers to Open Science stem from a lack of engagement with and understanding of epistemic diversity within the Open Science community, particularly regarding exploratory, interpretive, and feminist research traditions (Bennett, 2021; Chin et al., 2020; Pownall et al., 2021). This disconnect is articulated by some as a reason preregistration is incompatible with qualitative research. Those who feel this way point to the exploratory, emergent, and flexible nature of qualitative research as key barriers (Haven et al., 2020; Haven & Van Grootel, 2019; Tamminen & Poucher, 2018). As Haven and Van Grootel (2019) argue, preregistration is not compatible with qualitative research because it does not aim to predict or test theories, but rather to generate them. In addition, the fact that the researcher is typically “an active participant in the research” appears to be disregarded by those developing Open Science policies, practices, and services, which is problematic when Open Science practices are meant to facilitate reproducibility or replication (Perreault & Dienlin, 2022).
We note that the majority of epistemological barriers to Open Science are focused on data sharing and reuse (see Table 4). The logics of positivist epistemologies, associated with quantitative research, are perceived by authors as structuring data management plans (Karcher et al., 2016), data sharing expectations and practices (Alexander et al., 2020; Büthe & Jacobs, 2015; Feldman & Shaw, 2019; Field et al., 2021; Freese et al., 2022; Jacobs et al., 2021; Jones & Alexander, 2018; Joyce et al., 2022; Lorenz & Holland, 2020; Mauthner et al., 1998; Mauthner & Parry, 2009; McLeod & O’Connor, 2021; Parry & Mauthner, 2004; Prosser et al., 2022; Yoon, 2015) and how platforms/repositories operate (Antonio et al., 2019; Guishard, 2018), which make them incompatible with much qualitative research. As Antonio et al. (2019) point out, “current designs of data repositories may not be supporting the messy, unknown, and emergent aspects of qualitative analysis.” Guishard (2018) notes that certain methodologies, like participatory action research and critical ethnographic scholarship, result in the “messiness” that Antonio et al. (2019) describe, because they “include multi-perspectival analyses and unsettle what is ordinarily conceptualized as data and results of research” (Guishard, 2018). Additionally, McLeod and O’Connor (2021) argue that the empiricist and quantitative logics of data sharing presume a fixity and stasis of data, and a detachment from the researcher that does not apply to qualitative data.

Karcher et al. (2016) point out that the reliance on established repository categories used to classify archived data, like samples, variables, codebooks, descriptive statistics, and survey instruments, is incompatible with the diversity of data types present within qualitative research. Similarly, many authors argue that there is a fundamental epistemological mismatch between standard practices of data sharing (Feldman & Shaw, 2019; Freese et al., 2022; Joyce et al., 2022; Lorenz & Holland, 2020; Mauthner et al., 1998; Prosser et al., 2022) and reuse (Alexander et al., 2020; Chauvette et al., 2019; Feldman & Shaw, 2019; Kuula, 2011; Tsai et al., 2016; Yoon, 2015) and qualitative research, echoing the points about subjectivist, interpretive, constructivist, exploratory, and inductive or abductive research given in section 3.2.1. Some point to epistemic diversity within qualitative research as a barrier to data sharing and reuse (Jacobs et al., 2021; Jones & Alexander, 2018), as well as the diversity of methodologies (van den Berg, 2008), forms of evidence, and research settings that epistemic diversity implies (Jacobs et al., 2021). On this theme, Yoon (2015) states that researchers who attempted to reuse the qualitative data of others felt that they were trying to make “a square fit into a circle in some ways.” Feldman and Shaw (2019) highlight the efficiency argument for data sharing as a key aspect that makes it epistemologically incompatible with qualitative data, writing, “we query the adequacy and appropriateness of these efficiency criteria for meaningfully capturing the knowledge that is produced by interpretive practices with different underlying assumptions and purposes.”
3.2.1.3. The role of the researcher as a barrier
A key barrier related to the epistemological ones stated above is the central role of the researcher within qualitative research (Branney et al., 2019; Dienlin et al., 2021; Tamminen et al., 2021) – even “an active participant in the research”, as one qualitative researcher surveyed by Perreault and Dienlin (2022) put it. As Tamminen et al. (2021) frame it, this is a barrier to replication because the social reality as it is documented by qualitative researchers does not exist independent of the researcher, who creates data through observation and interaction. Guishard (2018) argues that this is particularly the case with ethnographic or participatory research, which are “relational analyses”, and Büthe and Jacobs (2015) present the same argument about “immersive research”. Additionally, some argue that qualitative researchers interpret data in inherently subjective ways (Dienlin et al., 2021; DuBois et al., 2018; Frohwirth et al., 2023), which makes reproducibility (and verification (Tsai et al., 2016)) unlikely to be achievable and an inappropriate evaluation criterion for qualitative research (Frohwirth et al., 2023). This is framed by some as a barrier to data sharing and reuse, because in such research epistemologies and methods, qualitative data is often co- constructed by researcher and participants (Büthe & Jacobs, 2015; Field et al., 2021; Parry & Mauthner, 2004). It is, as Field et al. (2021) call it, “researcher-laden” data, and according to Büthe and Jacobs (2015) never truly “raw” because it is socially constructed. Some authors even assert that nobody who is unfamiliar with the research context and the social relationships within the field in which the data were collected can truly understand the data and use them responsibly (Chauvette et al., 2019; Feldman & Shaw, 2019).
3.2.2. Context as a barrier to data sharing and reuse
The role of the research context in qualitative research is identified within our sample as a key barrier to data sharing and reuse (see Table 4). This is because it is generally understood that reuse of qualitative data requires contextual information about the study and wherein data were generated (Bishop, 2007; Broom et al., 2009; Cliggett, 2013; Coltart et al., 2013; Corti, 2000, 2006b; Corti & Thompson, 2004; Gervais et al., 2021; Grinyer, 2009; Hall, 2016; Hocker et al., 2021; Jones & Alexander, 2018; Kern & Mustasilta, 2023; Mauthner & Parry, 2009; McLeod & O’Connor, 2021; Moore, 2007; Opitz & Witzel, 2005; Pool, 2017; Pownall et al., 2023; van den Berg, 2008; Yoon, 2015). In fact, context is viewed as so central to qualitative data reuse, that as Mauthner and Parry (2009) argue, context is not separate from qualitative data but must be treated as part of the data. Therefore, many authors believe that qualitative data stripped of their context cannot be reused (Coltart et al., 2013; Guishard, 2018; Mason, 2007; Moore, 2007; van den Berg, 2008; Vučković Juroš, 2022). For some, context is so important to shaping the data and how a researcher interprets it that they believe that it cannot be reused outside of its original context (Coltart et al., 2013; Guishard, 2018; Moore, 2007), particularly those from interpretive studies (Chauvette et al., 2019), and not for alternative research purposes (Yoon, 2015). Even when (some) context is provided, reuse may still be limited because a secondary researcher will not have first-hand knowledge of or experience in the original context of the study (Carusi & Jirotka, 2009; Chauvette et al., 2019; Corti & Thompson, 2004; DuBois et al., 2018; Irwin, 2013; Tamminen & Poucher, 2018). For example, Chauvette et al. (2019) note that trying to reuse field notes from ethnographic research would pose a challenge for a secondary researcher who was never immersed in the field where the data were generated. Attempting to do so, they conclude, might lead to misunderstandings about the meaning of the data. Karcher et al. (2021) note that some believe that even an original researcher cannot revisit their own data, let alone enable a secondary researcher to do so, once they are removed temporally from the context of the research. Some believe that the act of archiving qualitative data makes them “detached” and “disembodied” from both context and the researchers that created them. Camfield (2019) suggests that this makes shared qualitative data a “danger” (Broom et al., 2009) because data may have unintended meanings imposed upon them by secondary users (Branney et al., 2019; Feldman & Shaw, 2019).
Therefore, because it is understood by many to be a barrier to reuse, context is also understood to be a barrier to sharing data in the first place (Branney et al., 2019; Broom et al., 2009; Corti, 2006a; Huma & Joyce, 2022; Humphreys et al., 2021; Karhulahti, 2022; Mannheimer et al., 2018; Mauthner et al., 1998; Tamminen & Poucher, 2018; Yoon, 2015). Authors state that this is the case because it is difficult to fully document qualitative research context (Broom et al., 2009; Carusi & Jirotka, 2009; Coltart et al., 2013; Hardy et al., 2016; Joyce et al., 2022; Verburg et al., 2023). Doing so is highly time-consuming (Jones & Alexander, 2018; Weller, 2023), and as others argue, it will never be possible to fully and completely document the original research context (Coltart et al., 2013; Corti, 2006a; Feldman & Shaw, 2019; Hall, 2016; Mannheimer et al., 2018) and the experience of the original researcher within it (Pool, 2017; Tsai et al., 2016). Others note that documenting context can present ethical issues in regard to obligations to participants (Morris MacLean et al., 2019; Vučković Juroš, 2022). Raising another ethical issue, Alexander et al. (2020) state that a researcher may not want to share context-bound data for reuse when the secondary aim is to create generalizability (and this is at odds with the original research aims).
3.2.3. Ethical barriers to data sharing and reuse
Within our sample, ethical barriers are mainly articulated around data sharing and reuse (see Table 6). Some note that the qualitative nature of the data requires additional ethical sensitivity to that required for quantitative data (Branney et al., 2019; McLeod & O’Connor, 2021), especially regarding raw data (Branney et al., 2023). Others state that there are conflicting ethical considerations at play. For example, there is a conflict between the ethical position that data should be shared for maximum benefit to science and society with the ethical duty among researchers to protect the safety, wellbeing and dignity of their participants (Alexander et al., 2020; Jacobs et al., 2021; Jones & Alexander, 2018; Prosser et al., 2022), and to respect the relationships that are forged with them (Feldman & Shaw, 2019; Prosser et al., 2022), especially when they are collaborators in data creation and analysis (Guishard, 2018). Writing from a feminist decolonial standpoint, Prosser et al. (2022) state, “…it could be argued that the imperatives of open science may at times directly oppose and prevent researchers from being able to act in accordance with their research ethics of accountability and treat their participants’ data with consideration and care.” Jacobs et al. (2021) concur and conclude that “researchers’ ethical obligations to protect human participants and their communities ought to take priority over the sharing of information with research consumers.” Taking a more hardline approach, many authors within our study not only flag ethical issues as needing consideration but argue that they are often immovable barriers to sharing and reusing qualitative data (Abramson & Dohan, 2015; Closa, 2021; Jacobs et al., 2021; Joyce et al., 2022; Karcher et al., 2016; Karhulahti, 2022; Kern & Gleditsch, 2017; Lui et al., 2022; McGrath & Nilsonne, 2018; Perreault & Dienlin, 2022; Pool, 2017; Pownall et al., 2021; Pratt et al., 2020; Prosser et al., 2022; Steltenpohl et al., 2023; Tonnesson, 2012; Tucker, 2016; Vučković Juroš, 2022).
Others note the conflict between various institutional and governmental policies and laws that seek to safeguard privacy, on the one hand, and foster shared data on the other (Carusi & Jirotka, 2009; Reeves et al., 2023). Verburg (2023) notes that there may be legal constraints to sharing qualitative data, even when it is anonymized. Others point to the risk averse stance of institutional review boards (IRBs) as often being a barrier to qualitative data sharing (Mannheimer et al., 2018; McCarthy et al., 2023; Mozersky, Walsh, et al., 2020). In some research settings, the need to get approval from community ethics boards or commissions, like those that work on behalf of indigenous communities, prevents some researchers from pursuing data sharing in the first place (Mannheimer et al., 2018). Other concerns raised by this group of authors include a lack of clarity on who bears responsibility for what happens with shared data (Mozersky, Walsh, et al., 2020; Vučković Juroš, 2022), and concern for the misuse or misappropriation of shared data (Bishop, 2014; Carusi & Jirotka, 2009; Corti, 2006b; Feldman & Shaw, 2019; Field et al., 2021; Freese et al., 2022; Guishard, 2018; Kuula, 2011; McLeod & O’Connor, 2021; Mozersky, Parsons, et al., 2020; Pool, 2017) – either of which may breach the duty of care that researchers have with their participants, and the relationship of trust that was established between them. Carusi and Jirotka (2009) suggest that these issues arise because “existing practice[s]” for the ethical handling of qualitative data have not caught up with the technological advancements that have enabled data sharing: “…the technologies are pushing us beyond existing practice and is often challenging its moral grounds.”
3.2.3.1. Participant anonymity as a barrier to data sharing and reuse
Authors within our sample see the imperative to protect research participant anonymity, confidentiality and privacy as a key barrier to data sharing due to risks to participants (Broom et al., 2009; Campbell et al., 2019; Chauvette et al., 2019; Cliggett, 2013; Corti, 2000; Corti et al., 2000; Corti, 2012; Field et al., 2021; Freese et al., 2022; Hammersley, 1997; Huma & Joyce, 2022; Jones & Alexander, 2018; Joyce et al., 2022; Karcher et al., 2016; Kirilova & Karcher, 2017; Kuula, 2011; Mannheimer et al., 2018; McGrath & Nilsonne, 2018; McLeod & O’Connor, 2021; Monroe, 2018; Moravcsik, 2014a; Mozersky et al., 2021; Myrick, 2021; Opitz & Witzel, 2005; Parry & Mauthner, 2004; Prosser et al., 2022; Ruggiano & Perry, 2019; Talkad Sukumar et al., 2020; Tamminen & Poucher, 2018; Yoon, 2015), especially when data are audio and/or visual in nature (Carusi & Jirotka, 2009; Corti et al., 2000). Taking a hardline stance, Bishop (2009) believes that the act itself of archiving qualitative data “violates confidentiality”.
Concern about the risk of re-identification of participants after “anonymized” data is shared is a common theme (Branney et al., 2017; Campbell et al., 2019; Dienlin et al., 2021; Feldman & Shaw, 2019; Guishard, 2018; Humphreys et al., 2021; Jacobs et al., 2021; Opitz & Witzel, 2005; Pownall et al., 2021; Tsai et al., 2016). The spreading use of computational methods compounds this risk, because such methods can be used to reconstruct a de-identified participant profile from a dataset (Karhulahti, 2022). Reeves et al. (2023) point out the risk this poses to those who participate in life course studies, when data are shared repeatedly over a lengthy period (Reeves et al., 2023), while Tsai et al. (2016) note that sharing “unstructured data” like interview transcripts can compromise anonymity. This concern is especially great when the research is sensitive in nature (Jesser, 2011; Kirilova & Karcher, 2017; Mozersky et al., 2021), like that pertaining to health (Gupta et al., 2022), trauma (Campbell et al., 2022), survivors of violence (Campbell et al., 2019), or conducted within vulnerable communities (Lui et al., 2022; Pownall et al., 2021; Tucker, 2016). In some cases, reidentification may pose not just risks to privacy but to physical safety (Campbell et al., 2022; DuBois et al., 2018; Mozersky, Parsons, et al., 2020). In the case of research that deals with political and legal circumstances, risks to participants’ freedom and liberty may be present (Büthe & Jacobs, 2015; Corti et al., 2000; McCurdy & Ross, 2018), and risks in the context of political violence may not be fully knowable when data are shared (Büthe & Jacobs, 2015). Others may face less serious, though no less consequential, social risks if they are reidentified (Carusi & Jirotka, 2009). In some cases, it may not be possible to fully anticipate such risks, as social circumstances are ever changing, and what is perceived as safe data at the point of sharing may become risky at a later stage (Prosser et al., 2022). Therefore, some conclude that “sensitive research” is not aligned with data sharing (Campbell et al., 2019; Kirilova & Karcher, 2017). Some point out that these risks are (of course) even greater when raw data are shared (Branney et al., 2023; Fielding, 2004).
Beyond the risks posed to participants, many authors observe that anonymizing qualitative data for sharing is challenging (Carusi & Jirotka, 2009; Gervais et al., 2021), especially when it is sensitive data (Humphreys et al., 2021). The process is time-intensive (Campbell et al., 2019; Jacobs et al., 2021; Jones & Alexander, 2018; McCarthy et al., 2023; McCurdy & Ross, 2018; Prosser et al., 2022; Tsai et al., 2016; Weller, 2023) and costly (Camfield, 2019; Campbell et al., 2019; Jacobs et al., 2021; McCarthy et al., 2023; McCurdy & Ross, 2018; Prosser et al., 2022) – both of which can prohibit researchers from doing it. Some may not know how to go about effectively doing so (Freese et al., 2022), resulting in a risk of inconsistent de-identification (Antonio et al., 2019). For example, Campbell et al. (2022) state that there is a lack of confidence in practices of de- identification within trauma studies. Others observe that providing anonymity can be a challenge when some participants want to be associated with their data (Field et al., 2021), and that despite a researcher’s best efforts, participants may be able to identify each other and may identify themselves via social media (McCurdy & Ross, 2018).
When data is properly and sufficiently anonymized for sharing, this fact itself may be a barrier to reuse because it may be rendered virtually useless due to the lack of detail and context (Alexander et al., 2020; Campbell et al., 2019; Corti et al., 2000; McCarthy et al., 2023; McCurdy & Ross, 2018; Opitz & Witzel, 2005; Parry & Mauthner, 2004; Tsai et al., 2016; Verburg et al., 2023) – what Verburg et al. (2023) refer to as “information loss.” Others point out that the integrity and trustworthiness of the data may be compromised as a result (Antonio et al., 2019; Weller, 2023).
Antonio et al. (2019) note that data security practices at some repositories may also limit the reuse of archived data.
3.2.3.2. Participant consent as a barrier to data sharing and reuse
Participant consent is framed as a barrier to data sharing and reuse for a variety of reasons (Alexander et al., 2020; Bishop, 2014; Branney et al., 2019; Carusi & Jirotka, 2009; Chauvette et al., 2019; Corti, 2000, 2006b; Corti & Thompson, 2004; Feldman & Shaw, 2019; Jones & Alexander, 2018; Kern & Mustasilta, 2023; Mannheimer et al., 2018; McCarthy et al., 2023; Mozersky et al., 2021; Prosser et al., 2022; Ruggiano & Perry, 2019; Tamminen & Poucher, 2018). A key reason for this is that many authors believe that fully informed prior consent is impossible to establish for reuse because the consent established between participants and a researcher is not responsive to future scenarios of use and interpretation by other researchers (Bishop, 2009, 2014; Branney et al., 2019; Campbell et al., 2019; Feldman & Shaw, 2019; Grinyer, 2009; Kuula, 2011; McLeod & O’Connor, 2021; Mozersky et al., 2021; Parry & Mauthner, 2004; Pool, 2017). Sharing ethnographic fieldnotes as data raises additional challenges because the scope of participation is sometimes wide in a field setting (all people who pass before the ethnographer’s gaze for observation) and direct contact with all involved is not necessarily established (in public settings, for example) (Kern & Gleditsch, 2017). Kern and Gleditsch (2017) therefore conclude that one cannot share ethnographic fieldnotes as data because it is impossible to establish consent to share from all relevant persons. As Feldman and Shaw (2019) point out, the trust established through the consent process does not automatically extend to other researchers who might reuse data, and private sector reuse of data available through public institutions may “breach the guarantee of informed consent” (Feldman & Shaw, 2019, citing Bishop 2009 and Broom et al. 2009). Therefore, ‘one-off’ consent is not appropriate for data sharing and reuse. As Kern and Gleditsch (2017) state, it is unreasonable for a researcher to expect participants to consent to anyone and everyone having access to data that documents their lives. In a similar vein, Vučković Juroš (2022) notes that participant consent does not release researchers from the responsibility for caring for shared data and the trust-based relationships established with participants that made data generation possible, suggesting that primary researchers have a potentially unending duty of care to participants when data are shared. This is understood to be especially important when participants are located in marginalized or vulnerable communities, or the data are sensitive. Sharing such data may lead to the breaching of trust (Broom et al., 2009; Carusi & Jirotka, 2009) and “radical solidarities” (Guishard, 2018) established between researchers and participants/communities and may foster harm through unsanctioned reuse that might conflict with participant values or beliefs (Carusi & Jirotka, 2009; Guishard, 2018). As Guishard (2018) put it, “The last thing that racialized, marginalized, poor/working-class folks, dis/abled people, LGBTQIA persons, and folk at the intersections of these diverse groups need is more research about them, without them” (Guishard, 2018).
Additionally, authors note that a lack of familiarity with and understanding of data sharing and reuse practices among research participants can make establishing informed consent difficult (Carusi & Jirotka, 2009; VandeVusse et al., 2021). A lack of awareness on the part of researchers may also be a barrier to data sharing, when they fail to specifically ask for this during the consent process (Alexander et al., 2020; Karcher et al., 2022).
3.3. Enablers of transparency, reproducibility, and replication
Key enablers of reproducibility and replication of qualitative research identified in our included sources include data sharing (40.73% of included sources), data reusability (22.98%), open analysis (21.37%) and open methods (20.16%) (Table 5). In turn, our data show that established qualitative research practices can be key enablers of these. We found that the practice of documentation and the process of establishing rapport and trust with research participants are thought to enable data sharing (19.35% and 10.48% of all sources, respectively); and that authors believe that documentation and qualitative data coding enable open analysis (16.13%, 13.31% respectively) and open methods (16.13%, 10.08% respectively). Additionally, we found that many authors in our sample offer responses to key barriers to data sharing described in section 3.2, with participant anonymity, informed consent, ethics, and research context framed as enablers of data sharing and reusability (Table 6). We also found that qualitative research practices of coding (12.90% of included sources), documentation (18.95%) and reflexivity (14.11%) are framed as enablers of transparency (Table 7). In what follows, we provide detailed findings on the key enablers described here.



3.3.1. Data sharing as an enabler of transparency, reproducibility and replication
In general, and despite many barriers to making qualitative datasets available for reuse, there is a shared understanding that data sharing, wherever possible, would support transparency (Dienlin et al., 2021; Elman, 2014; Gupta et al., 2022; Rainey et al., 2022; Tonnesson, 2012), verification (Dienlin et al., 2021; Freese et al., 2022; Gupta et al., 2022; Mozersky, Walsh, et al., 2020) and traceability of findings in qualitative research (e.g., through a paper trail by implementing active citations, data citations, anonymized data and coded data together with coding strategies) (Dienlin et al., 2021; Freese et al., 2022). Some also suggest that data sharing can enable reproducibility of qualitative research (Verburg et al., 2023), because some think that replicating and/or reproducing research studies relies on and is possible only when data has been shared (Aguinis & Solarino, 2019; Anczyk et al., 2019; Crosas et al., 2018; Feldman & Shaw, 2019; Jacobs et al., 2021; Opitz & Witzel, 2005; Steinhardt, 2020). However, Tsai et al. (2016) point out that data sharing is not required for reproduction, and that detailed descriptions of the methods together with a reporting checklist is sufficient. Others suggest that even though replication or reproducibility does not apply to qualitative research, data sharing still promotes transparency, trustworthiness, quality and reuse (Antes et al., 2018), and enables secondary data analysis (Gupta et al., 2022; Mozersky, Walsh, et al., 2020).
3.3.2. Enablers of data sharing and reuse
However, most often raw data sharing is particularly challenging for qualitative researchers, as discussed in section 3.2. While many epistemological barriers to qualitative data sharing are believed to exist, some authors within our sample see the possibility for qualitative data sharing, but still state that it is conditional on the epistemological approach of the research (Alexander et al., 2020; Prosser et al., 2022). To enable data sharing in qualitative research, the reviewed literature points to anonymization, de-identification and minimization of data (i.e., not collecting unnecessary identifying information) (Cliggett, 2013; Corti et al., 2000; Lin, 2009; Mozersky, Walsh, et al., 2020) to tackle the challenge of possible re-identification of research participants from the shared material. Note, however, that at the same time authors stress the importance of balancing anonymity with reusability of qualitative data (i.e. maximizing anonymity might reduce the content or substance in the research material which will in turn limit reusability) (Neale & Bishop, 2012). In this context, fully informing research subjects about what exactly will happen to the data, how it will be de-identified and shared, in addition to obtaining consent for data collection, is described as an important enabler of qualitative data sharing (Branney et al., 2023; Campbell et al., 2019; Corti et al., 2000; Mozersky, Walsh, et al., 2020; Steltenpohl et al., 2023).
Campbell et al. (2019) suggest that participants should be actively involved in the decision- making process regarding data sharing and that offering different levels of consent (to how much data can be shared) can be useful. The practice of establishing rapport and trust with research participants is framed as important here, because this is the basis for having ethical discussions with participants about data sharing and reuse (Jones & Alexander, 2018; Mannheimer et al., 2018), and for involving them in decisions on these matters, which can then enable openness (Bishop, 2009; Campbell et al., 2019; Jones & Alexander, 2018; Roller & Lavrakas, 2018; VandeVusse et al., 2021). Managing access to the archived data and providing different levels of access to data can respond to some of the barriers to data sharing articulated in section 3.2 (Corti, 2012; Jones & Alexander, 2018; Mozersky, Walsh, et al., 2020). Here, authors suggest new approaches to managing access to data such as, for example, a membership model where only approved members who receive training and advice for managing and analyzing data would receive access (Corti, 2012). Others suggest graded levels of access were, depending on the type of the data, appropriate levels of openness can be chosen (Jones & Alexander, 2018). Here, the role of data repositories in enabling sharing and reuse is key, as asserted by several authors (Antonio et al., 2019; Bishop, 2005; Corti, 2005; DuBois et al., 2018; Jesser, 2011; Karcher et al., 2021, 2022; Mozersky et al., 2021). For example, the development of robust infrastructures for sharing and which include the provision of expertise of competent staff that can assist with preparation of data for long-term archiving (e.g., de-identification, organization and documentation of data) are mentioned as critical (Antonio et al., 2019; DuBois et al., 2018; Jesser, 2011; Mozersky et al., 2021). Authors also stress that it is important for researchers to actively collaborate with data curators and repositories throughout the archiving process (Antonio et al., 2019; Bishop, 2005; Karcher et al., 2021).
Additionally, capturing and conveying context information is a recurring theme in discussions around facilitating data sharing and reuse in qualitative research, and careful documentation of that information is framed as an important enabler (Bishop, 2006; Corti, 2006a; Jesser, 2011; Karcher et al., 2021; Kern & Mustasilta, 2023; Tsai et al., 2016), with some authors stressing that full and rich documentation is a prerequisite to data sharing (Corti, 2006b). For example, annotations to research material are mentioned as helpful in making data sharing possible (Jacobs et al., 2022; Karcher et al., 2016) and active citations – in-text links to data sources – are noted as one of the practices that can enable sharing of textual material (Karcher et al., 2016; Moravcsik, 2010). These observations link to the broader view that open methods and analyses are viewed by some as enablers of data reuse (Hesse et al., 2019; Kern & Mustasilta, 2023). Reflecting this, some note that the practice of researcher reflexivity can aid secondary analyses of shared data or any future work with the material by providing insight into the interpretive process and the role of researcher positionality in shaping it (Walters, 2009).
Finally, education and training of qualitative researchers (Karcher et al., 2022; Kirilova & Karcher, 2017; Mannheimer et al., 2018) that supports improved data management and early planning for data sharing are also enablers (Fielding, 2000; Jones & Alexander, 2018; Karcher et al., 2021; Karhulahti et al., 2022; Mannheimer et al., 2018). Karcher et al. (2022) assert that data repositories are well poised to improve the training in qualitative data management and sharing.
3.3.3. Flexibility and adaptation as enablers of transparency, reproducibility and replicability
Because qualitative practices are often ontologically and epistemologically different from methodologies of quantitative research, as discussed in section 3.2, there are still different views around what open and reproducible research practices are, or would be, within the epistemically diverse terrain of qualitative research (Field et al., 2021; Karcher et al., 2021). Therefore, several authors point out the importance of awareness raising and training in this area more broadly (Corti, 2006b; Field et al., 2021; Lui et al., 2022; Prosser et al., 2022; Walters, 2009). Providing guidance and workshops for researchers, reviewers, and journal editors (Corti, 2006b; Prosser et al., 2022), mentoring that respects epistemic diversity and helps to engage with open research (Field et al., 2021), developing community standards (Corti, 2006b) and getting hands-on help from the librarians and repositories (Karcher et al., 2021) are some of the core enablers that are thought to contribute to higher awareness and competencies among researchers and various stakeholders in open research.
However, just as training, support, and awareness raising are prominent topics, others urgently call for the increased flexibility of definitions and requirements for open research practices, which were developed predominantly with quantitative research in mind, and tailoring open research tools and solutions to fit qualitative approaches (Branney et al., 2023; Huma & Joyce, 2022; Kirilova & Karcher, 2017; Pownall et al., 2021; Pujol Priego et al., 2022; Torka et al., 2023). Authors point out that open science reforms should accommodate epistemically diverse perspectives (Pownall et al., 2021), that open research practices should be collectively constructed, and that infrastructures and workflows must be appropriate to, and maximize the potential of, the full range of different epistemic approaches (Pujol Priego et al., 2022). For example, some suggest that generalizability of qualitative research applies more to concepts and theories than it does to results, as is the case in quantitative research (Tamminen & Poucher, 2018), and that methods sharing and not data sharing could be more central to qualitative research (Tsai et al., 2016). Thus, authors recommend considering a range of sharing options and diversity of practices (Kirilova & Karcher, 2017; Lorenz & Holland, 2020) that ought to be accepted by those requiring openness (Büthe & Jacobs, 2015). For example, this could include, encouraging metadata sharing (Branney et al., 2023; Jones & Alexander, 2018), which can provide rich information about the context of the research process, including methodological and practical considerations that could have influenced the content (Jones & Alexander, 2018), especially when raw or derived data cannot be shared due to legal or ethical reasons. Additionally, Tsai et al. (2016) advocate for sharing other aspects of the research process, like coding queries, instead of datasets (Tsai et al., 2016).
Some authors mention the development of qualitative-specific infrastructures and templates as tailored solutions that would help enable transparency and reproducibility in qualitative research, for example reporting templates for sharing of the qualitative analytic process (Abramson & Dohan, 2015; Antonio et al., 2019; Chauvette et al., 2019; Field et al., 2021; Guishard, 2018; Karcher et al., 2021; Torka et al., 2023); among those are examples of codebooks and code scheme sharing (Field et al., 2021; Hocker et al., 2021; Kern & Mustasilta, 2023; Roberts et al., 2019). Preregistration has been proposed by some as an important enabler of transparency in qualitative research, provided it is adapted to the specific needs of qualitative research, e.g., through better tailored templates (Chin et al., 2020; Field et al., 2021; Haven et al., 2020; Kern et al., 2020; Ross & Ballsun-Stanton, 2021).
3.3.4. Enabling transparency through sharing the research and analytic processes
Several authors focus on the importance of research process transparency (Aguinis & Solarino, 2019; Dolan et al., 2023; Elman, 2014; Frohwirth et al., 2023; Hendren et al., 2023; Jacobs et al., 2021; Lui et al., 2022; S. Smith, 2018) for enabling open research, rather than focusing on sharing research outputs. As Frohwirth et al. (2023) point out, reproducibility might not be achievable in qualitative research, but transparency of the process is – and for that materials, procedures, and analysis elements that include coding schemes, memos and analytic plans are important to share openly. Reflecting this same approach, Fujiura (2015) defines transparency itself as the ability of the reader to follow and evaluate the research process as well as the interpretation of the study. Going further, for improving transparency and replicability in qualitative research, Closa (2021) suggests sharing every decision made during the research process. Some point to careful documentation of the research process – a practice that is already established within many qualitative methodologies – to illuminate the iterative process of qualitative research, and make it possible to follow and interpret the methodological and analytic steps in the study, which according to some, increase research transparency, reproducibility, replicability and rigor (Coombs, 2017; Friedhoff et al., 2013; Jesser, 2011; Karcher et al., 2021; Karhulahti et al., 2022; Kern & Mustasilta, 2023). Additionally, documentation is framed by some as important to ensure that findings are trustworthy, rigorous and of high quality (Dolan et al., 2023; Jacobs et al., 2021; H. J. Smith et al., 2008). Lui et al. (2022) argue that documentation practices are enablers of open research, generally, regardless of research method or epistemic approach.
Others focus specifically on fostering analytic transparency by providing the details of the analytic process. For example, standardized transcriptions and implementing practices for “intersubjective verifiability of analytic claims” (Huma & Joyce, 2022), code sharing (Renbarger et al., 2023) or providing information about the analysis software used (Paulus et al., 2017) are some of the suggestions for helping others to understand analytic and interpretive processes (Tamminen & Poucher, 2018). According to some, doing so can facilitate reproducibility (Huma & Joyce, 2022) or replicability (Anczyk et al., 2019; Andriopoulos & Slater, 2013; Closa, 2021; Tonnesson, 2012) of the findings, improve the quality and rigor of qualitative research (Coombs, 2017; Paulus et al., 2017; H. J. Smith et al., 2008; Thompson et al., 2004), reliability (Davies & Dodd, 2002), and can enable verification of results (Freese et al., 2022; Renbarger et al., 2023; Thompson et al., 2004) through the provision of paper trails (Freese et al., 2022) or materials sharing (Renbarger et al., 2023).
Many state that documenting research context in shared methods and analyses is critical and an important enabler (Davidson et al., 2017; Friedhoff et al., 2013; Jesser, 2011; Kern & Mustasilta, 2023). Additionally, some methods of generating qualitative data and documentation, including field notes (Cramer, 2015; Koch, 2006; Phillippi & Lauderdale, 2017) and the provision of thick description (Hadi & Closs, 2016; Hendren et al., 2023; Renbarger et al., 2023; Roller & Lavrakas, 2018) have been proposed as existing practices that can help register context information and through that, facilitate transparency, trustworthiness and reproducibility, particularly in ethnography and cultural anthropology. By providing context information, which captures details of the data collection and its setting (Bishop, 2007; Feldman & Shaw, 2019; Fielding, 2004; Friedhoff et al., 2013; Karcher et al., 2021; Kern & Gleditsch, 2017; Tamminen & Poucher, 2018; Walters, 2009), some argue that field notes can be used to aid multi-synthesis, secondary analyses or any future analyses of the material (Cramer, 2015; Phillippi & Lauderdale, 2017). Likewise, some point out that thick description can be similarly helpful by providing essential details on the research context, participants and analytic process (Renbarger et al., 2023). Even though journals often do not provide enough space for including these details in the manuscript itself, Renbarger et al. (2023) point out that they could be included as additional materials and/or in an online repository.
Related to documentation, some claim that sharing the coding process through codebooks and coding schemes can be done relatively easily and can make the research process more transparent and reproducible, including supporting re-use and helping to accurately replicate methodological aspects of the original study (Becker et al., 2021; Field et al., 2021; Hocker et al., 2021; Kern & Mustasilta, 2023; Roberts et al., 2019; Tsai et al., 2016). For example, some authors included guidance on how to keep a transparent codebook (e.g., for thematic analysis, Roberts et al., 2019) or how to use a newly developed coding tool (e.g., QualiCO, Hocker et al., 2021) to support sharing of coding schemes, since data repositories have historically been unable to support qualitative coding sharing (Hocker et al., 2020). Some authors identify that technological improvements of existing software and formats also aid coding sharing, such as the standardization across the major qualitative data analysis software (QDAS; e.g., applications such as NVivo and ATLAS.ti, Karcher et al., 2021). Other authors proposed a refinement of the coding process and methodology to support transparency in and reproducibility of the research process. Examples include establishing the concepts and links between categories and themes in clear ways (Rainey et al., 2022), in-depth training of multiple coders (Hendren et al., 2023) and developing a detailed codebook that can be used by all coders (Becker et al., 2021; Noret et al., 2022). Some suggest that automation has the potential to make the process of documentation, coding and analysis more transparent and replicable. Automating these processes, for example through applying text mining to documenting workflows or natural language processing algorithms to coding textual data, could increase the production and analytic transparency, and reduce biases and errors by creating a more consistent approach to working with data documentation and coding, and therefore facilitate replicability (Abram et al., 2020) and generalizability (Chakrabarti & Frye, 2017).
In addition to coding and documentation, reflexivity has been indicated as an existing and already broadly used practice in qualitative methodologies that is essential for increasing transparency and trustworthiness of qualitative research. According to the literature, reflexivity practice is important for identifying assumptions made during the research process (Doyle et al., 2020), to consider researcher biases (Kawaguchi-Suzuki et al., 2023), and one’s own a priori values (Bennett, 2021) and how they might influence the study outcomes. In other words, how the researcher as a person with their own ‘subjectivity’ affects the study (Hale et al., 2007; Kawaguchi- Suzuki et al., 2023; Thoresen & Öhlén, 2015). Some claim that in practicing reflexivity, qualitative researchers can make this subjective reality visible (Davies & Dodd, 2002) and manage the flexibility of process inherent in some approaches to qualitative research (Dubois & Gadde, 2014), which is in turn necessary for qualitative research rigor and trustworthiness (Davies & Dodd, 2002; Doyle et al., 2020; Hackett & Strickland, 2019; Mackieson et al., 2019; Steltenpohl et al., 2023) and makes the research process more transparent (Bennett, 2021; Doyle et al., 2020; Thoresen & Öhlén, 2015). Positionality statements are one method for including the practice of reflexivity in the research process (note, however, that although it gives opportunity for reflexivity practice, simply writing a positionality statement does not guarantee that one practices reflexivity throughout the research process). For example, Steltenpohl et al. (2023) argue that they give researchers an opportunity to discuss and justify decisions they made and their impact on the study. Field et al. (2021) suggest that reflexivity practices as described above, including positionality statements, are enablers of transparency and reproducibility of all research methodologies – not just those that are qualitative in nature.
Additionally, preregistration is framed as useful for reflexivity practice because, it is argued, it enables researchers to fully explore assumptions ahead of the research process (Chin et al., 2020), and through planning and considering many aspects of the study before it begins, potentially reduce the levels of subjectivity that shape the research process (Noret et al., 2022). Others state that preregistration enables clarity and transparency in the research process (Haven et al., 2020), including in the often-iterative process of theory development in qualitative research (Kern et al., 2020), and gives space for information that is usually not included in published papers (Branney et al., 2023).
Mackieson et al. (2019) suggest that reflexivity can be integrated through the recording of an audit trail for increased rigor and transparency. Audit trails were noted by several authors as a practice that contributes to increased trustworthiness (Hadi & Closs, 2016; Mackieson et al., 2019), transparency (Dolan et al., 2023; Hackett & Strickland, 2019; Hendren et al., 2023; Koch, 2006), reliability and validity (Tsai et al., 2016) of qualitative research. Some argue that such audit trails provide other researchers with information that aids in understanding how the primary researcher arrived at the final conclusions (Tsai et al., 2016) and can comprise, if possible, of data, including data reduction, analysis and synthesis, as well as elements typically included in documentation such as memos and notes (Dolan et al., 2023). Hackett and Strickland (2019) add that codes and visualizations can also function as an audit trail if they help explain how the data was interpreted.
4. Discussion and conclusion
4.1. Review aims and key findings
This review investigated how reproducibility and replicability are conceptualized and discussed in relation to qualitative research, and which factors and practices enable, and which are barriers to, the potential for either. While a large pool of literature now exists related to these topics, this literature is scattered across a variety of discipline-specific journals and to date no comprehensive review had been conducted to capture this literature and synthesize its themes and debates.
We found that conceptualizations of reproducibility and replicability that stem from positivist, quantitative standpoints are overwhelmingly framed as inappropriate practices and epistemic criteria for (most) qualitative research. When they are conceptualized in alternative ways that are adapted to the epistemic conditions, aims and practices of qualitative research, some believe they can be both applicable and appropriate. In other words, neither the form nor function of reproducibility and replicability are conceptualized in uniform or universal ways in the literature we reviewed. Rather, both are often framed as dependent upon the underlying epistemology and aims of the research in question. Therefore, there is no one way that either are conceptualized or positioned in relation to qualitative research. Instead, there are various conceptualizations that align with or contradict the diverse array of epistemologies, ontologies, research methods and types of data that make up qualitative research.
Related to this, we found that much of the literature illuminates well-established ontological and epistemological barriers to reproducibility, replicability, and Open Science as they relate to qualitative research. There is a degree of consensus around the position that the epistemological approaches of qualitive research are at odds with mainstream definitions of reproducibility and replicability, and established ways of achieving these through Open Science practices that center on outputs. To that end, we found that identified barriers to Open Science are concentrated around practices of data sharing and reuse. Here, the variety of epistemic approaches and research methodologies within qualitative research, the role of the researcher in co-creating data and in interpreting it in subjective ways, the centrality of the research context, the difficult nature of fully and transparently documenting both, and ethical concerns about participant anonymity, risks, and informed consent emerge as key barriers to sharing and reusing qualitative data.
Yet, at the same time, we found many articulations of enablers of reproducible, replicable, and open qualitative research within the literature. We found, reflecting established beliefs about data sharing generally, that qualitative data sharing is framed as an enabler of transparency, reproducibility, and replication; and we found articulations of key enablers of data sharing (and reuse) that are intended to respond to the issues framed as barriers to varying degrees. Processes for carefully de-identifying qualitative data; thoughtful, interactive and iterative informed consent practices; infrastructures providing careful, ethical and managed access to data; and the provision of context documentation are framed as enablers of data sharing and reuse. Additionally, reflecting our finding that adapting conceptualizations of reproducibility and replication to suit the epistemic aims and methodologies of qualitative research, we found that adapting the expectations and norms for Open Science practices is framed as an enabler of open qualitative research. Alternative sharing practices, like sharing metadata or composite data instead of sensitive data, focusing on open research processes instead of outputs, and the provision of epistemically responsive and flexible infrastructures, research support services, and tools and templates are understood to enable transparency, openness, and some forms of redoing. Notably, we found that established qualitative research practices, like setting and process documentation, and engaging in reflexivity and interrogations of researcher positionality are thought to enable open, and in some cases reproducible or replicable, qualitative research.
4.2. Interpretation of key findings
Reflecting broader trends within metascience and science policy, many of the papers included in our review demonstrate complex and diverse relationships between reproducibility, replicability, and Open Science. This suggests that the development of reproducibility and Open Science policies, and Open Science mandates, have impacted the qualitative research community. In particular, we see the emphasis on data sharing and reuse within our sample (more than half of all papers have this in focus) as evidence that open data policies and mandates can clash with the realities of qualitative research, but they have also prompted valuable discussions and debates about the underlying ontologies and epistemologies of qualitative research, of the quality criteria and research ethics, and the relationships between socio-technical systems and structures and research values, practices, norms and expectations. The steep growth in papers published since 2010 and included in our review signals the same.
The articulation of barriers to reproducibility, replicability and Open Science within our sample are evidence of a shared sense of imposition regarding science policies developed for positivist and quantitative research, and concern for their implications among the authors included in our review. One could argue, as do some included in this review, that epistemic exclusion within science reform movements generally and within Open Science specifically is a driver of universalist policies, practices, and expectations that are a poor fit for (much) qualitative research.
Not only is a sense of exclusion present within the reviewed literature but also concern for the negative implications that might follow it. For example, when data sharing is mandated, concerns about these risks, both among participants and researchers, may have a “chilling effect” on certain research topics or with certain communities (Branney et al., 2023; Hall, 2016; Monroe, 2018). Both members of vulnerable groups (Parry & Mauthner, 2004; VandeVusse et al., 2021) and the elite (Tsai et al., 2016) may be less willing to participate considering such risks (Reeves et al., 2023; Tsai et al., 2016) or may participate but not fully disclose (Mozersky et al., 2021). Pownall et al. (2021), citing Siegel and LaMarre (2019), expressed concerns for potential negative career ramifications for early career researchers whose research does not align with established Open Science practices. And, as was noted in section 3.2, it appears that Open Science policies and the science reform movement that drives them have historically had a blind spot for open processes while focusing mainly on open outputs.
It makes sense, then, that epistemic flexibility is framed as an enabler of reproducibility, replicability and open research by many authors included in our review. We see this theme both in terms of how these terms are conceptualized (adapting definitions to make them relevant to and feasible for the underlying epistemologies and methods of qualitative research), and in terms of how openness is practiced and accepted. This manifests in discussions of how data sharing or output sharing might look different for some qualitative research, depending on epistemology and ethical concerns, and is reflected in the community of practice that has grown around and fostered open qualitative research, which includes the focused work being done by researchers, repositories, libraries, and reproducibility networks to enable open qualitative research. It also manifests in the emphasis within the literature on enabling open research processes, both to support the reusability of open qualitative data and the potential reproducibility/replicability of qualitative research, but also as a valuable and acceptable way of practicing open research when data sharing isn’t feasible, and reproducibility or replicability are not relevant. As some authors included in our review point out, there are established practices within qualitative research methods that are meant to provide transparency of research design, data collection, interpretation and analysis, including the role of the researcher in shaping the full process. These practices, broadly described as documentation, reflexivity, and considering positionality, presented in the literature as enabling open methods and open analysis, precede Open Science. Although they have received far less attention within the mainstream science reform movement than have open data and Open Access publishing (see Chtena et al., 2023), we note that they could be incorporated as part of a more epistemically inclusive version of Open Science.
The considerable evidence presented in this review that demonstrates how qualitative research practices can enable openness suggests that the science reform movement and mainstream Open Science can be diversified and strengthened by incorporating the expertise of the qualitative research community. We echo the call to bring researcher reflexivity into quantitative research made by Field & Derksen (2020), Jamieseon et al. (2023), and Lammer (2023), in support of increased transparency, openness and research quality. In addition, we believe that the findings of this review make a case for developing standards and guidance for open methods and open analysis that center the longstanding practices of qualitative research that enable them. At the same time, we take caution from our findings and recognize the critical voices who assert that not all (qualitative) research is meant to be, nor can it be, reproducible, replicable, nor open. We agree with Leonelli (2018) that reproducibility cannot be a universally accepted criterion for research quality, and with Penders’ (2019) similar critique of universality of replication (see also Guttinger, 2020; Ulpts & Schneider, 2023). There is far too much evidence to the contrary.
4.3. Limitations
Having identified key themes in the included literature gives us confidence that our findings are valid and meaningful, however, as with any review study, we recognize that there are limitations to both the study design and our findings. Our review lacks language diversity and primarily includes literature that is indexed in established academic databases. While we did conduct a snowball search, included preprints and searched for relevant grey literature, we recognize that the included studies reflect varying degrees of academic privilege that likely impact their content. Including sources in other languages or acquiring them through other search methods might have changed the key findings and conclusions we are able to draw from them. Further, we recognize that the focus in the included literature on text-based data and the methods that generate them likely also influenced the themes we were able to identify. While we made every effort to rigorously co-create a code system and process that could be carried out systematically by various members of our study team, we acknowledge that this was not a perfect process and that some inconsistencies in our coding likely exist. And, while we attempted to achieve “radical reproducibility” (Ross-Hellauer et al., 2022) throughout our research process, by documenting our methodological choices, analytic debates and decisions, providing an extensive methods section and supplements, and sharing the maximum amount of data, materials and code possible, we suspect that others would still be unable to recreate exactly our process and/or findings, given our roles as researchers in the processes of data extraction, coding, analysis, and reporting, and the iterative nature of the coding and analytic process.
4.4. Implications for future research and policy
Our findings signal a critical need for consensus building within the science reform movement around the meanings and functions of reproducibility and replicability, which echoes the findings of Ulpts & Schneider (2024) from their more broad-based review of conceptualizations of these terms. This work must center epistemic diversity if it is to be broadly relevant and legitimate, and it must acknowledge that what these terms mean is always, already ontologically and epistemologically dependent. Our call is therefore not for singular definitions of these terms, but for a recognition of a set of definitions and functions that are understood to be relevant in certain ontological and epistemological contexts. Here, specifying different types of reproducibility and replicability is critical. Our constant struggle with terminology through writing this paper, often feeling that the words we have available to use – reproducibility and replicability – do not adequately capture the diversity of meanings and practices that we aimed to describe, is evidence of this.
They also signal, reflecting Leonelli’s (2022) assessment, a critical need for the development of epistemically inclusive Open Science practices, education, training, guidance, services, tools and templates. Epistemic diversity must be centered within the Open Science movement and its implementation if it is to succeed in its mission of making research (and scholarship) more open and accessible, generally speaking. But beyond this, our findings demonstrate the need for qualitative-specific versions of all the above, given the well-articulated epistemic mismatch between existing Open Science practices, services and infrastructure, and most qualitative research. Our experience in conducting this review has also surfaced the need for FAIR resources to support those who wish to practice open qualitative research. The literature suggests that resources are scattered around the web, with many embedded in or attached to published papers, and that they are not yet integrated into institutional guidance and trainings on Open Science. Gathering resources together and systematizing training around open practices for qualitative research would be of great benefit to the qualitative research community specifically and the scientific community more broadly.
Acknowledgments
The authors thank members of the TIER2 project consortium for their feedback and support throughout the conduct of this study, members of the Quala Lab working group, and attendees at STS Conference Graz, the annual meeting of the American Sociological Association meeting 2024, and STI Berlin 2024 for critical feedback on presentations of results. We are especially grateful for the close read and comments on an earlier draft of this paper given by Tamarinde Haven.
Data availability
The data associated with this study are published on Open Science Framework (Kormann et al., 2024) and available here: https://osf.io/javz2/. The code used to compute descriptive statistics and create figures is also available on the Open Science Framework here: https://osf.io/asgvc/ (Kormann, 2024).
Financial disclosure statement
This research is supported by the TIER2 project, funded by Horizon Europe research and innovation programme under grant agreement No 101094817. Funds provided to the QualiFAIR hub by University of Oslo, (hub-node funds for IT in research (2021-23, extended through 2024) also supported this project.
Competing interests
The authors declare that we hold no competing interests.
Notes
-
See, e.g., Academy of Science of South Africa, 2024; Directorate-General for Research and Innovation, 2024b; Environment and Climate Change Canada, 2022; Nelson, 2022; Policy and Strategy Branch, 2023.
-
See, e.g., Austrian Science Fund, 2024; European Commission, 2024; Policy and Strategy Branch, 2023; UK Research and Innovation, 2024; U.S. National Science Foundation, 2024.
-
See e.g., Open Research Europe, 2024; SpringerNature, 2024; The Royal Society, 2024. Open Research Europe, 2024; SpringerNature, 2024; The Royal Society, 2024.
-
See e.g., Chan Zuckerberg Initiative, 2024; Directorate-General for Research and Innovation (European Commission) et al., 2020; Horizon Europe, 2022, 2023; NWO, 2023; U.S. National Science Foundation, 2022, 2024.
-
In the interest of brevity, we do not define all specialist and academic terms in the text. Instead, we provide a glossary of key terms as Supplement 1.
-
See, for example, the virtual Quala Lab (↗️) and the QualiFAIR project at resource hub at the University of Oslo (↗️), the specialized Qualitative Data Repository based at Syracuse University in New York (↗️), the attention given to qualitative data sharing and reuse by UK Data Service (↗️), the development of best practices and services for qualitative data sharing offered by ICPSR at the University of Michigan (↗️), the qualitative data sharing webinar hosted by the Center for Open Science (↗️) and its introduction of a template for qualitative preregistration (↗️), and international conference sessions (↗️) and symposia (↗️) dedicated to such topics.
-
Numbers of individual publication outlets and authors were estimated by checking all identified names for duplicates with deviations in spelling, correcting these manually and then calculating the number of unique values.
References
Abram, M. D., Mancini, K. T., & Parker, R. D. (2020). Methods to Integrate Natural Language Processing Into Qualitative Research. In International Journal of Qualitative Methods (Vol. 19). ↗️
Abramson, C. M., & Dohan, D. (2015). Beyond Text: Using Arrays to Represent and Analyze Ethnographic Data. Sociological Methodology, 45(1), 272–319. ↗️
Academy of Science of South Africa. (2024). ACADEMY OF SCIENCE OF SOUTH AFRICA OPEN SCIENCE POLICY (p. 20). Department of Science and Innovation. ↗️
Aguinis, H., & Solarino, A. M. (2019). Transparency and replicability in qualitative research: The case of interviews with elite informants. Strategic Management Journal, smj.3015. ↗️
Alexander, S. M., Jones, K., Bennett, N. J., Budden, A., Cox, M., Crosas, M., Game, E. T., Geary, J., Hardy, R. D., Johnson, J. T., Karcher, S., Motzer, N., Pittman, J., Randell, H., Silva, J. A., da Silva, P. P., Strasser, C., Strawhacker, C., Stuhl, A., & Weber, N. (2020). Qualitative data sharing and synthesis for sustainability science. Nature Sustainability, 3(2), 81–88. Scopus. ↗️
Anczyk, A., Grzymała-Moszczyńska, H., Krzysztof-Świderska, A., & Prusak, J. (2019). The Replication Crisis and Qualitative Research in the Psychology of Religion. The International Journal for the Psychology of Religion, 29(4), 278–291. ↗️
Andriopoulos, C., & Slater, S. (2013). Exploring the landscape of qualitative research in international marketing: Two decades of IMR. In International Marketing Review (Vol. 30, Issue 4, pp. 384–412). ↗️
Antes, A. L., Walsh, H. A., Strait, M., Hudson-Vitale, C. R., & DuBois, J. M. (2018). Examining Data Repository Guidelines for Qualitative Data Sharing. Journal of Empirical Research on Human Research Ethics, 13(1), 61–73. Scopus. ↗️
Antonio, M. G., Schick-Makaroff, K., Doiron, J. M., Sheilds, L., White, L., & Molzahn, A. (2019). Qualitative Data Management and Analysis within a Data Repository. Western Journal of Nursing Research, 42(8), 640–648. ↗️
Athena RC, Directorate-General for Research and Innovation (European Commission), Know- Center, & PPMI. (2022). Assessing the reproducibility of research results in EU Framework Programmes for Research: Final report. Publications Office of the European Union. ↗️
Atmanspacher, H., & Maasen, S. (2015). Reproducibility: Principles, Problems, Practices, and Prospects (p. 574). Scopus. ↗️
Austrian Science Fund. (2024). Open Access Policy for Research Data. Austrian Science Fund (FWF). ↗️
Bahor, Z., Liao, J., Currie, G., Ayder, C., Macleod, M., McCann, S. K., Bannach-Brown, A., Wever, K., Soliman, N., Wang, Q., Doran-Constant, L., Young, L., Sena, E. S., & Sena, C. (2021). Development and uptake of an online systematic review platform: The early years of the CAMARADES Systematic Review Facility (SyRF). BMJ Open Science, 5(1). ↗️
Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452–454. ↗️
Bazzoli, A. (2022). Open science and epistemic pluralism: A tale of many perils and some opportunities. Industrial and Organizational Psychology, 15(4), 525–528. ↗️
Becker, M., Markowitz, J., Orsborn, S., Dasaraju, S., Lauder, L., & Nazha, I. (2021). Replicating the Resource Curse: A Qualitative Replication of Ross 2004 [Preprint]. MetaArXiv. ↗️
Befani, B. (2020). Quality of quality: A diagnostic approach to qualitative evaluation. In Evaluation (Vol. 26, Issue 3, pp. 333–349). https://doi.org/10.1177/1356389019898223 Begley, C. G., & Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature, 483(7391), 531–533. ↗️
Bennett, E. A. (2021). Open Science From a Qualitative, Feminist Perspective: Epistemological Dogmas and a Call for Critical Examination. Psychology of Women Quarterly, 45(4), 448–456. ↗️
Bienefeld, M., Boehm-Kasper, O., & Demmer, C. (2020). Highly recommended and yet neglected: The rarity of replication studies in educational science. In JOURNAL FOR EDUCATIONAL RESEARCH ONLINE-JERO (Vol. 12, Issue 3, pp. 3–22). WAXMANN VERLAG GMBH.
Biernacki, R. (2012). Reinventing Evidence in Social Inquiry: Decoding Facts and Variables. Springer.
Bishop, L. (2005). Protecting Respondents and Enabling Data Sharing: Reply to Parry and Mauthner. Sociology, 39(2), 333–336. ↗️
Bishop, L. (2006). A Proposal for Archiving Context for Secondary Analysis. Methodological Innovation Online, 1(2), 10–20. ↗️
Bishop, L. (2007). A Reflexive Account of Reusing Qualitative Data: Beyond Primary/Secondary Dualism. ↗️
Bishop, L. (2009). Ethical Sharing and Reuse of Qualitative Data. Australian Journal of Social Issues, 44(3), 255–272. ↗️
Bishop, L. (2014). Re-using Qualitative Data: A Little Evidence, On-going Issues and Modest Reflections. Studia Socjologiczne, 3(214). ↗️
Bochynska, A., Keeble, L., Halfacre, C., Casillas, J. V., Champagne, I.-A., Chen, K., Röthlisberger, M., Buchanan, E. M., & Roettger, T. B. (2023). Reproducible research practices and transparency across linguistics. Glossa Psycholinguistics, 2(1). ↗️
Borgstede, M., & Scholz, M. (2021). Quantitative and Qualitative Approaches to Generalization and Replication-A Representationalist View. In FRONTIERS IN PSYCHOLOGY (Vol. 12). FRONTIERS MEDIA SA. ↗️
Branney, P., Brooks, J., Kilby, L., Newman, K., Norris, E., Pownall, M., Talbot, C. V., Treharne, G. J., & Whitaker, C. M. (2023). Three steps to open science for qualitative research in psychology. Social and Personality Psychology Compass, 17(4), e12728. ↗️
Branney, P., Reid, K., Frost, N., Coan, S., Mathieson, A., & Woolhouse, M. (2019). A context- consent meta-framework for designing open (qualitative) data studies. In QUALITATIVE RESEARCH IN PSYCHOLOGY (Vol. 16, Issues 3, SI, pp. 483–502). ROUTLEDGE JOURNALS, TAYLOR & FRANCIS LTD. ↗️
Branney, P., Woolhouse, M., & Reid, K. (2017). The ‘innocent collection of details’ and journal requests to make qualitative datasets public post-consent: Open access data, potential author response and thoughts for future studies. QMiP Bulletin, 23, 19–23. ↗️
Broom, A., Cheshire, L., & Emmison, M. (2009). Qualitative Researchers’ Understandings of Their Practice and the Implications for Data Archiving and Sharing. Sociology, 43(6), 1163–1180. ↗️
Buckley, J., Adams, L., Aribilola, I., Arshad, I., Azeem, M., Bracken, L., Breheny, C., Buckley, C., Chimello, I., Fagan, A., Fitzpatrick, D. P., Garza Herrera, D., Gomes, G. D., Grassick, S., Halligan, E., Hirway, A., Hyland, T., Imtiaz, M. B., Khan, M. B., … Zhang, L. (2022). An assessment of the transparency of contemporary technology education research employing interview-based methodologies. In INTERNATIONAL JOURNAL OF TECHNOLOGY AND DESIGN EDUCATION (Vol. 32, Issue 4, pp. 1963–1982). SPRINGER. ↗️
Büthe, T., & Jacobs, A. (2015). Conclusion: Research Transparency For Diverse Discipline. Newsletter of the American Political Science Association Organized Section for Qualitative and Multi-Method Research, 13(1), 52–64. ↗️
Camfield, L. (2019). Rigor and Ethics in the World of Big-team Qualitative Data: Experiences From Research in International Development. American Behavioral Scientist, 63(5), 604– 621. ↗️
Campbell, R., Goodman-Williams, R., Engleton, J., Javorka, M., & Gregory, K. (2022). Open Science and Data Sharing in Trauma Research: Developing a Trauma-Informed Protocol for Archiving Sensitive Qualitative Data. Psychological Trauma: Theory, Research, Practice, and Policy. Scopus. ↗️
Campbell, R., Goodman-Williams, R., & Javorka, M. (2019). A Trauma-Informed Approach to Sexual Violence Research Ethics and Open Science. Journal of Interpersonal Violence, 34(23–24), 4765–4793. ↗️
Carusi, A., & Jirotka, M. (2009). From data archive to ethical labyrinth. Qualitative Research, 9(3), 285–298. ↗️
Chakrabarti, P., & Frye, M. (2017). A mixed-methods framework for analyzing text data: Integrating computational techniques with qualitative methods in demogra. In Demographic Research (Vol. 37, Issue 1, pp. 1351–1382). ↗️
Chan Zuckerberg Initiative. (2024). Open Science. Chan Zuckerberg Initiative. ↗️
Chauvette, A., Schick-Makaroff, K., & Molzahn, A. E. (2019). Open Data in Qualitative Research. International Journal of Qualitative Methods, 18, 160940691882386. ↗️
Chin, J., DeHaven, A. C., Heycke, T., Holcombe, A. O., Mellor, D. T., Pickett, J., Steltenpohl, C. N., Vazire, S., & Zeiler, K. (2020). Improving the credibility of empirical legal research: Practical suggestions for researchers, journals, and law schools. ↗️
Chtena, N., Alperin, J. P., Morales, E., Fleerackers, A., Dorsch, I., Pinfield, S., & Simard, M.-A. (2023). The neglect of equity and inclusion in open science policies of Europe and the Americas. SciELO Preprints. ↗️
Clemens, M. A. (2017). THE MEANING OF FAILED REPLICATIONS: A REVIEW AND PROPOSAL. Journal of Economic Surveys, 31(1), 326–342. ↗️
Cliggett, L. (2013). Qualitative Data Archiving in the Digital Age: Strategies for Data Preservation and Sharing. The Qualitative Report, 18(24), 1–11. ↗️
Closa, C. (2021). Planning, implementing and reporting: Increasing transparency, replicability and credibility in qualitative political science research. In EUROPEAN POLITICAL SCIENCE (Vol. 20, Issue 2, pp. 270–280). PALGRAVE MACMILLAN LTD. ↗️
Cole, N. L., Ulpts, S., Ross-Hellauer, T., Bochynska, A., & Klebel, T. (2023). Integrative review of conceptions and facilitators of and barriers to reproducibility of qualitative research. ↗️
Coltart, C., Henwood, K., & Shirani, F. (2013). Qualitative Sekundäranalyse in schwierigen Zeiten: Ethische, fachliche und methodologische ÜberlegungenQualitative Secondary Analysis in Austere Times: Ethical, Professional and Methodological Considerations. Historical Social Research, 38, 271292. ↗️
Coombs, C. (2017). Coherence and transparency: Some advice for qualitative researchers. Production, 27(0). ↗️
Corti, L. (2000). Progress and Problems of Preserving and Providing Access to Qualitative Data for Social Research—The International Picture of an Emerging Culture. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, Vol 1, No 3 (2000): Text . Archive .ReAnalysis. ↗️
Corti, L. (2005). User Support. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(2), Article 2. ↗️
Corti, L. (2006a). Editorial. Methodological Innovation Online, 1(2), 1–9. ↗️
Corti, L. (2006b). Qualitative Archiving and Data Sharing: Extending the Reach and Impact of Qualitative Data. IASSIST Quarterly, 29(3), 8. ↗️
Corti, L. (2012). Recent developments in archiving social research. International Journal of Social Research Methodology, 15(4), 281–290. ↗️
Corti, L., Day, A., & Backhouse, G. (2000). Confidentiality and Informed Consent: Issues for Consideration in the Preservation of and Provision of Access to Qualitative Data Archives. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, Vol 1, No 3 (2000): Text . Archive . ReAnalysis. ↗️
Corti, L., & Thompson, P. (2004). Secondary Analysis of Archived Data. In Qualitative Research Practice (pp. 297–313). SAGE Publications Ltd. ↗️
Cramer, K. (2015). Transparent explanations, yes. Public transcripts and fieldnotes, no: Ethnographic research on public opinion. Newsletter of the American Political Science Association Organized Section for Qualitative and Multi-Method Research, 13(1), 17–20. ↗️
Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (Fifth edition). SAGE.
Crosas, M., Gautier, J., Karcher, S., Kirilova, D., Otalora, G., & Schwartz, A. (2018). Data policies of highly-ranked social science journals [Preprint]. SocArXiv. ↗️
Davidson, J., Thompson, S., & Harris, A. (2017). Qualitative Data Analysis Software Practices in Complex Research Teams: Troubling the Assumptions About Transparency and Portability. In QUALITATIVE INQUIRY (Vol. 23, Issues 10, SI, pp. 779–788). SAGE PUBLICATIONS INC. ↗️
Davies, D., & Dodd, J. (2002). Qualitative Research and the Question of Rigor. Qualitative Health Research, 12, 279–289. ↗️
Deterding, N. M., & Waters, M. C. (2021). Flexible Coding of In-depth Interviews: A Twenty- first-century Approach. Sociological Methods & Research, 50(2), 708–739. ↗️
Dienlin, T., Johannes, N., Bowman, N. D., Masur, P. K., Engesser, S., Kümpel, A. S., Lukito, J., Bier, L. M., Zhang, R., Johnson, B. K., Huskey, R., Schneider, F. M., Breuer, J., Parry, D.
A., Vermeulen, I., Fisher, J. T., Banks, J., Weber, R., Ellis, D. A., … de Vreese, C. (2021). An Agenda for Open Science in Communication. Journal of Communication, 71(1). Scopus. ↗️
Directorate-General for Research and Innovation. (2024a). Trends for open access to publications. Research and Innovation. ↗️
Directorate-General for Research and Innovation. (2024b, September 20). Open Science— European Commission. Research and Innovation. ↗️
Directorate-General for Research and Innovation (European Commission), Iagher, R., Monachello, R., Warin, C., Delaney, N., & Tornasi, Z. (2020). Science with and for society in Horizon 2020: Achievements and recommendations for Horizon Europe. Publications Office of the European Union. ↗️
Dolan, S., Nowell, L., & Moules, N. J. (2023). Interpretive description in applied mixed methods research: Exploring issues of fit, purpose, process, context, and design. Nursing Inquiry, 30(3), e12542. ↗️
Doyle, L., McCabe, C., Keogh, B., Brady, A., & McCann, M. (2020). An overview of the qualitative descriptive design within nursing research. Journal of Research in Nursing : JRN, 25(5), 443–455. ↗️
Drummond, C. (2019). Is the drive for reproducible science having a detrimental effect on what is published? Learned Publishing, 32(1), 63–69. ↗️
Dubois, A., & Gadde, L.-E. (2014). “Systematic combining”-A decade later. In JOURNAL OF BUSINESS RESEARCH (Vol. 67, Issue 6, pp. 1277–1284). ELSEVIER SCIENCE INC. ↗️
DuBois, J. M., Strait, M., & Walsh, H. (2018). Is It Time to Share Qualitative Research Data? Qualitative Psychology, 5(3), 380–393. ↗️
Elman, C. (2014). Data Access and Research Transparency in the Qualitative Tradition. Political Science and Politics, 47, 43. ↗️
Environment and Climate Change Canada. (2022). Environment and Climate Change Canada Open Science Action Plan: 2021-2026 (p. 11). Environment and Climate Change Canada. ↗️
European Commission. (2024). Open science in Horizon Europe. European Research Executive Agency. ↗️
Evans, T. R., Branney, P., Clements, A., & Hatton, E. (2023). Improving evidence-based practice through preregistration of applied research: Barriers and recommendations.
Accountability in Research, 30(2), 88–108. https://doi.org/10.1080/08989621.2021.1969233 Fecher, B., Friesike, S., & Hebing, M. (2015). What Drives Academic Data Sharing? PLOS ONE. ↗️
Feldman, S., & Shaw, L. (2019). The Epistemological and Ethical Challenges of Archiving and Sharing Qualitative Data. American Behavioral Scientist, 63(6), 699–721. ↗️
Field, S. M., & Derksen, M. (2020). Experimenter as automaton; experimenter as human: Exploring the position of the researcher in scientific research. European Journal for Philosophy of Science, 11(1), 11. ↗️
Field, S. M., Van Ravenzwaaij, D., Pittelkow, M.-M., Hoek, J. M., & Derksen, M. (2021). Qualitative Open Science – Pain Points and Perspectives. ↗️
Fielding, N. (2000). The Shared Fate of Two Innovations in Qualitative Methodology: The Relationship of Qualitative Software and Secondary Analysis of Archived Qualitative Data. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 1(3), 12.
Fielding, N. (2004). Getting the most from archived qualitative data: Epistemological, practical and professional obstacles. International Journal of Social Research Methodology, 7(1), 97–104. ↗️
Freese, J., & Peterson, D. (2017). Replication in Social Science. Annual Review of Sociology, 43(1), 147–165. ↗️
Freese, J., Rauf, T., & Voelkel, J. G. (2022). Advances in transparency and reproducibility in the social sciences. Social Science Research, 107, 102770. ↗️
Friedhoff, S., Meier Zu Verl, C., Pietsch, C., Meyer, C., Vompras, J., & Liebig, S. (2013). Replicability and Comprehensibility of Social Research and Its Technical Implementation. SSRN Electronic Journal. ↗️
Frohwirth, L., Karcher, S., & Lever, T. A. (2023). A Transparency Checklist for Qualitative Research. SocArXiv. ↗️
Fujiura, G. T. (2015). Perspectives on the publication of qualitative research. Intellectual and Developmental Disabilities, 53(5), 323–328. APA PsycInfo <2015>. ↗️
Gervais, S. J., Baildon, A. E., & Lorenz, T. K. (2021). On Methods and Marshmallows: A Roadmap for Science That Is Openly Feminist and Radically Open. Psychology of Women Quarterly, 45(4), 430–447. ↗️
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science Translational Medicine, 8(341). ↗️
Grinyer, A. (2009). The ethics of the secondary analysis and further use of qualitative data. Social Research Update, 56(4), 1–4.
Guishard, M. A. (2018). Now’s not the time! Qualitative data repositories on tricky ground: Comment on Dubois et al. (2018). Qualitative Psychology, 5(3), 402–408. ↗️
Gundersen, O. E. (2021). The fundamental principles of reproducibility. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(2197), 20200210. ↗️
Gupta, A., Lai, A., Mozersky, J., Ma, X., Walsh, H., & DuBois, J. M. (2022). Enabling qualitative research data sharing using a natural language processing pipeline for deidentification: Moving beyond HIPAA Safe Harbor identifiers (vol 4, ooab069, 2021). In JAMIA OPEN (Vol. 5, Issue 1). OXFORD UNIV PRESS. ↗️
Guttinger, S. (2020). The limits of replicability. European Journal for Philosophy of Science, 10(2), 10. ↗️
Hackett, A., & Strickland, K. (2019). Using the framework approach to analyse qualitative data: A worked example. Nurse Researcher, 26(2), 8–13. ↗️
Hadi, M. A., & Closs, S. J. (2016). Ensuring rigour and trustworthiness of qualitative research in clinical pharmacy. International Journal of Clinical Pharmacy, 38(3), 641–646. ↗️
Hair, K., Bahor, Z., Macleod, M., Liao, J., & Sena, E. S. (2021). The Automated Systematic Search Deduplicator (ASySD): A rapid, open-source, interoperable tool to remove duplicate citations in biomedical systematic reviews (p. 2021.05.04.442412). bioRxiv. ↗️
Hale, E. D., Treharne, G. J., & Kitas, G. D. (2007). Qualitative methodologies I: asking research questions with reflexive insight. Musculoskeletal Care, 5(3), 139–147. ↗️
Hall, P. A. (2016). Transparency, Research Integrity and Multiple Methods. APSA-CP: Comparative Politics Newsletter.
Hammersley, M. (1997). Qualitative Data Archiving: Some Reflections on its Prospects and Problems. Sociology, 31(1), 131–142. ↗️
Hardy, L. J., Hughes, A., Hulen, E., & Schwartz, A. L. (2016). Implementing Qualitative Data Management Plans to Ensure Ethical Standards in Multi-Partner Centers. In JOURNAL OF EMPIRICAL RESEARCH ON HUMAN RESEARCH ETHICS (Vol. 11, Issue 2, pp. 191–198). SAGE PUBLICATIONS INC. ↗️
Haven, T. L., Errington, T. M., Gleditsch, K. S., Van Grootel, L., Jacobs, A. M., Kern, F. G., Piñeiro, R., Rosenblatt, F., & Mokkink, L. B. (2020). Preregistering Qualitative Research: A Delphi Study. International Journal of Qualitative Methods, 19, 160940692097641. ↗️
Haven, T. L., & Van Grootel, L. (2019). Preregistering qualitative research. In ACCOUNTABILITY IN RESEARCH-ETHICS INTEGRITY AND POLICY (Vol. 26, Issue 3, pp. 229–244). TAYLOR & FRANCIS INC. ↗️
Hendren, K., Newcomer, K., Pandey, S. K., Smith, M., & Sumner, N. (2023). How qualitative research methods can be leveraged to strengthen mixed methods research in public policy and public administration? In Public Administration Review (Vol. 83, Issue 3, pp. 468–485). ↗️
Hesse, A., Glenna, L., Hinrichs, C., Chiles, R., & Sachs, C. (2019). Qualitative Research Ethics in the Big Data Era. American Behavioral Scientist, 63(5), 560–583. ↗️
Hocker, J., Bipat, T., McDonald, D. W., & Zachry, M. (2021). Evaluating QualiCO: an ontology to facilitate qualitative methods sharing to support open science. In JOURNAL OF INTERNET SERVICES AND APPLICATIONS (Vol. 12, Issue 1). SOC BRASILEIRA COMPUTACAO. ↗️
Hocker, J., Bipat, T., Zachry, M., & McDonald, D. W. (2020). Sharing your coding schemas: Developing a Platform to fit within the Qualitative Research Workflow. In PROCEEDINGS OF THE 16TH INTERNATIONAL SYMPOSIUM ON OPEN COLLABORATION (OPENSYM). ASSOC COMPUTING MACHINERY. ↗️
Hoogeveen, S., & Elk, M. van. (2021). Advancing the Cognitive Science of Religion through Replication and Open Science. ↗️
Horizon Europe. (2022). Increasing the reproducibility of scientific results. EU Funding & Tenders Portal. ↗️
Horizon Europe. (2023). Support to the development and implementation of policies and practices for reproducibility of scientific results. EU Funding & Tenders Portal. ↗️
Hostler, T. J. (2023). The Invisible Workload of Open Research. Journal of Trial and Error, 4(1), 21–36. ↗️
Høyland, S. A., Hagen, J. M., & Engelbach, W. (2017). Developing and Applying a Framework for Assessing the Research Quality of Qualitative Project Methods in the EU Project SECUR-ED. In SAGE Open (Vol. 7, Issue 2). ↗️
Huma, B., & Joyce, J. B. (2022). ‘One size doesn’t fit all’: Lessons from interaction analysis on tailoring Open Science practices to qualitative research. British Journal of Social Psychology. ↗️
Humphreys, L., Lewis, N. A., Sender, K., & Won, A. S. (2021). Integrating Qualitative Methods and Open Science: Five Principles for More Trustworthy Research*. Journal of Communication, jqab026. ↗️
Irwin, S. (2013). Qualitative secondary data analysis: Ethics, epistemology and context. Progress in Development Studies, 13(4), 295–306. ↗️
Jacob, A., Kapiszewski, D., & Karcher, S. (2021). Using Annotation for Transparent Inquiry (ATI) to Teach Qualitative Research Methods. ↗️
Jacobs, A. M., Büthe, T., Arjona, A., Arriola, L. R., Bellin, E., Bennett, A., Björkman, L., Bleich, E., Elkins, Z., Fairfield, T., Gaikwad, N., Greitens, S. C., Hawkesworth, M., Herrera, V., Herrera, Y. M., Johnson, K. S., Karakoç, E., Koivu, K., Kreuzer, M., … Yashar, D. J. (2021). The Qualitative Transparency Deliberations: Insights and Implications. Perspectives on Politics, 19(1), 171–208. ↗️
Jacobs, A. M., Kapiszewski, D., & Karcher, S. (2022). Using Annotation for Transparent Inquiry (ATI) to Teach Qualitative Research Methods. In PS-POLITICAL SCIENCE & POLITICS (Vol. 55, Issue 1, pp. 216–220). CAMBRIDGE UNIV PRESS. ↗️
Jamieson, M. K., Govaart, G. H., & Pownall, M. (2023). Reflexivity in quantitative research: A rationale and beginner’s guide. Social and Personality Psychology Compass, 17(4), e12735. ↗️
Jesser, A. C. (2011). Archiving Qualitative Data: Infrastructure, Acquisition, Documentation, Distribution. Experiences from WISDOM, the Austrian Data Archive. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 12(3), Article 3. ↗️
Jones, K., & Alexander, S. M. (2018). Qualitative data sharing and re-use for socio-environmental systems research: A synthesis of opportunities, challenges, resources and approaches. ↗️
Joyce, J. B., Douglass, T., Benwell, B., Rhys, C. S., Parry, R., Simmons, R., & Kerrison, A. (2022). Should we share qualitative data? Epistemological and practical insights from conversation analysis. International Journal of Social Research Methodology, 0(0), 1–15. ↗️
Karcher, S., Kirilova, D., Pagé, C., & Weber, N. (2021). How Data Curation Enables Epistemically Responsible Reuse of Qualitative Data. The Qualitative Report. ↗️
Karcher, S., Kirilova, D., & Weber, N. (2016). Beyond the matrix: Repository services for qualitative data. IFLA Journal, 42(4), 292–302. ↗️
Karcher, S., Secen, S., & Weber, N. (2022). Protecting Sensitive Data Early in the Research Data Lifecycle [Preprint]. SocArXiv. ↗️
Karhulahti, V. (2022). Reasons for qualitative psychologists to share human data. British Journal of Social Psychology, bjso.12573. ↗️
Karhulahti, V., Branney, P., Siutila, M., & Syed, M. (2022). A Primer for Choosing, Designing and Evaluating Registered Reports for Qualitative Methods [Preprint]. MetaArXiv. ↗️
Kawaguchi-Suzuki, M., Blakely, M. L., Childs-Kean, L. M., Devraj, R., Frenzel, J. E., Medina, M. S., Pate, A. N., Thurston, M. M., & Franks, A. M. (2023). Guidance for Qualitative Research Manuscripts in Pharmacy Education. American Journal of Pharmaceutical Education, 87(7), 100089. ↗️
Kern, F. G., & Gleditsch, K. S. (2017). Exploring Pre-registration and Pre-analysis Plans for Qualitative Inference. ↗️
Kern, F. G., Haven, T., Rosenblatt, F., & Pineiro, R. (2020). Qualitative Preregistration. ↗️
Kern, F. G., & Mustasilta, K. (2023). Beyond Replication: Secondary Qualitative Data Analysis in Political Science. In COMPARATIVE POLITICAL STUDIES (Vol. 56, Issue 8, pp. 1224– 1256). SAGE PUBLICATIONS INC. ↗️
Kirilova, D., & Karcher, S. (2017). Rethinking Data Sharing and Human Participant Protection in Social Science Research: Applications from the Qualitative Realm. Data Science Journal, 16, 43. ↗️
Kivunja, C., & Kuyini, A. B. (2017). Understanding and Applying Research Paradigms in Educational Contexts. International Journal of Higher Education, 6(5), 26. ↗️
Koch, T. (2006). Establishing rigour in qualitative research: The decision trail. Journal of Advanced Nursing, 53(1), 91–100. ↗️
Kormann, E. (2024). Code for Quantitative Analysis within Integrative Review [Dataset]. OSF. ↗️
Kormann, E., Cole, N. L., Ulpts, S., Bochynska, A., Good, M., & Ross-Hellauer, T. (2024). Open Data Set from Integrative Review [Dataset]. OSF. ↗️
Kuula, A. (2011). Methodological and Ethical Dilemmas of Archiving Qualitative Data. IASSIST Quarterly, 34(3–4), 12. ↗️
Lammer, L. (2023). Why we should all be reflexive: The case for reflexivity in quantitative science. OSF. ↗️
Leonelli, S. (2018). Rethinking Reproducibility as a Criterion for Research Quality. In L. Fiorito, S. Scheall, & C. E. Suprinyak (Eds.), Research in the History of Economic Thought and Methodology (Vol. 36, pp. 129–146). Emerald Publishing Limited. ↗️
Leonelli, S. (2022). Open Science and Epistemic Diversity: Friends or Foes? Philosophy of Science, 89(5), 991–1001. ↗️
Lin, L.-C. (2009). Data Management and Security in Qualitative Research. Dimensions of Critical Care Nursing : DCCN, 28, 132–137. ↗️
Lorenz, T. K., & Holland, K. J. (2020). Response to Sakaluk (2020): Let’s Get Serious About Including Qualitative Researchers in the Open Science Conversation. Archives of Sexual Behavior, 49(8), 2761–2763. ↗️
Lui, P. P., Gobrial, S., Pham, S., Giadolor, W., Adams, N., & Rollock, D. (2022). Open Science and Multicultural Research: Some Data, Considerations, and Recommendations. Cultural Diversity and Ethnic Minority Psychology, 28(4), 567–586. ↗️
Mackieson, P., Shlonsky, A., & Connolly, M. (2019). Increasing rigor and reducing bias in qualitative research: A document analysis of parliamentary debates using applied thematic analysis. In Qualitative Social Work (Vol. 18, Issue 6, pp. 965–980). ↗️
Makel, M. C., Meyer, M. S., Simonsen, M. A., Roberts, A. M., & Plucker, J. A. (2022). Replication is relevant to qualitative research COMMENT. In EDUCATIONAL RESEARCH AND EVALUATION (Vol. 27, Issues 1–2, SI, pp. 215–219). ROUTLEDGE JOURNALS, TAYLOR & FRANCIS LTD. ↗️
Malich, L., & Rehmann-Sutter, C. (2022). Metascience Is Not Enough – A Plea for Psychological Humanities in the Wake of the Replication Crisis. Review of General Psychology, 26(2), 261–273. ↗️
Mannheimer, S., Pienta, A., Kirilova, D., Elman, C., & Wutich, A. (2018). Qualitative Data Sharing: Data Repositories and Academic Libraries as Key Partners in Addressing Challenges. American Behavioral Scientist, 1–22. ↗️
Mason, J. (2007, May 30). “Re-Using” Qualitative Data: On the Merits of an Investigative Epistemology [Text.Article]. Sociological Research Online. ↗️
Matarese, V. (2022). Kinds of Replicability: Different Terms and Different Functions. Axiomathes, 32(2), 647–670. ↗️
Mauthner, N. S., & Parry, O. (2009). Qualitative data preservation and sharing in the social sciences: On whose philosophical terms? The Australian Journal of Social Issues, 44(3).
Mauthner, N. S., Parry, O., & Backett-Milburn, K. (1998). The Data are Out there, or are They? Implications for Archiving and Revisiting Qualitative Data. Sociology, 32(4), 733–745. ↗️
McCarthy, M., Gillies, K., Rousseau, N., Wade, J., Gamble, C., Toomey, E., Matvienko-Sikar, K., Sydes, M., Dowling, M., Bryant, V., Biesty, L., & Houghton, C. (2023). Qualitative data sharing practices in clinical trials in the UK and Ireland: Towards the production of good practice guidance. HRB Open Research, 6, 10. ↗️
McCurdy, S. A., & Ross, M. W. (2018). Qualitative data are not just quantitative data with text but data with context: On the dangers of sharing some qualitative data: Comment on DuBois et al. (2018). In Qualitative Psychology (Vol. 5, Issue 3, pp. 409–411). ↗️
McGrath, C., & Nilsonne, G. (2018). Data sharing in qualitative research: Opportunities and concerns. MedEdPublish, 7, 255. ↗️
McLeod, J., & O’Connor, K. (2021). Ethics, archives and data sharing in qualitative research. Educational Philosophy and Theory, 53(5), 523–535. ↗️
Monroe, K. R. (2018). The Rush to Transparency: DA-RT and the Potential Dangers for Qualitative Research. In PERSPECTIVES ON POLITICS (Vol. 16, Issue 1, pp. 141–148). CAMBRIDGE UNIV PRESS. ↗️
Moore, N. (2007, May 30). (Re)Using Qualitative Data? [Text.Article]. Sociological Research Online. ↗️
Moravcsik, A. (2010). Active Citation: A Precondition for Replicable Qualitative Research. In PS- POLITICAL SCIENCE & POLITICS (Vol. 43, Issue 1, pp. 29–35). CAMBRIDGE UNIV PRESS. ↗️
Moravcsik, A. (2014a). Trust, but Verify: The Transparency Revolution and Qualitative International Relations. Security Studies, 23(4), 663–688. ↗️
Moravcsik, A. (2014b). Transparency: The Revolution in Qualitative Research. In PS-POLITICAL SCIENCE & POLITICS (Vol. 47, Issue 1, pp. 48–53). CAMBRIDGE UNIV PRESS. ↗️
Morris MacLean, L., Posner, E., Thomson, S., & Wood, E. J. (2019). Research Ethics and Human Subjects: A Reflexive Openness Approach. SSRN Electronic Journal. ↗️
Mozersky, J., McIntosh, T., Walsh, H. A., Parsons, M. V., Goodman, M., & DuBois, J. M. (2021). Barriers and facilitators to qualitative data sharing in the United States: A survey of qualitative researchers. PLoS ONE, 16(12 December). Scopus. ↗️
Mozersky, J., Parsons, M., Walsh, H., Baldwin, K., McIntosh, T., & DuBois, J. M. (2020). Research Participant Views regarding Qualitative Data Sharing. Ethics & Human Research, 42(2), 13–27. ↗️
Mozersky, J., Walsh, H., Parsons, M., McIntosh, T., Baldwin, K., & DuBois, J. M. (2020). Are we ready to share qualitative research data? Knowledge and preparedness among qualitative researchers, IRB members, and data repository curators. IASSIST Quarterly, 43(4), 1–23. ↗️
Myrick, R. (2021). Reflections on Using Annotation for Transparent Inquiry in Mixed-Methods Research. PS: Political Science & Politics, 54(3), 492–495. ↗️
Neale, B., & Bishop, L. (2012). The Ethics of Archiving and Re-Using Qualitative Longitudinal Data: A Stakeholder Approach. Timescapes Methods Guides Series.
Nelson, A. (2022). MEMORANDUM FOR THE HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES: Ensuring Free, Immediate, and Equitable Access to Federally Funded Research (p. 8) [Memorandum]. Office of Science and Technology Policy, Executive Office of the President of the United States. ↗️
Nixon, A., & Power, C. (2007). Towards a framework for establishing rigour in a discourse analysis of midwifery professionalisation. Nursing Inquiry, 14(1), 71–79. ↗️
Noret, N., Hunter, S. C., Pimenta, S., Taylor, R., & Johnson, R. (2022). Open Science: Recommendations for Research on School Bullying. In International Journal of Bullying Prevention. ↗️
NWO. (2023, December 13). Open Science Fund. NWO. ↗️
Open Research Europe. (2024). Open Data, Software and Code Guidelines. Open Research Europe. ↗️
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. ↗️
Opitz, D., & Witzel, A. (2005). The Concept and Architecture of the Bremen Life Course Archive. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(2), Article 2. ↗️
Parry, O., & Mauthner, N. S. (2004). Whose Data are They Anyway?: Practical, Legal and Ethical Issues in Archiving Qualitative Research Data. Sociology, 38(1), 139–152. ↗️
Paulus, T., Woods, M., Atkins, D. P., & Macklin, R. (2017). The discourse of QDAS: reporting practices of ATLAS.ti and NVivo users with implications for best practices. In INTERNATIONAL JOURNAL OF SOCIAL RESEARCH METHODOLOGY (Vol. 20, Issue 1, pp. 35–47). ROUTLEDGE JOURNALS, TAYLOR & FRANCIS LTD. ↗️
Penders, B., Holbrook, J. B., & de Rijcke, S. (2019). Rinse and Repeat: Understanding the Value of Replication across Different Ways of Knowing. Publications, 7(3), Article 3. ↗️
Perreault, G. P., & Dienlin, T. (2022). How open is it really? A qualitative study on the normalization of open science practices in Mass Communication. ↗️
Perrier, L., Blondal, E., Ayala, A. P., Dearborn, D., Kenny, T., Lightfoot, D., Reka, R., Thuna, M., Trimble, L., & MacDonald, H. (2017). Research data management in academic institutions: A scoping review. PLOS ONE, 12(5), e0178261. ↗️
Phillippi, J., & Lauderdale, J. (2017). A Guide to Field Notes for Qualitative Research: Context and Conversation. Qualitative Health Research, 28(3), 381–388. ↗️
Plesser, H. E. (2018). Reproducibility vs. Replicability: A Brief History of a Confused Terminology. Frontiers in Neuroinformatics, 11. ↗️
Policy and Strategy Branch. (2023). Open Access Policy (No. Version 2021.1; p. 15). Australian Research Council. ↗️
Pool, R. (2017). The verification of ethnographic data. Ethnography, 18(3), 281–286. ↗️
Porte, G., & Richards, K. (2012). Focus article: Replication in second language writing research. In JOURNAL OF SECOND LANGUAGE WRITING (Vol. 21, Issue 3, pp. 284–293). PERGAMON-ELSEVIER SCIENCE LTD. ↗️
Pownall, M. (2022). Is replication possible for qualitative research? [Preprint]. PsyArXiv. ↗️
Pownall, M., Talbot, C. V., Henschel, A., Lautarescu, A., Lloyd, K. E., Hartmann, H., Darda, K. M., Tang, K. T. Y., Carmichael-Murphy, P., & Siegel, J. A. (2021). Navigating Open Science as Early Career Feminist Researchers. Psychology of Women Quarterly, 45(4), 526–539. ↗️
Pownall, M., Talbot, C. V., Kilby, L., & Branney, P. (2023). Opportunities, challenges and tensions: Open science through a lens of qualitative social psychology. British Journal of Social Psychology. Scopus. ↗️
Pownall, M., Terry, J., Collins, E., Sladekova, M., & Jones, A. R. R. (2022). UK Psychology PhD Researchers’ Knowledge, Perceptions, and Experiences of Open Science. ↗️
Pratt, M. G., Kaplan, S., & Whittington, R. (2020). Editorial Essay: The Tumult over Transparency: Decoupling Transparency from Replication in Establishing Trustworthy Qualitative Research. Administrative Science Quarterly, 65(1), 1–19. ↗️
Prosser, A. M. B., Hamshaw, R. J. T., Meyer, J., Bagnall, R., Blackwood, L., Huysamen, M., Jordan, A., Vasileiou, K., & Walter, Z. (2022). When open data closes the door: A critical examination of the past, present and the potential future for open data guidelines in journals. British Journal of Social Psychology, bjso.12576. ↗️
Pujol Priego, L., Wareham, J., & Romasanta, A. K. S. (2022). The puzzle of sharing scientific data. Industry and Innovation, 29(2), 219–250. ↗️
Rainey, J., Macfarlane, S., Puussaar, A., Vlachokyriakos, V., Burrows, R., Smeddinck, J. D., Briggs, P., & Montague, K. (2022). Exploring the Role of Paradata in Digitally Supported Qualitative Co-Research. CHI Conference on Human Factors in Computing Systems, 1–16. ↗️
Reeping, D., & Edwards, C. D. (2020). Exemplars of integration in engineering education’s use of mixed methods research. In ASEE Annual Conference and Exposition, Conference Proceedings (Vols. 2020-June). ↗️
Reeves, J., Treharne, G. J., Ratima, M., Theodore, R., Edwards, W., & Poulton, R. (2023). A one- size-fits-all approach to data-sharing will not suffice in lifecourse research: A grounded theory study of data-sharing from the perspective of participants in a 50-year-old lifecourse study about health and development. BMC Medical Research Methodology, 23(1), 118. ↗️
Reischer, H. N., & Cowan, H. R. (2020). Quantity Over Quality? Reproducible Psychological Science from a Mixed Methods Perspective. Collabra: Psychology, 6(1), 26. ↗️
Renbarger, R., Adelson, J. L., Rosenberg, J. M., Stegenga, S. M., Lowrey, O., Buckley, P. R., & Zhang, Q. (2023). Champions of Transparency in Education: What Journal Reviewers Can Do to Encourage Open Science Practices. Gifted Child Quarterly, 00169862231184575. ↗️
Riley, S., Brooks, J., Goodman, S., Cahill, S., Branney, P., Treharne, G. J., & Sullivan, C. (2019). Celebrations amongst challenges: Considering the past, present and future of the qualitative methods in psychology section of the British Psychology Society. Qualitative Research in Psychology, 16(3), 464–482. ↗️
Roberts, K., Dowell, A., & Nie, J.-B. (2019). Attempting rigour and replicability in thematic analysis of qualitative research data; a case study of codebook development. In BMC MEDICAL RESEARCH METHODOLOGY (Vol. 19). BMC. ↗️
Roller, M. R., & Lavrakas, P. J. (2018). A total quality framework approach to sharing qualitative research data: Comment on Dubois et al. (2018). Qualitative Psychology, 5(3), 394–401. ↗️
Ross, S. A., & Ballsun-Stanton, B. (2021). Introducing Preregistration of Research Design to Archaeology [Preprint]. SocArXiv. ↗️
Ross-Hellauer, T., Klebel, T., Bannach-Brown, A., Horbach, S. P. J. M., Jabeen, H., Manola, N., Metodiev, T., Papageorgiou, H., Reczko, M., Sansone, S.-A., Schneider, J., Tijdink, J., & Vergoulis, T. (2022). TIER2: Enhancing Trust, Integrity and Efficiency in Research through next-level Reproducibility. Research Ideas and Outcomes, 8, e98457. ↗️
Rubin, M. (2023). Opening up open science to epistemic pluralism: Comment on Bazzoli (2022) and some additional thoughts. ↗️
Ruggiano, N., & Perry, T. E. (2019). Conducting secondary analysis of qualitative data: Should we, can we, and how? Qualitative Social Work, 18(1), 81–97. ↗️
Serghiou, S., Contopoulos-Ioannidis, D. G., Boyack, K. W., Riedel, N., Wallach, J. D., & Ioannidis, J. P. A. (2021). Assessment of transparency indicators across the biomedical literature: How open is open? PLOS Biology, 19(3), e3001107. ↗️
Siegel, J. A., & LaMarre, A. (2019, October 10). Navigating “Publish or Perish” as Qualitative Researchers. Research Communities by Springer Nature. ↗️
Silverman, M., Terry, M. A., Zimmerman, R. K., Nutini, J. F., & Ricci, E. M. (2002). The Role of Qualitative Methods for Investigating Barriers to Adult Immunization. Qualitative Health Research, 12(8), 1058–1075. ↗️
Smith, H. J., Chen, J., & Liu, X. (2008). Language and rigour in qualitative research: Problems and principles in analyzing data collected in Mandarin. BMC Medical Research Methodology, 8(1), 44. ↗️
Smith, S. (2018). Integrated work-based placements – shifting the paradigm. In Higher Education, Skills and Work-based Learning (Vol. 8, Issue 2, pp. 134–150). ↗️
SpringerNature. (2024). Research data policy. SpringerNature. ↗️
Steinhardt, I. (2020). Learning Open Science by doing Open Science. A reflection of a qualitative research project-based seminar. Education for Information, 36(3), 263–279. ↗️
Steltenpohl, C. N., Lustick, H., Meyer, M. S., Lee, L. E., Stegenga, S. M., Reyes, L. S., & Renbarger, R. L. (2023). Rethinking Transparency and Rigor from a Qualitative Open Science Perspective. Journal of Trial & Error. ↗️
Talkad Sukumar, P., Avellino, I., Remy, C., DeVito, M. A., Dillahunt, T. R., McGrenere, J., & Wilson, M. L. (2020). Transparency in Qualitative Research: Increasing Fairness in the CHI Review Process. In CHI’20: EXTENDED ABSTRACTS OF THE 2020 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS. ASSOC COMPUTING MACHINERY. ↗️
Talkad Sukumar, P., & Metoyer, R. (2019). Replication and Transparency of Qualitative Research from a Constructivist Perspective. ↗️
Tamminen, K. A., Bundon, A., Smith, B., McDonough, M. H., Poucher, Z. A., & Atkinson, M. (2021). Considerations for making informed choices about engaging in open qualitative research. Qualitative Research in Sport, Exercise and Health, 13(5), 864–886. Scopus. ↗️
Tamminen, K. A., & Poucher, Z. A. (2018). Open science in sport and exercise psychology: Review of current approaches and considerations for qualitative inquiry. Psychology of Sport and Exercise, 36, 17–28. Scopus. ↗️
The Royal Society. (2024). Data sharing and mining. The Royal Society. ↗️
Thompson, C., McCaughan, D., Cullum, N., Sheldon, T. A., & Raynor, P. (2004). Increasing the visibility of coding decisions in team-based qualitative research in nursing. In International Journal of Nursing Studies (Vol. 41, Issue 1, pp. 15–20). ↗️
Thoresen, L., & Öhlén, J. (2015). Lived Observations: Linking the Researcher’s Personal Experiences to Knowledge Development. Qualitative Health Research, 25(11), 1589–1598. ↗️
Tonnesson, S. (2012). Active citation through hyperlinks: The retarded replication revolution. In INTERNATIONAL AREA STUDIES REVIEW (Vol. 15, Issue 1, pp. 83–90). SAGE PUBLICATIONS LTD. ↗️
Torka, A.-K., Mazei, J., Bosco, F. A., Cortina, J. M., Götz, M., Kepes, S., O’Boyle, E. H., & Hüffmeier, J. (2023). How well are open science practices implemented in industrial and organizational psychology and management? In European Journal of Work and Organizational Psychology. ↗️
Toronto, C. E., & Remington, R. (Eds.). (2020). A Step-by-Step Guide to Conducting an Integrative Review. Springer International Publishing. ↗️
Torraco, R. J. (2016). Writing Integrative Literature Reviews: Using the Past and Present to Explore the Future. Human Resource Development Review, 15(4), 404–428. ↗️
Tricco, A. C., Lillie, E., Zarin, W., O’Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., … Straus, S. E. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine, 169(7), 467–473. ↗️
Tsai, A. C., Kohrt, B. A., Matthews, L. T., Betancourt, T. S., Lee, J. K., Papachristos, A. V., Weiser, S. D., & Dworkin, S. L. (2016). Promises and pitfalls of data sharing in qualitative research. Social Science & Medicine, 169, 191–198. ↗️
Tucker, T. N. (2016). Grounded Theory Generation: A Tool for Transparent Concept Development. In INTERNATIONAL STUDIES PERSPECTIVES (Vol. 17, Issue 4, pp. 426–438). OXFORD UNIV PRESS. ↗️
Tuval-Mashiach, R. (2021). Is replication relevant for qualitative research? Qualitative Psychology, 8(3), 365–377. ↗️
UK Research and Innovation. (2024). Making your research data open. UKRI. ↗️
Ulpts, S., & Schneider, J. W. (2023). Knowledge Production Modes: The Relevance and Feasibility of ‘Reproducibility.’ MetaArXiv. ↗️
Ulpts, S., & Schneider, J. W. (2024). A conceptual review of uses and meanings of reproducibility and replication. ↗️
U.S. National Science Foundation. (2022, October 25). Reproducibility and Replicability in Science. NSF. ↗️
U.S. National Science Foundation. (2024). NSF Public Access Initiative. NSF. ↗️
van den Berg, H. (2008). Reanalyzing Qualitative Interviews from Different Angles: The Risk of Decontextualization and Other Problems of Sharing Qualitative Data. Historical Social Research / Historische Sozialforschung, 33(3 (125)), 179–192. ↗️
VandeVusse, A., Mueller, J., & Karcher, S. (2021). Qualitative Data Sharing: Participant Understanding, Motivation, and Consent. Qualitative Health Research, 32(1), 182–191. ↗️
Verburg, M., Braukmann, R., & Mahabier, W. (2023). Making Qualitative Data Reusable—A Short Guidebook For Researchers And Data Stewards Working With Qualitative Data. Zenodo. ↗️
Vuckovic Juros, T. (2022). Challenges of qualitative data sharing in social sciences. European Science Editing, 48, e77781. ↗️
Vučković Juroš, T. (2022). Challenges of qualitative data sharing in social sciences. European Science Editing, 48. Scopus. ↗️
Wallach, J. D., Boyack, K. W., & Ioannidis, J. P. A. (2018). Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017. PLOS Biology, 16(11), e2006930. ↗️
Walters, P. (2009). Qualitative archiving: Engaging with epistemological misgivings. Australian Journal of Social Issues, 44(3), 309–320. ↗️
Weller, S. (2023). Fostering habits of care: Reframing qualitative data sharing policies and practices. Qualitative Research, 23(4), 1022–1041. ↗️
Whittemore, R., & Knafl, K. (2005). The integrative review: Updated methodology. Journal of Advanced Nursing, 52(5), 546–553. ↗️
Wilkinson, M. D., Dumontier, M., Aalbersberg, Ij. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., … Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3(1), 160018. ↗️
Yardley, L. (2000). Dilemmas in qualitative health research. In Psychology and Health (Vol. 15, Issue 2, pp. 215–228). ↗️
Yoon, A. (2015). “Making a square fit into a circle”: Researchers’ experiences reusing qualitative data. Proceedings of the American Society for Information Science and Technology, 51(1), 1–4. ↗️
Editors
Kathryn Zeiler
Adrian Barnett
Editorial assessment
by Adrian Barnett
In this review article, the authors examine reproducibility and replicability in qualitative research, including hundreds of published papers and the grey literature. Both reviewers found the article to be technically sound, interesting, and of likely value to the field. The reviewers’ suggestions for improvement largely concern interpretations. Both reviewers had suggestions for how barriers and enablers are presented. There’s a valid question on framing the need for this review based on the replication crisis. The authors should consider their use of vote counting to provide interpretations, as this potentially lacks nuance. The authors should consider a clearer separation between replication and transparency. The reviewers highlight some citations and related arguments that should be checked.
Discussion phase: Both reviewers read the other reviewer’s review. The reviewer’s felt their reviews were complementary, but highlighted a few small differences too. Reviewer 2 was not as concerned about “vote counting” as reviewer 1.
Minor editorial comments: You could reduce the decimal places in the table and reduce the clutter. Percentages when the sample size is under 100 can always be rounded. I would recommend adding ORCIDs to the PDF as well as the online version.
Peer review 1
Anonymous reviewer
Summary
This paper is an integrative review that synthesizes both peer-reviewed and grey literature on reproducibility and open science in qualitative research. The analysis included 248 papers and concluded that
Strengths
This is a topic of great interest to many in the field. Perhaps more importantly, it is a topic of relevance to even more people in the field, meaning a lot of people have opinions/thoughts on the topic even if it is not their area of expertise. Because of this, I think having a synthetic review of what has been published about the topic would be of great value.
Inclusion of grey literature is valuable here as it helps provide broader representation of how the issues are being discussed. As a reader interested in how others have written about this topic, I learned a lot reading this paper and greatly appreciate the summarization work done by the authors. If nothing else, this summarization and synthesis is hugely valuable.
Opportunities for Improvement
In a bit of irony, the paper appears to rely on an “informal’’ “count the votes” weighting to conclude that quantitative framings are inappropriate for qualitative research and that alternative conceptualizations can be applicable and appropriate in qualitative research. I note irony because in quantitative synthetic work (e.g., meta-analysis), it has long been believed that simple vote counting in syntheses (e.g., how many studies found the intervention works?) is sub-optimal. And yet in this paper, vote counting appears to be the foundation of the conclusions. Moreover, it is vote counting by paper, and not by individuals, meaning some prolific individuals may be running up the score by repeating their stance in multiple papers.
I find this particularly important to note because the authors conclude things like, “We found that conceptualizations of reproducibility and replicability that stem from positivist, quantitative standpoints are overwhelmingly framed as inappropriate practices and epistemic criteria for (most) qualitative research” p. 39. Use of words like “overwhelmingly” suggest a clear “vote counting” strategy played a key part in their impressions.
Abstract Clarity
Despite being someone familiar with the topic and discussion, I found the abstract difficult to follow. Even after repeated reading, I could not understand the summary of their findings/conclusion in the Abstract (e.g., “key barriers” to what?, “Key enablers” to what?). I also note a missed opportunity to find parallels and overlap between quantitative and qualitative research. For example, the end of the abstract states, “We conclude that reproducibility, replicability and Open Science practices must be adapted to the aims and epistemic conditions of qualitative research for them to be applicable and feasible, and that they will not always be both for all qualitative research.” Is this not also the case in quantitative research? (Although, after reading only the abstract, it remains unclear how replicability and reproducibility can be–much less must be– adapted. What is an adapted replication or adapted reproduction attempt?
Assessment of Accuracy/Relevance of Concern: Barriers or Perceived Barriers?
Almost without fail, when I have given talks about open science practices, I will get a question about data sharing and “being scooped” (someone else using shared data before the data collectors can use it). I typically respond with a version of: one can’t be scooped with data that’s already been analyzed (i.e., data are typically shared with a paper, not before the paper). If it’s a multi-wave longitudinal study, then a limited form of scooping may be possible, but only for papers that the original author team did not appear to plan to conduct themselves. I mention this example of a commonly expressed concern that can easily be addressed/assuaged because if I were to “count the votes”, then scooping would appear high as a barrier to data sharing. But this simple “count the votes” of concerns ignores the validity/applicability of the concern.
Similarly, another common question I get is “but what about participant consent?”, to which I typically respond saying that agreeing to have their data shared needs to be included in consent forms; if you’ve already collected your data, then sharing data for that project is likely unfeasible. I raise these as two examples of common concerns that are easily addressed (and have been by the field for years). But the ease with which the concerns can be addressed does not appear in any simple “count the votes” analysis. I acknowledge that my examples are not equivalent to the concerns covered in the paper. Asking a question at a talk does not require the same amount of forethought as publishing a paper. At the same time, I am not convinced that publishing a paper removes all instances of expressing a concern that can be addressed.
Related: Sometimes, the authors use language associated with perceptions (e.g., “are overwhelmingly framed as inappropriate practices and epistemic criteria for (most) qualitative research.” p. 39. But don’t assess whether the perceptions are accurate. I also noted this in their discussion of Barriers. Are they truly barriers or perceived barriers? Or perhaps we can’t really distinguish the difference in all instances?
Scope of Paper
The title and abstract largely frame the paper around replicability and reproducibility. But other times (e.g., what was coded for in Tables 2 and 3, key issues and barriers in Table 4) feels like the authors were not limiting their scope to just those two practices but to open science generally (sometimes even switching to the broader topic mid-paragraph, e.g., “Some authors offer similar accounts of the relationship between Open Science and qualitative research” p. 20). Without greater clarification, it seems that the authors may be guilty themselves of an activity that one of the papers they cite notes as troublesome: coupling replication with transparency (“Advocates for transparency in qualitative research mistakenly couple it with replication. Tying transparency tightly to replication is deeply troublesome for qualitative research, where replication misses the point of what the work seeks to accomplish.” Pratt et al., abstract).
Seeking Explicit Interpretation
I was expecting the “Interpretation of key findings” section to include the authors’ interpretations of their review about replicability and reproducibility and qualitative research. The section certainly includes this at times. However, the bulk of this section:
A. seems to avoid interpreting and continues to report the claims of others (e.g., “One could argue, as do some included in this review, that epistemic exclusion within science reform movements generally and within Open Science specifically is a driver of universalist policies, practices, and expectations that are a poor fit for (much) qualitative research.” p. 41 Is this the authors suggesting this should be argued? Or intentionally avoiding arguing for it? Or implicitly communicating disagreement amongst the authors on this topic?)
B. covers many things other than “Reproducibility and replicability of qualitative research”. These topics include: open science applicability more broadly and not about replicability/reproducibility and “bring researcher reflexivity into quantitative research”.
Limitations
The authors communicate their positionality on reproducibility in their Limitations section (although not necessarily in their positionality statement) when they state, “we suspect that others would still be unable to recreate exactly our process and/or findings, given our roles as researchers in the processes of data extraction, coding, analysis, and reporting, and the iterative nature of the coding and analytic process.” p. 43 Conceptually, I understand their suspicion about findings (although it appears to ignore the concept of reliability and how it can be calculated even with coding of qualitative data; or are the authors implicitly agreeing with their brief mention of Roberts on p. 21?). But it is unclear what about their process could not be recreated (is this different from reproducibility?) or why they suspect others could not “recreate” it. Did they suspect this prior to their paper (if so, then why go through all the steps?) or after (if after, what happened that fostered this suspicion?) I say all this while also completely agreeing that I could not recreate their methods. For example, based solely on their Methods section, I cannot even begin to guess how they crafted many of their codes (e.g., Barrier).
Moreover, if the authors honestly and truly suspect that their findings would not be reproduced, then how can there be implications for future research and policy based on their findings? It seems that the authors are saying that they suspect their findings are unique to them and then immediately proceed making policy recommendations on them. Or, to frame the question around a topic they cover: is generalizability a goal/relevant to this paper?
Peer review 2
This integrative review surveys the growing literature on qualitative open science, focusing on its conceptualization of reproducibility/replicability and what the authors call barriers and enablers of open science (especially reuse-related) practices. Overall, this is a well-done paper. The methodology (and the transparency with which it is documented) are excellent as are, best I can tell, the results in identifying the relevant literature: I know the field quite well and did not notice any missing literature and was, in fact, quite impressed by the inclusion of some of the hard-to-find grey literature on the topic (such as various contribution to newsletters of sections of the American Political Science Association). I also found the summary and interpretation of the findings to be in line with my reading of the literature. The paper is also very well written: evidence and methods are presented very clearly as are the findings and their interpretation — not an easy feat given the number of moving parts in such a massive review endeavor.
I should note here, as a positionality statement of sorts, that I am an active contributor to the literature reviewed and will reference the parts of the literature I know best (which includes, of course, my own works) — not because I consider them the most important parts of the literature but simply because my familiarity with it facilitates informed commentary. I’m also a political scientist by training and thus particularly familiar with the debate around qualitative research transparency in that discipline.
I do have some concerns/suggestions for improvement — three major and a number of minor concerns:
Major concerns
-
I urge the authors to reconsider the replication crisis framing for the paper. I appreciate that the replication crisis is qualified here as ‘perceived’, but its role in open science is exaggerated: as the literature review finds, there is a significant literature on even qualitative open science that predates the replication crisis (which is typically dated around 2011 with the Bem and the “False Positive Psychology” papers), including the seminal debates around qualitative data sharing involving Bishop, Corti, Fielding, Mauthner, and Parry that foreshadowed many of the themes raised in the paper. Similarly, concerns about reproducibility of science go back to the late 1980s/early 1990s. The crisis framing is mostly used in the first couple of pages and has little function beyond that — it would cost little to discard it entirely, or at least qualify it significantly (maybe along the lines of ‘the perceived replication crisis increased interest in open science in both qual and quant, causing a spike in the literature’).
-
I was left dissatisfied with the barriers vs enablers classification of themes in the second part of the review. For one, what was classified as an ‘enabler’ was unclear to me. Data sharing, for example, doesn’t _enable_ open science, it is one of its constitutive elements. I also think the way barriers and enablers are listed with lots of citations to authors who are fundamentally at odds with each other (more on that in a bit) suggests more consensus in the literature than there really is and misses important nuances. To take one example: some authors cited (Mauthner and Parry, Feldman and Shaw) view the epistemological challenges to data sharing as a fundamental and in surmountable concern, both ethically and epistemologically. Other authors (Corti, Bishop, Mannheimer et al., my own work in Karcher et al 2021) see it as a challenge that can be addressed with a combination of high-quality documentation and responsible re-use). I’m not entirely sure how to resolve this — maybe distinguishing between objections and challenges might work? Similarly, “context” is cited as a barrier to research — it is unclear to me what that means, and many of the works cited (Corti, Mannheimer et al.) describe context not as a barrier but as a necessary condition for (and if provided, facilitator of) data re-use. Overall, I found the barriers section to suggest a much more skeptical view as many of the authors cited in it (certainly myself) hold about the possibility of open qualitative research.
-
Which brings me to my third concern. In a surprising number of places, papers that advocate explicitly for qualitative open science (while calling for a nuanced, responsible approach) are cited as if they suggest the opposite. For example, Haven and Van Grootel (2019) are cited for the claim that preregistration is “an ontological mismatch with qualitative research” — when the article is one of the foundational pieces establishing the practice and was followed by Haven’s work setting up a template for doing so in OSF. Similarly, Karcher et al (2021) is cited to support “that some believe that even an original researcher cannot revisit their own data” — a claim that we indeed report (and associate with the work of Mauthner and Parry), but clearly disagree with (given that our article is about espistemically responsible re-use). Similarly, Van de Vusse et al (2021) (of which I’m also a co-author) is cited around the claim that members of vulnerable groups “may be less willing to participate” in data sharing — when the paper reports a >90% opt-in rate to data sharing among a relatively vulnerable group (participants in a study about abortion, half of whom had abortions). There are more examples along these lines (Libby Bishop, whose 2009 article is a seminal piece advocating for the ethical sharing of qualitative data, would be the surprised to see her views characterized as a ‘hardline stance’ against it) — the citations may not be technically wrong, but I did find their effect misleading and would closely review them closely and cite more selectively in a revised version. This may be helped by more systematically categorizing the papers reviewed based on their overall view about the practices they discuss.
Minor issues
-
In the introductory paragraph to section 3 (Results), the paper describes 67% of studies as addressing reproducibility and 81.2% as open science — together with the following Venn diagram, that suggests about 19% of studies that address reproducibility but not open science. Given that I understand (and I took the paper to similarly understand) open science as an umbrella term encompassing reproducibility as well as other practices like open access, what these 19% are merits clarification.
-
The “Methods” listed in Figure 3 are a bit odd — most of these (interviews, focus groups) are data collection methods, but then ‘discourse/content’ is listed, which is an analysis method (with no other analysis methods listed). This seems odd. I would either remove or explain what discourse/content refers to and then describe the methods explicitly as data collection methods.
-
I would typically not remark on this, but since this an integrated review deals with literature as data, I felt worthwhile pointing out some citation oddities: Elman 2014 is missing a co-author (Kapiszewski). Jones and Alexander (2018) is missing about a dozen co-authors. It’s also an early preprint version of the also cited Alexander et al. (2020), which should probably be cited instead. The paper heavily cited Büthe and Jacobs (2015), the concluding remark in a section newsletter — it would seem more appropriate to rely on the (peer reviewed, based on extensive disciplinary consultations) Jacobs et al. (2021) which makes many of the same points. Overall, I was sometimes left wondering why some papers were selected for citation (in may own case, e.g., Karcher et al. 2016, which is mostly about data infrastructure, is cited heavily, whereas what I would consider my most relevant paper — Kapiszewski and Karcher 2021— is not cited, though present in the Zotero library.) This might warrant some discussion given the authors’ overall remarkable transparency about the generation of the evidence presented.
These issues non-withstanding, this was an enjoyable read. I do hope the authors are able to use some of the above to further improve the paper, which provides a significant service to the emerging ‘field’ of work on qualitative open science.


